Optimize code structure for `pathos.multiprocessing`

35 views Asked by At

I have the following code structure:

My Base class is defined in the first module. It contains a number of attributes, properties and methods. Some of the latter use the former two to calculate additional results which are also stored as cached properties, because this calculations can be quite CPU expensive.

# in base.py

from optional import Optional

class Base:
    
    def __init__(self, a,b,c):
        self.a = a
        self.b = b        
        ##### more attributes...
    
    @property
    def prop1(self):
        if not hasattr(self, '_prop1'):
            self._prop1 = self.calc_prop1()
        return self._prop1
    
    ##### more properties like this...
    
    def calc_prop1(self):
        # expensive calculation that uses self.a, self.b etc...
        return prop1
    
    
    def add_optional(self):
        self.optional = Optional(self)
    
    
    def expensive_function(self,x,y):
        a = self.a
        b = self.b
        
        d = self.optional.d
        
        # CPU expensive code
        
        return results

Instances of Base can also contain an instance of Optional which is defined in a second module:

# in optional.py

import base        
        
class Optional:

    def __init__(self, base : 'base.Base'):
        self.a = base.a
        self.prop1 = base.prop1
    
    def calc_d(self):
        # expensive calculation that uses self.a, self.b, self.prop1 etc...
        self.d = result

The instance of Optional uses many of the data contained in the Base instance (attributes and properties) to do its own expensive calculations that deliver results which also have a large data footprint.

Eventually, this means that instances of Base are relatively large, especially when they contain an Optional object.

Now I want to employ pathos.multiprocessing for e.g. Base.expensive_function when iterating over many x and y. Although pathos uses dill and is thus capable of serializing the method, this will require the whole Base instance to be sent to all workers for each process (right?). The resulting overhead kills all potential benefits of the multiprocessing.

My current workaround is to define a Reduced_Base class and move expensive_function there, like this:

class Reduced_Base:
    def __init__(self,base : Base):
        self.a = base.a
        self.d = base.optional.d
    
    def expensive_function(self,x,y):
        a = self.a
        
        d = self.d
        
        # CPU expensive code
        
        return results

I can then use Reduced_Base.expensive_function for my parallelization instead of Base.expensive_function. This way, the object sent to the workers (an instance of Reduced_Base) only contains the necessary data from Base and Optional for expensive_function.

Is there a better solution? I first tried to to contain the data for expensive_function in a closure but multiprocessing doesn't seem to work with dynamically created function objects.

Apart from that: Are there additional measures that could increase the performance when using multiprocessing?

0

There are 0 answers