Multiprocessing return objects created by process

Question

I am using multiprocessing lib for accelerating process of class creation. Minimal example of my code:

from matplotlib.backends.backend_qt5agg import FigureCanvas
class Custom_Class(FigureCanvas)
    .
    .
    .

def generate_class_func(list_of_dfs, arg1, arg2):
    list_of_custom_classes = list()
    for df in list_of_dfs:
        custom_class = Custom_Class(df, arg1, arg2)
        list_of_custom_classes.append(custom_class)
    return list_of_custom_classes

def main():
    import multiprocessing as mp
    with mp.Pool() as p:
        list_of_classes_list = p.starmap(generate_class_func, zip(list_of_dfs, repeat(arg1), repeat(arg2)))
        p.close()
        p.join()
    

if __name__ == '__main__':
    main()

However, I got multiprocess.pool.MaybeEncodingError: Error sending result: ... Reason: 'TypeError("cannot pickle 'Custom_Class' object")' error. I also tried pathos.ProcessingPool module but the error is same. How can I return custom objects with multiprocessing?

Answer 1

I'm the author of dill and multiprocess. One easy trick to make thing more serializable is to use multiprocess instead of multiprocessing. The former is a fork of the latter that uses dill instead of pickle, so you immediately get the ability to serialize more objects, including most custom classes.

>>> from matplotlib.pylab import FigureCanvasBase
>>> class Custom_Class(FigureCanvasBase):
...   def __init__(self, df, arg1, arg2):
...     super().__init__()
...     self.df = df
...     self.arg1 = arg1
...     self.arg2 = arg2
... 
>>> def generate_class_func(list_of_dfs, arg1, arg2):
...     list_of_custom_classes = list()
...     for df in list_of_dfs:
...         custom_class = Custom_Class(df, arg1, arg2)
...         list_of_custom_classes.append(custom_class)
...     return list_of_custom_classes
... 
>>> 
>>> import multiprocess as mp
>>> with mp.Pool() as p:
...   from itertools import repeat
...   list_of_classes_list = p.starmap(generate_class_func, zip([[0,1],[2,3],[4,5],[6,7]], repeat('arg1'), repeat('arg2')))
...   p.close()
...   p.join()
... 
>>> list_of_classes_list
[[<__main__.Custom_Class object at 0x11b35a390>, <__main__.Custom_Class object at 0x11b0fcb10>], [<__main__.Custom_Class object at 0x11b373110>, <__main__.Custom_Class object at 0x11ae1fa10>], [<__main__.Custom_Class object at 0x11b37b990>, <__main__.Custom_Class object at 0x11b3965d0>], [<__main__.Custom_Class object at 0x11b114f90>, <__main__.Custom_Class object at 0x11adfdb90>]]