Matlab engine version R2022b inside multiprocessing pool get stuck. Any suggestion?

Question

I developed Python code, version 3.8.5, through which I called R2018a Matlab using matlab engine function to process the data using ubuntu 20.04. I used the pool command from the multiprocessing library and I was able to use 20 threads while using the Matlab engine to speed up the process. The code was working fine and I was able to obtain results multiple times. However, I have upgraded Matlab to the R2022b version and now using the exact same code, we are facing an issue with the MATLAB engine being called inside the pool. The code stuck when it reaches the Matlab engine function, and it doesn’t even open the Matlab function to process data. It gets stuck at "Starting Matlab Engine" and does not proceed. I am using Jupyter notebook, however, I have tried my code using the .py inside the terminal and still got the same result. I have created an example code as follows for your review. Please note that the Matlab engine works perfectly fine when it is being called separately. Also, the pool is working as expected when called separately. However, the code gets stuck when the Matlab engine is called inside the pool. I’d appreciate any comment or feedback to assist me to tackle the issue.

import multiprocessing
from multiprocessing import Pool
import matlab.engine
import time

    
print("defining function ...")


    
def function88 (j):
    print("cal result1 ... ")
    result1=j**2
    print(result1)

    print("Starting Matlab engine...")
    eng = matlab.engine.start_matlab()
    print("Matlab engine started...")

    print("Calling Matlab function...")
    out = eng.test(j)
                    
                    
    print("Matlab function output:")
    print(out)

    return result1

print("calling pool ...")
pool = multiprocessing.Pool(1)

print("calling pool.map")
time.sleep(3)
result = pool.map(function88, range(10))

print("printing result")
print (result)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The result is as follows:

defining function ...
calling pool ...
calling pool.map
cal result1 ... 
0
Starting Matlab engine...

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

My expectation was to see the Matlab function output.

Answer 1

In case this question is still relevant. I had similar issues and for me it helped starting and ending the matlab engine within the multi-process function call.

The pseudo code for calling multiple matlab executions per process looks something like this:

def run_matlab_in_parallel(batch_data):
    res_list = []
    ## Start Matlab Engine
    eng = matlab.engine.start_matlab(background=False)  
    eng.cd(r'''Directory of Matlab Script''')
    
    ## Run Script 
    for params in enumerate(batch_data):
        a, b, c = params
        res = eng.my_matlab_script(a, b, c, background=True).result()
        res_list.append(res)
    
    ## End Engine
    eng.quit()
    return res_list


def main():
    ## Start Pool with n_cpu = 2
    with Pool(2) as p:
        matlab_matrix = p.starmap(run_matlab_in_parallel, multi_process_params)
    
    ## Convert Matlab Mat into Numpy Array
    matlab_arr = np.array(matlab_matrix)
    return matlab_arr