Parallel processing in python and need for if __name__ == '__main__'

Question

I've been battling the need for parallel processing in my application. I have relatively good understanding that GIL prevents concurrent processing in a single interpreter, and that there can only be one interpreter per process. At least until 3.10 (I need to use 3.8 to integrate manufacturer's drivers). But my application is proving challenging.

My application requirements:

In my application we I'm creating an image acquisition module that will handle an arbitrary number of Flir cameras connected to a computer, for robotics and machine vision. This means the image acquisition module will run in parallel with other modules that control important things, such as motors and DAQs.

When initialized, the image acquisition module will check all cameras in the system to see which one has the name requested:

import ImageAcquisition as ia

ia.Initialize("cam1", "cam2") # Creates two parallel process. One for each camera.

For each camera a parallel process should be created to handle the image analysis associated with it. This means grabbing frames, displaying them, processing them, and also making some image analysis results available to the main process.

One important requirement is the module is self-contained. However, I'm finding that the if __name__ == "__main__": test is always needed in the program entry code (client code). This is because the new processes will run the main program on their interpreters as well.

Is there an alternative to having if __name__ == "__main__":? So I can make my module fully self-contained.

One alternative could be to use the subprocess library and run the process without the "spawn" mechanism that is used by multiprocessing library in Windows. But there is some overhead when it comes to sharing information between processes that is better handled in the multiprocessing library.

Another alternative is to inject the if __name__ == "__main__": in the user's strings, in our main application.

Answer 1

I'm not sure that I understood you correctly. But it seems I had a similar problem a while ago. One solution to bypass if __name__ was to do a nested import. The function I wanted to run via multiprocessing was hidden in a second module that was imported into my main module.

See this solution here: https://stackoverflow.com/a/29751273/4141279

Answer 2

You could try the following:

import os

_env_variable = f'{__name__}-multiprocessing'
if os.environ.get(_env_variable) is None:
    os.environ[_env_variable] = True  # Child process will not now execute following code

    # Create the processes:
    import ImageAcquisition as ia

On platforms that use the spawn method of creating child processes, the above code will create the child processes when the module is initially imported but will set an environment variable inherited by child processes that will prevent the same code from being executed in the child process.