How to do multiprocessing using Python for .NET on Windows?

Question

Now I'm developing C# app running on Windows. Some of processes are written in Python, that called via pythonnet (Python for .NET). The processes are calculation-heavy, so I want to do them in parallel.

They are CPU-bounded and can be handled independently.

As far as I know, there are 2 possible ways to realize it:

Launch multiple Python runtime
The first way is launching multiple Python interpreters but it seems unfeasible. Because pythonnet aparently can manage only one interpreter that initialialized by static method, PythonEngine.Initialize().
From the Python.NET documentation:

Important Note for embedders: Python is not free-threaded and uses a global interpreter lock to allow multi-threaded applications to interact safely with the Python interpreter. Much more information about this is available in the Python C-API documentation on the www.python.org Website.
When embedding Python in a managed application, you have to manage the GIL in just the same way you would when embedding Python in a C or C++ application.
Before interacting with any of the objects or APIs provided by the Python.Runtime namespace, calling code must have acquired the Python global interpreter lock by calling the PythonEngine.AcquireLock method. The only exception to this rule is the PythonEngine.Initialize method, which may be called at startup without having acquired the GIL.
Use multiprocessing package in Python
The other way is using multiprocessing package. According to Python documentation, following statement is necessary if the code runs on Windows to ensure spawn finite process:
if __name__ == "__main__":
However, the function written in Python is taken as a part of module since it's embedded to .NET.
For example, following code is executable, but spawns processes infinitely.

//C#
static void Main(string[] args)
    {
        using (Py.GIL())
        {
            PythonEngine.Exec(
                "print(__name__)\n" + //output is "buitlins"
                "if __name__ == 'builtins':\n" +
                "   import test_package\n" +  //import Python code below
                "   test_package.async_test()\n"
                );
        }
    }

# Python
import concurrent.futures

def heavy_calc(x):
    for i in range(int(1e7) * x):
        i*2

def async_test():
    # multiprocessing
    with concurrent.futures.ProcessPoolExecutor(max_workers=8) as executor:
        futures = [executor.submit(heavy_calc,x) for x in range(10)]
        (done, notdone) = concurrent.futures.wait(futures)
        for future in futures:
            print(future.result())

Is there good idea to solve above problem? Any comments would be appreciated. Thanks in advance.

What about using ThreadPoolExecutor instead of ProcessPool one? — Evk
Just to have it mentioned atleast one time - is it a possibility to transalte your pyhton stuff into c# and be happy with easy to use Threadpools etc? :) " spawns processes infinitely." do these processes atleast do their jobs? Do they get sh** done? If yes, your only problem is that endless spawn of processes. I'm asking, because i did not fully unterstand what is going wrong (besides infinite process spawn) — Tobias Theel
@Evk As I mentioned in the post, the calculation process is CPU-bounded, so ProcessPool is suitable because ThreadPoolExecuter cannot make it faster due to Python's GIL. Please see stackoverflow.com/questions/1226584/… — sfb
@TobiasTheel The Python stuff includes a lot of advanced calculation such as machine learning, so translation may take long time. That's true, my only problem is "endless spawn of processes". If run the code in the post, perhaps do their jobs at least once, but spawns processes infinitely, so my Windows froze. According to Python documentation, we have to protect "entry point" by if name == "main":. I want to know how to do that in embedding application. — sfb
I don't fully understand. How do the two code blocks you posted go together? Does the upper one call the lower one? If so: why do you make the main executable in C# and not in python and import pythondotnet and call your C# code from within python? — hansaplast

Ctznkane525 Ctznkane525 · Accepted Answer · 2018-01-03T00:20:56

For each python call, 1. Create an appDomain 2. Create a task in the appdomain that will run the python asynchronously.

Since it's separate AppDomains, the static methods will be independent.

Creating an using an AppDomain is heavy, so I couldn't do it if the number of calls you have is extremely large, but it sounds like you just might have a small number of processes to run asynchronously.

How to do multiprocessing using Python for .NET on Windows?

1 Answers