python Global Interpreter Lock GIL problem

3
votes

I want to provide a service on the web that people can test out the performance of an algo, which is written in python and running on the linux machine

basically what I want to do is that, there is a very trivial PHP handler, let's say start_algo.php, which accepts the request coming from browser, and in the php code through system() or popen() (something like exec( "python algo.py" ) ) to issue a new process running the python script, I think it is doable in this part

problem is that since it is a web service, surely it has to serve multiple users at the same time, but I am quite confused by the Global Interpreter Lock GIL http://wiki.python.org/moin/GlobalInterpreterLock that the 'standard' CPython has implemented, does it mean, if I have 3 users running the algo now (which means 3 separated processes, correct me if I am wrong plz), at a particular moment, there is only one user is being served by the Python interpreter and the other 2 are waiting for their turns?

Many thanks in advance

Ted

2
pythongil
Are you using Apache and mod_wsgi? If so, concurrent requests are handled at that level well outside Python. If not, please explain why you're not using Apache (or nginx) and mod_wsgi. – S.Lott
@S.Lott thx, but the problem is not about the concurrent REQUESTS, it is about concurrently running the python script cuz those script will take a long time to run whereas the php code will simply return after issuing the "python algo.py" cmd – Ted Xu
The question conflates concurrent requests with each request running a script. The question is confusing. Any time you mention the GIL in the context of Apache and mod_wsgi, you have to explain what -- precisely -- you think is going on. Apache separates the requests. Each request forks a separate process. Please be perfectly clear that you understand that each separate request (made separate by Apache and mod_wsgi) actually submits a separate subprocess. Please be very clear -- in the question -- that you understand this. – S.Lott

2 Answers

10
votes

If you are opening each script by invoking a new process; you will not run afoul of the GIL. Each process gets its own interpreter and therefore its own interpreter lock.

7
votes

The GIL is per-process. If you start multiple python processes, each will have its own GIL that prevents the interpreter(s) in this specific process from running more than one thread at a time. But independent processes can run at the same time.

Also, multiple threads inside one Python process do take turns on running (rather frequently, IIRC once per hundred opcode instructions or a few dozen milliseconds depending on the version), so it's not like the GIL prevents concurrency at all - it just prevents multi-threading.