Spawn and detach process in python

Question

I'm writing a daemon process in python3.4 that manages 2-3 other external processes. I try to avoid the GIL by just compartmentalizing the work they do. These other processes are temporary in nature. So what I need:

Start a process from within a python interpreter and immediately detach it from the one that started it. I want them to be fully isolated without sharing stdout, stderr, stdin or other resources. I do not want to wait for the process exit, return code, stdout, stderr or provide data through stdin.

According to the docs, it's now recommended to subprocess. This is what I use now:

subprocess.Popen(["python3.4", daemonfile], env=env)

It works, but:

when the process is started, stdout and stderr are printed on the same console the main daemon was started. I want them to be fully detached.
with 'shell=True', the python interpreter enters 'interpreter mode' instead.
subprocess.Popen was recommend in favour of os.spawn*** calls

What is the best way to start a complete separate process (technically, not even a child?) and have it fully detached from the parent, such that they don't share resources?

There is no point to worry about GIL in this case: each CPython process has its own GIL. — jfs
you probably don't need a true daemon but here's an example implementation and a description (click on the pep link) — jfs
it is wrong to use shell=True together with a list argument. Compare: sh -c 'python a.py' vs. sh -c 'python' a.py (you don't want the latter). — jfs
It looks like all you need is to set stdin, stdout, stderr to DEVNULL in your case. — jfs

Wingzero Wingzero · Accepted Answer · 2015-06-24T01:09:15

First of all, there is no process that is completely 'detached' from any process, even the OS kernel, except the first process. You could find that there always be one root process for all the other processes.

So, there is no way to start a complete separate process not even a child.

What you can do is to use multiprocessing module to avoid GIL limitation, and it is a standalone process that communicate with the main process via pipe, so they don't share the same memory space.

regards other questions:

You can change the stdout and stderr to whatever you like as long as it is a valid descriptor.
you can turn off shell=True... just to make sure you know what is shell=True
subprocess is just a mean to execute something and get its output or errors. Do what is recommended. If you don't care the output or exit code, just ignore it, and use subprocess.PIPE to replace stdout/stderr, then it will not print anything on the console.

I am not sure why it is so important for you to use a standalone process, unless GIL has limited your performance. Using threads instead of processes could improve the performance too, because switching process context is much expensive than thread context. If it is just some IO operations or simple tasks, using threads is more convenient.

Spawn and detach process in python

1 Answers