Can Go spawn and communicate with external processes without starting one OS-thread per external process?

Question

Short version:

Is it possible in Golang to spawn a number of external processes (shell commands) in parallel, such that it does not start one operating system thread per external process ... and still be able to receive its output when it is finished?

Longer version:

In Elixir, if you use ports, you can spawn thousands of external processes without really increasing the number of threads in the Erlang virtual machine.

E.g. the following code snippet, which starts 2500 external sleep processes, is managed by only 20 operating system threads under the Erlang VM:

defmodule Exmultiproc do
  for _ <- 1..2500 do
    cmd = "sleep 3600"
    IO.puts "Starting another process ..."
    Port.open({:spawn, cmd}, [:exit_status, :stderr_to_stdout])
  end
  System.cmd("sleep", ["3600"])
end

(Provided you set ulimit -n to a high number, such as 10000)

On the other hand, the following code in Go, which is supposed to do the same thing - starting 2500 external sleep processes - does also start 2500 operating system threads. So it obviously starts one operating system thread per (blocking?) system call (so as not to block the whole CPU, or similar, if I understand correctly):

package main

import (
    "fmt"
    "os/exec"
    "sync"
)

func main() {
    wg := new(sync.WaitGroup)
    for i := 0; i < 2500; i++ {
        wg.Add(1)
        go func(i int) {
            fmt.Println("Starting sleep ", i, "...")
            cmd := exec.Command("sleep", "3600")
            _, err := cmd.Output()
            if err != nil {
                panic(err)
            }
            fmt.Println("Finishing sleep ", i, "...")
            wg.Done()
        }(i)
    }
    fmt.Println("Waiting for WaitGroup ...")
    wg.Wait()
    fmt.Println("WaitGroup finished!")
}

Thus, I was wondering if there is a way to write the Go code so that it does the similar thing as the Elixir code, not opening one operating system thread per external process?

I'm basically looking for a way to manage at least a few thousand external long-running (up to 10 days) processes, in a way that causes as little problems as possible with any virtual or physical limits in the operating system.

(Sorry for any mistakes in the codes, as I'm new to Elixir and, and quite new to Go. I'm eager to get to know any mistakes I'm doing.)

EDIT: Clarified about the requirement to run the long-running processes in parallel.

This is definitely not an answer to the q as posed, but 1) I know Go can start OS threads to let code keep running during blocking syscalls; I don't know if those threads have, e.g., the big stack allocation you'd tend to associate with a pthreads thread or similar, and 2) it seems plausible to me that the fork+exec for the process would dominate the cost of this operation, and the thread on your program's side is small in comparison. — twotwotwo
@twotwotwo: True, the Go threads don't seem to be terribly heavy-weight. But still, based on my experiments, they still seem to limit the number of external processes that can be practically managed at least more than in Elixir, because of the (small) extra cost. — Samuel Lampa
A thread is an OS thread, aka a pthread, and is completely separate from goroutines. — JimB
Spawning a subprocess will probably consume up to three goroutines blocked on read/write of the usual file descriptors. The scheduler will also spawn three OS threads to replace those consumed by blocking on those read/write operations. How big of a problem is this ? I don't know but it depends on how many threads is too many, vs how many open file descriptors is too many. — Dave Cheney
I tested a 10-thread and 100-thread under go1.5 on Ubuntu amd64. The incremental cost of each of the 90 extra open processes in resident size (roughly, physical memory occupied) worked out to 30.4kb. The virtual-memory size increased 108kb/process, but by definition that doesn't mean 108kb physical RAM occupied. You could run far better tests, but this > nothing i figure. — twotwotwo

Jiang YD Jiang YD · Accepted Answer · 2015-11-27T06:27:51

I find that if we not wait processes, the Go runtime will not start 2500 operating system threads. so please use cmd.Start() other than cmd.Output().

But seems it is impossible to read the process's stdout without consuming a OS thread by golang os package. I think it is because os package not use non-block io to read the pipe.

The bottom, following program runs well on my Linux, although it block the process's stdout as @JimB said in comment, maybe it is because we have small output and it fit the system buffers.

func main() {
    concurrentProcessCount := 50
    wtChan := make(chan *result, concurrentProcessCount)
    for i := 0; i < concurrentProcessCount; i++ {
        go func(i int) {
            fmt.Println("Starting process ", i, "...")
            cmd := exec.Command("bash", "-c", "for i in 1 2 3 4 5; do echo to sleep $i seconds;sleep $i;echo done;done;")
            outPipe,_ := cmd.StdoutPipe()
            err := cmd.Start()
            if err != nil {
                panic(err)
            }
            <-time.Tick(time.Second)
            fmt.Println("Finishing process ", i, "...")
            wtChan <- &result{cmd.Process, outPipe}
        }(i)
    }

    fmt.Println("root:",os.Getpid());

    waitDone := 0
    forLoop:
    for{
        select{
        case r:=<-wtChan:
            r.p.Wait()
            waitDone++
            output := &bytes.Buffer{}
            io.Copy(output, r.b)
            fmt.Println(waitDone, output.String())
            if waitDone == concurrentProcessCount{
                break forLoop
            }
        }
    }
}

Can Go spawn and communicate with external processes without starting one OS-thread per external process?

1 Answers