Calling GNU parallel from GNU parallel

5
votes

What's the proper way to nest calls to GNU parallel?

Silly example:

seq 1 100 | parallel echo {} | parallel seq {} 1000

My understanding is that on an 8-CPU box, each parallel would launch 8 jobs for a total of 64 jobs. If you're calling something more substantial than seq this could potentially overload the box. Is there a way to limit the number of jobs but still make full use of parallelism?

1
shellgnu-parallel
The above would start 16 jobs: 8 echo jobs and 8 seq jobs. You probably wanted: seq 1 100 | parallel "echo {} | parallel -I // seq // 1000" which would start 64 seqs on an 8 core machine - Ole Tange

1 Answers

0
votes

Use -j to limit either the outer or the inner parallel:

seq 1 100 | parallel -j1 "echo {} | parallel -I // seq // 1000"

Often you can express the nested command using mulitple :::: or ::: like this:

parallel seq {1} {3} {2} :::: <(seq 10) <(seq 20 30) ::: 1 2 3

It is better because you will keep 8 jobs running at all time where as the above will in periods run fewer than 8 jobs on an 8 core machine.