2
votes

I want to be able to alternate an argument for each execution of my parallel command.

I have a script to process images and I use GNU Parallel to launch several process at the same time. I use GPU core in this script but only one of my 2 GPU core is used.

I want to be able to use 2 different configuration file (one for GPU 1 et one for GPU 2) but I don't know how to do that.

gsutil ls -d $ee_repo | parallel -j 10 -k "task {} $PWD $year $config" With $config changing from gpu_1 to gpu_2 every other execution

Is this possible or not ?

Thanks ????

EDIT:

Using the job number of the parallel command ({#}) as an argument, then get 0 or 1 by using the modulo of this job number:

gsutil ls -d $ee_repo | parallel -j 10 -k "task {} $PWD $year {#}"

In the task function:

task(){
...
jobnumber=$4    
gpu=$(expr $jobnumber% 2)
...
}
2
what about job slot number {%} or job number {#} - Nahuel Fouilleul
Yes ! Using the job number {#} and then using a modulo 2 on the job number to get 0 or 1. Thanks ! - Pierre-Antoine DOUCHET
great, the job slot number seems better. If for any reason odd job number take more time, 10 jobs could run simultaneously on gpu1 whereas with slot number it couldn't be more than 5. - Nahuel Fouilleul

2 Answers

1
votes

Given your description you probably do not want GNU Parallel to alternate. Imagine that all odd jobs take 1 second and all even jobs take 2 seconds. Then you end up with jobs running on the same GPU while none are running on the other.

What you instead want is to have a job starting on GPU2 as soon as a job finished on GPU2. {%} (aka. job slot) is made for this:

parallel -j2 'sleep {}; echo {}-GPU{%} seq {#}' ::: 1 2 1 2 1 2 1 2 1 2 1 2
0
votes

Something like this should work:

seq 14 | parallel -k echo task {} {= '$_=seq() % 2 ? "gpu1" : "gpu2"' =}

Output

task 1 gpu1
task 2 gpu2
task 3 gpu1
task 4 gpu2
task 5 gpu1
task 6 gpu2
task 7 gpu1
task 8 gpu2
task 9 gpu1
task 10 gpu2
task 11 gpu1
task 12 gpu2
task 13 gpu1
task 14 gpu2