1
votes

I want to use GNU parallel for the following problem:

I have a few files each with several lines of text. I would like to understand how I can run a script (code.sh) on each line of text of each file and for each file in parallel. I should be able to write out the output of the operation on each input file to an output file with a different extension.

Seems this is a case of multiple parallel commands running parallel over all files and then running parallel for all lines inside each file.

This is what I used:

ls mydata_* |
    parallel -j+0 'cat {} | parallel -I ./explore-bash.sh > {.}.out'

I do not know how to do this using GNU parallel. Please help.

1
Have you walked through the tutorial? man parallel_tutorial or gnu.org/software/parallel/parallel_tutorial.html - Ole Tange
Thanks. It worked like a charm. fantastic library. - RoyS

1 Answers

2
votes

Your solution seems reasonable. You just need to remove -I:

ls mydata_* | parallel -j+0 'cat {} | parallel ./explore-bash.sh > {.}.out'

Depending on your setup this may be faster as it will only run n jobs, where as the solution above will run n*n jobs in parallel (n = number of cores):

ls mydata_* | parallel -j1 'cat {} | parallel ./explore-bash.sh > {.}.out'