1
votes

I'm struggling to try and run gnu parallel. I have a shell script that calls a python program several thousand times with different input params:

python /path/to/program/run.py A_02_01 input.fasta > /path/to/output/out.txt
python /path/to/program/run.py A_02_02 input.fasta > /path/to/output/out.txt

I tried using gnu parallel like so:

cat iedb_classi_call.sh  | parallel  --recstart 'python' --recend '\n' --pipe   bash

But all my output files are empty. I'm struggling to figure out why. I'm not getting errors from gnu parallel.

Before I added the recstart and recend options, I was getting non-empty output files for some python calls, but other program calls weren't executing and getting errors like:

run.py: error: incorrect number of arguments
bash: line 422: 01_ input.fasta: command not found
Usage: run.py allele fasta_file

Which made me think parallel was reading in chunks not separated properly and I added the --recstart / --recend parameters to parallel

I'm using gnu parallel version 20180722

1
Ugh! What parameters do you need to run with - I mean where are they? - Mark Setchell
Why don't you put a shebang at the start of the Python script and make it executable so you don't need to keep typing python to run it? - Mark Setchell
NVM - I was misunderstanding how to use parallel. I just wrote a wrapper script to generate calls with needed parameters to my program then just piped it to parallel. It seems to be working: generate-runs.sh | parallel - aindap
In your example the output file is named the same. If that is correct, then that will explain why it is empty. The output files must have different names. - Ole Tange

1 Answers

0
votes

(This should be a comment as it does not answer the actual question, but formatting code does not work very well).

If the lines are like:

python /path/to/program/run.py A_02_01 input.fasta > /path/to/output/out.txt
python /path/to/program/run.py A_02_02 input.fasta > /path/to/output/out.txt

Then you might be able to do:

cat file-with-A_names |
  parallel --results /path/{}.out python /path/to/program/run.py {} input.fasta >/dev/null

The output will be stored in /path/A....out.