Variable python arguments not accepted by GNU Parallel

Question

I have a python code that runs a simulation for me and I need to run this code for a number of different parameters (a string, an integer, and another stringy integer). My problem is that when I attempt to parallelize this with GNU-parallel part of my arguments don't go through (my first string). The particular line that is likely responsible for this is:

parallel --bar --eta -j14 'python3 compute-heat-capacity-generational-2 $sim {1} {2}' ::: \
$(seq 16) ::: \
${gens[@]}

where the error msg I get from python is:

Traceback (most recent call last):
File "compute-heat-capacity-generational-2", line 20, in

iterNum = int(argv[3]) > IndexError: list index out of range

So only two of my arguments are going through, the ones assigned to {1} and {2} in my parallel line. For whatever reason it doesn't seem like $sim is working as intended when I try to use parallel.

The full script is written below where I take the simulation directory as input, loop through the directory to parse the generation number, and then hand over both of those to parallel where I iterate across all generations and a third parameter iterated by ${seq 16}

Any advice on how to work with parallel would be greatly appreciated. I've spent the better part of the last few days reading manuals and example but I haven't been able to solve this problem. I'm fairly certain the problem is something simple with how I label my arguments `{}' and feed them the inputs, but I just can't seem to find the right combination to get this off the ground.

#!/bin/bash

echo "Choose a simulation: "
read sim
echo "Starting Heat Capacity Calculations"

gens=(save/$sim/isings/*) 

#find all generations in simulations folder, put into array
for ((i=0; i<${#gens[@]}; i++));
do
    #find the generation number from file string
    gens[i]=$(echo ${gens[$i]} | cut -d "[" -f2 | cut -d "]" -f1)
done

parallel --bar --eta -j14 'python3 compute-heat-capacity-generational-2 $sim {1} {2}' ::: \
$(seq 16) ::: \
${gens[@]}

sneep sneep · Accepted Answer · 2018-03-29T05:53:00

Try using either double-quotes in the argument to parallel, or do export sim before calling parallel.

parallel --bar --eta -j14 "python3 compute-heat-capacity-generational-2 $sim {1} {2}" ::: \
$(seq 16) ::: \
${gens[@]}

bash doesn't expand variables in strings if the string uses single quotation marks (').
If you do export sim, the variable will be visible to sub-processes, and your variable will be expanded later.

Variable python arguments not accepted by GNU Parallel

1 Answers