run command taking two arguments with GNU parallel

Question

I have a perl program that takes two arguments, dictionary file composed of english words one per line, and file with concatenated words also one per line, something like this:

lovetoplayguitar
...
...

So normally program is used like:

perl ./splitwords.pl words-en.txt bigfile.txt

It prints results to stdout.

I am trying to put it through GNU parallel like this:

time parallel -n 2 -j8 -k perl ./splitwords.pl {1} {2} ::: words-en.txt bigfile.txt > splitted.txt

but it doesn't work that way.. Tried many combinations so far but was unable to run it using parallel.

EDIT

Actually this seems to be working, however it is using only one core..? Why..?

What do you expect GNU Parallel to run? GNU Parallel cannot magically parallelize your program, but it can run multiple copies of your program with different arguments. What arguments do you expect it to run your program with? — Ole Tange
that is a problem I am facing, because the two arguments are two filenames, which get passed to that ./splitwords.pl program, so it accepts only those two, which are the same — branquito
If you do not know how to split your task into smaller tasks by hand, then it will be close to impossible to have GNU Parallel do it for you: It cannot guess how to parallelize the task. It seems you believe the task can be parallelized, can you explain how you would split the task up into smaller jobs? — Ole Tange
only way I can think of is, by splitting that bigfile into smaller chunks, then feed those to parallel as 2nd argument, with first one being constant, as it holds the dictionary — branquito

Ole Tange Ole Tange · Accepted Answer · 2015-10-04T00:44:03

This will chop bigfile into 1 MB chunks:

cat bigfile.txt | parallel --pipe --cat -k perl ./splitwords.pl words-en.txt {}

If the perlscript only reads the file then this will be faster:

cat bigfile.txt | parallel --pipe --fifo -k perl ./splitwords.pl words-en.txt {}

run command taking two arguments with GNU parallel

1 Answers