1
votes

I want to process many files on many database instances. Instances are on various computers, one computer may host up to 4 instances.

A process is run on an instance via a bash shell :

process-on.sh <file_to_process> <instance_name>

process-on.sh knows how to access the instance (server, script-name, and so on) from the instance_name. The same unix client is used to run processes for all instances on one host, just the remote script name varies.

I would like instances act as a pool of processors, and want to use gnu-parallel as a distributor of files between instances.

I would like to use gnu-parallel, but it is mainly to distribute between computers, not isntances... Maybe there is another tool which can work with pools...

Any hints will be much appreciated.

Best, Christophe

1

1 Answers

0
votes

You will need a way to convert a number into an instance - maybe some sort of lookup table.

instances=13
parallel -j$instances 'sleep 0.$RANDOM; echo this is file {} being processed on instance {%}' ::: {1..100}