How to distribute slurm tasks evenly over the nodes?

Question

I want to run a script on a cluster ~200 times using srun commands in one sbatch script. Since executing the script takes some time it would be great to distribute the tasks evenly over the nodes in the cluster. Sadly, I have issues with that.

Now, I created an example script ("hostname.sh") to test different parameters in the sbatch script:

echo `date +%s` `hostname`
sleep 10

This is my sbatch script:

#SBATCH --ntasks=15
#SBATCH --cpus-per-task=16

for i in `seq 200`; do
    srun -n1 -N1 bash hostname.sh &
done

wait

I would expect that hostname.sh is executed 200 times (for loop) but only 15 tasks running at the same time (--ntasks=15). Since my biggest node has 56 cores only three jobs should be able to run on this node at the same time (--cpus-per-task=16).

From the ouptut of the script I can see that the first nine tasks are distributed over nine nodes from the cluster but all the other tasks (191!) are executed on one node at the same time. The whole sbatch script execution just took about 15 seconds.

I think I misunderstand some of slurm's parameters but looking at the official documentation did not help me.

damienfrancois damienfrancois · Accepted Answer · 2018-07-04T12:38:31

You need to use the --exclusive option of srun in that context:

srun -n1 -N1 --exclusive bash hostname.sh &

From the srun manpage:

By default, a job step has access to every CPU allocated to the job. To ensure that distinct CPUs are allocated to each job step, use the --exclusive option.

See also the last-but-one example in said documentation.

How to distribute slurm tasks evenly over the nodes?

1 Answers