slurm, srun and parallelism

Question

I have seen the following two very similar schemes used when submitting jobs with multiple steps to slurm:

On the one hand you can do

#SBATCH -N1 -c1 -n5 # 5 tasks in total on 1 node, 1 cpu per task
for j in {1..4}; do
    srun --exclusive -n1 script $j &
done
srun --exclusive -n1 script 5
wait

On the other hand you can do

#SBATCH -N1 -c1 -n5 # 5 tasks in total on 1 node, 1 cpu per task
for j in {1..5}; do
    srun --exclusive -n1 script $j &
done
wait

Because the job should have only 5 CPUs allocated to it I don't really understand how the second one can work correctly, since after four job steps have been started with srun there is no way the scheduler can allocate a fifth job 'in the background' and then return to the original script... where would the original script run? (I admit my knowledge of these things is pretty limited though).

However, I have personally tested both ways and they both seem to work exactly the same. The second script is a bit simpler in my opinion, and when dealing with somewhat larger input scripts this can be an advantage, but I'm worried that I don't understand 100% what is going on here. Is there a preferred way to do this? What is the difference? What is Bash/slurm doing behind the scenes?

Answer 1

They both work the same in principle, though the second one is clearer (and correct - see below). Each invocation of srun will run script on a separate CPU (probably - though if it runs very fast they could run on a subset of the sbatch-allocated CPUs).

I think the first example doesn't need wait, since the last command isn't run in the background.

By the way, the first example has an error: %j is local to the for-loop, so the last run inside the loop and the run outside the loop both invoke script 4.