SLURM job taking up entire node when using just one GPU

Question

I am submitting multiple jobs to a SLURM queue. Each job uses 1 GPU. We have 4 GPUs per node. However once a job is running, it takes up the entire node, leaving 3 GPUs idle. Is there any way to avoid this, so that I can send multiple jobs to one node, using one GPU each?

My script looks like this:

#SLURM --gres=gpu:1
#SLURM --ntasks-per-node 1
#SLURM -p ghp-queue
myprog.exe

Slurm does not allow that behavior. When your job is assigned a GPU-enabled node, it will keep allocated the node (and its resources) during its whole execution. You can read a related research in dx.doi.org/10.1109/SBAC-PAD.2014.49 — Bub Espinja
What is the output of scontrol show config | grep SelectType ? — damienfrancois
@siserte I beg to disagree. The paper you cite is 4 years old and is about remote use of GPUs. On my cluster, I can submit as many 1-GPU jobs to a multi-GPU node as there are GPUs on the node — damienfrancois
Check that the partition is not configured with 'Shared=Exclusive'(scontrol show partition) and that the node is indeed configured to manage 4 GPUs (gres=... in scontrol show node) — damienfrancois

kepler kepler · Accepted Answer · 2018-09-12T17:38:17

I was also unable to run multiple jobs on different GPUs. What helped was adding OverSubscribe=FORCE to the partition configuration in slurm.conf, like this:

PartitionName=compute Nodes=ALL ... OverSubscribe=FORCE

After that, I was able to run four jobs with --gres=gpu:1, and each one took a different GPU (a fifth job is queued, as expected).

SLURM job taking up entire node when using just one GPU

1 Answers