Run two scripts on the same node

Question

I have a node with 20 cores and I would like to know how to make two jobs take each 10 cores of this node. (Both jobs.sh are the same, after launching job.sh I modify parameters of main.R then I run the job.sh again.)

#!/bin/bash
#SBATCH --job-name=main
#SBATCH [email protected]
#SBATCH --mail-type=ALL
#SBATCH --nodes=1                # number of nodes
#SBATCH --ntasks-per-node=10     # number of cores
#SBATCH --time=24:00:00          # walltime
#SBATCH --output=sortie/job.out
#SBATCH --error=sortie/error.err

module load python
module load R
# module load openmpi

R CMD BATCH '--args ../../' main.R sortie/main.Rout

Katia Katia · Accepted Answer · 2018-05-18T19:52:52

One way to do this is to request all 20 cores:

#SBATCH --ntasks-per-node=20

And then call both R scripts and run them on the background. Each of them should not use more than 10 cores. So do not use detectCores() or similar function to set up the number of cores within R, but rather set the number of cores to be 10:

R CMD BATCH '--args ../../' main1.R sortie/main1.Rout &
R CMD BATCH '--args ../../' main2.R sortie/main2.Rout &

wait

At the same time it's not clear why you need to run 2 scripts within one batch script. A more optimal way would be to submit 2 jobs with 10 cores running a single R script each. You can pass parameters to the shell script you submit and then use them to pass to your R script. For example,

If you are running R script main.R and want to pass 2 values, you can do within your submission script:

module load R
Rscript main.R $1 $2

And then call this script twice:

sbatch myscript.sh 3.14 outfile1.csv
sbatch myscript.sh 1.57 outfile2.csv

Then the first job will run as:

Rscript main.R 3.14 outfile1.csv

And the second: Rscript main.R 1.57 outfile1.csv

If you prefer R CMD BATCH format better than Rscript you can use it as well. Just use $1, $2 etc as your arguments to your R script.

Run two scripts on the same node

1 Answers