Executing bash script with parallel for many directories

Question

If I have a bash script (chunks.sh) that execute several mini scripts in parallel, I was wondering how to properly execute chunks.sh so that it runs in parallel for many folders? I have about 1000 folders with files that need to be processed. Here is my script:

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --time=16:00:00
#SBATCH --output=mpi_output_%j.txt
#SBATCH --mail-type=FAIL

cd $SLURM_SUBMIT_DIR

module load gcc
module load gnu-parallel
module load bwa
module load samtools

parallel -j 10 < ../1convertfiles.sh
parallel -j 10 < ../2sortfiles.sh
parallel -j 10 < ../3indexfiles.sh
parallel -j 10 < ../4converttopile.sh
parallel -j 10 < ../5createconsensus.sh
parallel -j 10 < ../6concatenateconsensus.sh

Each folder has a name such as THAKID0001_dir, THAKID0010_dir, etc. So I was wondering how to properly apply a command in this script to loop through my current directory, find all the directories with *_dir attached, then execute all these mini scripts within that directory?

I tried putting my parallel commands into for loops but it was rerunning the mini scripts so many times. I think I can use:

parallel -j 10 < 1convertfiles.sh ::: *_dir/*  
parallel -j 10 < 2sortfiles.sh ::: *_dir/*
etc.

But this logic to me seems that each parallel command block will not be running for the SAME directory at once. Each parallel line will be finding it's own directory to work in and these mini scripts have to run in order, hence why I tried writing a for loop but it was creating a huge mess.

Expected Results:

 $ ./chunks.sh
 ### Should run the list of commands per folder ###
 ### For example, it will execute all the parallel commands in THAK0001_dir then it will execute all the parallel commands in THAK0002_dir, etc ####

TL;DR: How to make chunk.sh execute these parallel commandblocks for all directories with a certain tag (i.e. THAK*_dir) but each line should run once the previous line completed. Hope this made sense..thank you!

The description is a little bit confusing. Can you list the expected sequence that you want to run in sequential mode, and the way to enumerate all jobs ? — dash-o
Oh I had the scripts numbered to show the sequence, so 1blahblah.sh runs first followed by 2blahblah.sh etc. They should all run and do their various transformations for the files in each directory. However I don't want to cd into the directory then call chunks.sh each time because I have 1000s of directories. So I wanted to know how to properly loop through them. — CuriousDude
I believe that you want to executed: for each folder _dir., all the jobs in the stages files: 1convertfile, 2sortfiles.sh, 3indefiles, .... but for a given folder, each stage should complete before the next step can start. At any point of time, you want to have no more than 10 jobs running. Is this correct ? — dash-o
How many jobs in each stage (1convertfiles, 2sortfiles, ...), can you show (or describe) one of the stages ? do the various folder take approximately same amount of time to process ? — dash-o
Each folder has 36 files to process simultaneously but I made it run 10 jobs while accessing 4 tasks per node (tbh I just learned this so I hope I explained that right) so as not to go over 40 tasks per node. I did add that for loop before: for dir in THAK*_dir; do parallel -j 10 < 1convertfiles.sh etc.; done But it was replicating, for example I would see the same process rerun for a file that was already being processed (based on my error message). It was looping through every file again when it doesn't need to waste time doing that. — CuriousDude

dash-o dash-o · Accepted Answer · 2019-10-29T17:49:05

On surface, the problem require helper script that will perform the sequential processing:

process-dir.sh in $SLURM_SUBMIT_DIR

#! /bin/bash
# Process all jobs for current folder, sequentially.
# Input: Folder, e.g. THAKID0001_dir
cd $1
../1convertfiles.sh
../2sortfiles.sh
../3indexfiles.sh
../4converttopile.sh
../5createconsensus.sh
../6concatenateconsensus.sh

And then run in parallel

#! /bin/bash
cd $SLURM_SUBMIT_DIR

module load gcc
module load gnu-parallel
module load bwa
module load samtools

parallel -j10 process-dir.sh ::: *_dir

Or avoid the file process-dir.sh by including a bash function directly:

#! /bin/bash
cd $SLURM_SUBMIT_DIR

module load gcc
module load gnu-parallel
module load bwa
module load samtools

process-dir() {
  # Process all jobs for current folder, sequentially.
  # Input: Folder, e.g. THAKID0001_dir
  cd "$1"
  ../1convertfiles.sh
  ../2sortfiles.sh
  ../3indexfiles.sh
  ../4converttopile.sh
  ../5createconsensus.sh
  ../6concatenateconsensus.sh
}
export -f process-dir

parallel -j10 process-dir ::: *_dir

Executing bash script with parallel for many directories

1 Answers