Automatic qsub job completion status notification

Question

I have a shell script that calls five other scripts from it. The first script creates 50 qsub jobs in the cluster. Individual job execution time varies from a couple of minutes to an hour. I need to know when all the 50 jobs get finished because after completing all the jobs I need to run the second script. How to find whether all the qsub jobs are completed or not? One possible solution can be using an infinite loop and check job status by using qstate command with job ID. In this case, I need to check the job status continuously. It is not an excellent solution. Is it possible that after execution, qsub job will notify me by itself. Hence, I don't need to monitor frequently job status.

Answer 1

I never heard about how to do that, and I would be really interested if someone came with a good answer.

In the meanwhile, I suggest that you use file tricks. Either your script outputs a file at the end, or you check for the existence of the log files (assuming they are created only at the end).

while [ ! -e ~/logs/myscript.log-1 ]; do
    sleep 30;
done

Answer 2

qsub is capable of handling job dependencies, using -W depend=afterok:jobid.

e.g.

#!/bin/bash

# commands to run on the cluster
COMMANDS="script1.sh script2.sh script3.sh"
# intiliaze JOBID variable
JOBIDS=""
# queue all commands
for CMD in $COMMANDS; do
    # queue command and store the job id
    JOBIDS="$JOBIDS:`qsub $CMD`"
done
# queue post processing, depended on the submitted jobs
qsub -W depend=afterok:$JOBIDS postprocessing.sh

exit 0

More examples can be found here http://beige.ucs.indiana.edu/I590/node45.html