How to a execute list of commands in parallel in snakemake

Question

I have a snakemake rule that creates a text file will many shell commands as its output. I would like to design a second rule that would take the file as an input and run all the commands specified in the file in parallel - taking advantage of multiple threads/cores or submitting the commands to a cluster if --cluster is specified. Is that possible?

Answer 1

You could write python code, either using the "run:" block in snakemake or "script:" with an external python script, which uses the subprocess module to run each command in the file in a separate process. I don't think there's a way to do it more directly in Snakemake.