How to define slurm output file name using wildcards for grouped rules

Question

I am using a slurm cluster to launch snakemake jobs. My problem is that I do not manage to use the --output and --error options of slurm with custom names for each launched job.

For example I have two rules in a group named "filtmap" (so that the 2 rules are launched in the same job instance). I tried to set up the cluster config like presented in the documentation

here is the config_cluster.json

{
    "__default__":
    {
        "account": "mytilus",
        "time": "10-00:00",
        "nodes": 1,
        "ntasks": 1,
        "partition": "long",
        "mem": 100,
        "output": "logs/cluster/{rule}.{wildcards}.out",
        "error": "logs/cluster/{rule}.{wildcards}.err"
    },
}

and starting snakemake with those options

snakemake --use-singularity \
--jobs 40 --cluster-config config_cluster.json \
-s Snakefile
--cluster "sbatch -A {cluster.account} -p {cluster.partition} \
--output {cluster.output} --error {cluster.error} \
-t {cluster.time} --error {cluster.error} \
--nodes {cluster.nodes} \
--ntasks {cluster.ntasks} --mem {cluster.mem}G \
-D /shared/projects/mytilus/Preprocessing \
--cpus-per-task {threads}"

However this always return an error of the following type whatever wildcard I try to use

WorkflowError:
NameError with group job efafdfe8-225c-594b-a71a-d0d58516876c: The name 'rule' is unknown in this context. Please make sure that you defined that variable. Also note that braces not used for variable access have to be escaped by repeating them, i.e. {{print $1}}`

When removing all use of the --output and --error flags both in the call and in the config file snakemake runs fine. However I would really like to have output and errors files with custom names.

EDIT: After a few tests, it seems the problem does not appear when each rule is run on its own, without group definition. So my question is transforming into "How do I set a job name for each group using common wildcards between included rules?"

Any reason you use --output and --error options twice? May be it has to do something with the error? — Manavalan Gajapathy
oops sorry this was just a copy pasting error for the double output flag. I edited to correct. — Alexis

Manavalan Gajapathy Manavalan Gajapathy · Accepted Answer · 2019-05-13T17:17:41

As per developer on this error,

Yes, things like {rule} and {wildcards} are not supported in combination with groups

How to define slurm output file name using wildcards for grouped rules

1 Answers