How to get the external (cluster) job id and not internal snakemake job id in cluster log files?

Question

Background I am using snakemake and I want to add the external (cluster) jobid to my cluster output and error log files. Below is my shell script.

snakemake --configfiles config/config_resources.yml config/config_parameters.yml --latency-wait 60 --use-conda --printshellcmds --jobname PipeName_{name}.jobid{jobid} --jobs 300 --cluster "bsub -W 480 -n {threads} -M {resources.mem_gb}G -R 'rusage[mem={resources.mem_gb}G]' -o logs/{name}_{jobid}.out -e logs/{name}_{jobid}.err".

But I only get the internal one in de log name from snakemake (e.g. 1 to 5), instead of the one from the cluster.

Question How to I get the external job id in there and not the local one made by snakemake? Or where in my shell script did I go wrong?

Cheers and thanks in advance.

Answer 1

bsub supports some filename placeholders using a percent sign, like %J for the LSF job ID, so you can use that:

snakemake ... --cluster "bsub ... -o logs/{name}_%J.out -e logs/{name}_%J.err"

That way snakemake will format the cluster command string with its wildcards (like name) as it constructs the bsub command, and then bsub will put the job ID in place in those filenames. Since Snakemake just sees that as one big command to run for the cluster and doesn't know the details other than formatting the text, I think the two shouldn't clash.