When a rule fails in snakemake workflow, it returns a descriptive message in log file like following:
Error in rule step1:
jobid: 99
output: results/foo/step1/sample5/outdir/output1.tsv, results/foo/step1/sample5/outdir/output2.tsv
log: logs/step1/sample5_out.log (check log file(s) for error message)
shell:
...
(exited with non-zero exit code)
Since I run workflow over many samples, I have defined a postrun.sh bash script in jobscript.sh directive that runs after snakemake {exec_job}
section and notifies me via userbot in slack channel when a step failed for one or more samples.
jobscript.sh
########################### START SNAKEMAKE CMD ############################
{exec_job}
############################ END SNAKEMAKE CMD #############################
exitstat=$?
export exitstat
echo "END at $(date)"
. "${{HOME}}"/.smk_confs/postrun.sh
postrun.sh
## execute after running each of snakemake job
echo 'Running postrun.sh'
ERRMSG="MYJOB ID: $PBS_JOBID failed at $(pwd) on $(hostname) for ${USER} with exit status: ${exitstat}."
## ping to slack
ping_slack -m "$ERRMSG"
sleep 5
I am looking for a snakemake exported environment variable that I can use in postrun.sh to know which rule ID failed for one or more samples. For now, I can think of using job scheduler variable matching job name, e.g, PBS_JOBNAME, but better if I have a variable from snakemake API for rule ID or preferably, path to log file.
Thanks