1
$\begingroup$

When a rule fails in snakemake workflow, it returns a descriptive message in log file like following:

Error in rule step1:
    jobid: 99
    output: results/foo/step1/sample5/outdir/output1.tsv, results/foo/step1/sample5/outdir/output2.tsv
    log: logs/step1/sample5_out.log (check log file(s) for error message)
    shell:
    ...
        (exited with non-zero exit code)

Since I run workflow over many samples, I have defined a postrun.sh bash script in jobscript.sh directive that runs after snakemake {exec_job} section and notifies me via userbot in slack channel when a step failed for one or more samples.

jobscript.sh

        ########################### START SNAKEMAKE CMD ############################
    {exec_job}
        ############################ END SNAKEMAKE CMD #############################
    exitstat=$?
    export exitstat

    echo "END at $(date)"
. "${{HOME}}"/.smk_confs/postrun.sh

postrun.sh

## execute after running each of snakemake job
echo 'Running postrun.sh'
ERRMSG="MYJOB ID: $PBS_JOBID failed at $(pwd) on $(hostname) for ${USER} with exit status: ${exitstat}."

## ping to slack
ping_slack -m "$ERRMSG"
sleep 5

I am looking for a snakemake exported environment variable that I can use in postrun.sh to know which rule ID failed for one or more samples. For now, I can think of using job scheduler variable matching job name, e.g, PBS_JOBNAME, but better if I have a variable from snakemake API for rule ID or preferably, path to log file.

Thanks

$\endgroup$
1
$\begingroup$

You could add an exit trap to the beginning of the shell scripts inside your Snakefile, which is executed after each script is done and sends a message when there was an error:

 shell:
    """
    trap 'if [ $? -neq 0 ]; then ping_slack -m "Error in rule xxx with input {input}"; exit 1' EXIT
    ...
|improve this answer|||||
$\endgroup$
  • $\begingroup$ I believe this needs to be added to all of workflow rules. I am hoping to set a trap similar to existing onerror handler of snakemake but prefer to execute it at completion of each rule rather an entire workflow. $\endgroup$ – Samir Aug 21 '19 at 15:30
  • $\begingroup$ yes, that would be needed in each rule. $\endgroup$ – Peter Menzel Aug 22 '19 at 5:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.