How are snakemake's --cluster and --drmaa options implemented?

Question

I'm fairly new to snakemake and I'm trying to understand the difference between the --cluster and --drmaa flags, both of which are used to submit jobs to compute clusters/nodes.

The docs give a few hints about the advantages of using --drmaa here:

If your cluster system supports DRMAA, Snakemake can make use of that to increase the control over jobs. E.g. jobs can be cancelled upon pressing Ctrl+C, which is not possible with the generic --cluster support.

And here:

If available, DRMAA is preferable over the generic cluster modes because it provides better control and error handling.

So I have a conceptual understanding of the advantages of using --drmaa. However, I don't consider the above a very complete explanation, and I don't know how these flags are implemented in snakemake under the hood, can someone elaborate?

Note, that although this could be considered a more general programming question, snakemake is predominantly used in bioinformatics; I was persuaded that this question would be considered on-topic because of this meta-post and this answer.

Answer 1

I'd always kind of wondered how this worked too, so I took this as an excuse to look into the snakemake code. At the end of the day this becomes a question of (1) how are jobs actually submitted and (2) how is it determined if jobs are done (and then whether they failed)?

For DRMAA, python has a module (appropriately named "drmaa") that wraps around the libdrmaa library that comes with most schedulers. This is a very popular route, for example the Galaxy project uses this for dealing with most clusters (e.g., I use it to connect our internal Galaxy instance to our slurm cluster). The huge benefit here is that DRMAA does magic to allow you to simply submit commands to your cluster without needing to know whether you should execute qsub or srun or something else. Further, it then provides methods to simply query whether a job is running or not and what its exit status was.

Using the --cluster command requires a LOT more magic on the side of snakemake. At the end of the day it creates a shell script that it then submits using the command you provided. Importantly, it includes some secret files in that script that it can then watch for (ever noticed that .snakemake directory where you execute a script? That appears to be among the things it's used for.). These are named {jobid}.jobfinished and {jobid}.jobfailed and one of them will get touched, depending on the exit status of your command/script. Once one of those is there, then snakemake can move on in its DAG (or not, if there's a failure). This is obviously a LOT more to keep track of and it's then not possible for snakemake to cancel running jobs, which is something one can do easily with DRMAA.

Answer 2

DRMAA (Distributed Resource Management Application API) appears to be an open API that describes a specification for submission and management of work submitted to some grid/cluster. If your scheduler is DRMAA compliant, my guess would be that using snakemake's --drmaa flag will afford you the additional controls exposed by that API.

As mentioned in your question with DRMAA support, a Ctrl+C on one's console would be passed on to the grid to kill or stop submitted jobs. A task that would otherwise require use of the qdel command on Sun Grid Engine, for example.

I could see why the suggestion would be to enable --drmaa where possible as it's likely to expose functionality (such as control of jobs, and availability of logs and errors) that makes the submission and management of jobs a little easier for you and snakemake. Perhaps someone with snakemake experience can explain a little more, but I hope this helps?

For what it's worth, I use Sun Grid Engine without DRMAA and it's quite a pain to get good information out of qstat, and its qacct log format is quite possibly the worst file format I have ever encountered. I suspect DRMAA provides a nice (or at least more reasonable) API for snakemake (and others) to get at this information more readily.

Appended:

Under the hood, the flags are parsed by Python's argparse module in the __init__.py. A mutually exclusive group of options forces a selection of --cluster or --drmaa (or --cluster-sync).

--cluster or -c requires you to specify an argument; the command for which to submit a job on your cluster (the example names qsub). The --drmaa flag just seems to indicate to snakemake that DRMAA is to be used, which doesn't change a lot in terms of how the command you run from your console is processed in the __init__.py or workflow.py.

However, when it is time to interface with your scheduler (as specified in scheduler.py), an elif statement checks whether you're in some form of cluster mode and the else catches the case where you've raised the --drmaa flag instead.

Here's where the magic happens, as now your job is submitted with the DRMAAExecutor as specified in executors.py, instead of the default executor.

Without investigating much more, I can see that the DRMAAExecutor features some more class attributes and exposes additional functions when compared to the others.

Hope this is of more use? Again, I've only just looked at the code myself but in lieu of a snakemake user chiming in, I thought I'd have a nose.