# How are snakemake's --cluster and --drmaa options implemented?

I'm fairly new to snakemake and I'm trying to understand the difference between the --cluster and --drmaa flags, both of which are used to submit jobs to compute clusters/nodes.

The docs give a few hints about the advantages of using --drmaa here:

If your cluster system supports DRMAA, Snakemake can make use of that to increase the control over jobs. E.g. jobs can be cancelled upon pressing Ctrl+C, which is not possible with the generic --cluster support.

And here:

If available, DRMAA is preferable over the generic cluster modes because it provides better control and error handling.

So I have a conceptual understanding of the advantages of using --drmaa. However, I don't consider the above a very complete explanation, and I don't know how these flags are implemented in snakemake under the hood, can someone elaborate?

Note, that although this could be considered a more general programming question, snakemake is predominantly used in bioinformatics; I was persuaded that this question would be considered on-topic because of this meta-post and this answer.

• Hi Chris! What's the actual question here? Seems that DRMAA is an open API describing submission and control of cluster/grid jobs. So if you have a cluster or submission system that has support for it (such as SLURM), snakemake will afford you the additional controls as supported by that API -- including as per your question, the ability to pass on a Ctrl+C at your console to the grid -- a scenario that would require use of the qdel command on Sun Grid Engine, for example. So if you've infrastructure with a scheduler that has DRMAA support, I can see why you might as well use --drmaa. Jun 6 '17 at 15:18
• Maybe wait until we've actually poached the snakemake developers :) Jun 6 '17 at 19:01
• The code is open source and fairly easy to follow, I found it quite easy to work out how DRMAA jobs were submitted by nosing around the source for 10 minutes. I'm not entirely sure what the question is looking for, though. Jun 6 '17 at 19:26

I'd always kind of wondered how this worked too, so I took this as an excuse to look into the snakemake code. At the end of the day this becomes a question of (1) how are jobs actually submitted and (2) how is it determined if jobs are done (and then whether they failed)?

For DRMAA, python has a module (appropriately named "drmaa") that wraps around the libdrmaa library that comes with most schedulers. This is a very popular route, for example the Galaxy project uses this for dealing with most clusters (e.g., I use it to connect our internal Galaxy instance to our slurm cluster). The huge benefit here is that DRMAA does magic to allow you to simply submit commands to your cluster without needing to know whether you should execute qsub or srun or something else. Further, it then provides methods to simply query whether a job is running or not and what its exit status was.

Using the --cluster command requires a LOT more magic on the side of snakemake. At the end of the day it creates a shell script that it then submits using the command you provided. Importantly, it includes some secret files in that script that it can then watch for (ever noticed that .snakemake directory where you execute a script? That appears to be among the things it's used for.). These are named {jobid}.jobfinished and {jobid}.jobfailed and one of them will get touched, depending on the exit status of your command/script. Once one of those is there, then snakemake can move on in its DAG (or not, if there's a failure). This is obviously a LOT more to keep track of and it's then not possible for snakemake to cancel running jobs, which is something one can do easily with DRMAA.

• Having written a package for submitting jobs to Sun Grid Engine without DRMAA, can definitely confirm the magic must be strong here. Though instead of monitoring whether files have been touched, I chose to fire a POST request at the monitoring server. I can see that DRMAA must expose some nice API to get job status without having to do some odd workarounds. Jun 6 '17 at 19:42
• Yup, it's more convenient than what I would have come up with too :) Jun 6 '17 at 19:50

DRMAA (Distributed Resource Management Application API) appears to be an open API that describes a specification for submission and management of work submitted to some grid/cluster. If your scheduler is DRMAA compliant, my guess would be that using snakemake's --drmaa flag will afford you the additional controls exposed by that API.

As mentioned in your question with DRMAA support, a Ctrl+C on one's console would be passed on to the grid to kill or stop submitted jobs. A task that would otherwise require use of the qdel command on Sun Grid Engine, for example.

I could see why the suggestion would be to enable --drmaa where possible as it's likely to expose functionality (such as control of jobs, and availability of logs and errors) that makes the submission and management of jobs a little easier for you and snakemake. Perhaps someone with snakemake experience can explain a little more, but I hope this helps?

For what it's worth, I use Sun Grid Engine without DRMAA and it's quite a pain to get good information out of qstat, and its qacct log format is quite possibly the worst file format I have ever encountered. I suspect DRMAA provides a nice (or at least more reasonable) API for snakemake (and others) to get at this information more readily.

Appended:

Under the hood, the flags are parsed by Python's argparse module in the __init__.py. A mutually exclusive group of options forces a selection of --cluster or --drmaa (or --cluster-sync).

--cluster or -c requires you to specify an argument; the command for which to submit a job on your cluster (the example names qsub). The --drmaa flag just seems to indicate to snakemake that DRMAA is to be used, which doesn't change a lot in terms of how the command you run from your console is processed in the __init__.py or workflow.py.

However, when it is time to interface with your scheduler (as specified in scheduler.py), an elif statement checks whether you're in some form of cluster mode and the else catches the case where you've raised the --drmaa flag instead.

Here's where the magic happens, as now your job is submitted with the DRMAAExecutor as specified in executors.py, instead of the default executor.

Without investigating much more, I can see that the DRMAAExecutor features some more class attributes and exposes additional functions when compared to the others.

Hope this is of more use? Again, I've only just looked at the code myself but in lieu of a snakemake user chiming in, I thought I'd have a nose.

• Thanks, but this tells me what I already know; i have a conceptual understanding but i'd like the details on how the 2 different approaches are implemented in snakemake, i'll try to clarify the question a bit Jun 6 '17 at 15:36
• Sorry I pre-empted what you might have been asking for as I didn't have much more room to write in the comments. I'll try and expand to address your edited question if I can help! Jun 6 '17 at 15:37
• @Chris_Rands I've expanded, is this relevant? Jun 6 '17 at 16:05
• Thanks, upvoted, sorry if I wasn't very clear with my question, I just thought this might be useful question for others,i was planning on answering myself if i didn't get any answers, but i've not delved into the source yet Jun 6 '17 at 20:31
• @Chris_Rands No problem (and thanks)! I enjoy digging around code bases to see how they work. Between Devon and I, have we found what you were looking for? Happy to chat on this with you if not. Jun 6 '17 at 20:33