1
$\begingroup$

I'm familiar with running Snakemake in a DRMAA-enabled grid environment. (Most often, I write Python programs as wrappers and invoke the workflow via Snakemake's Python API.) Typically, I'm running workflows with rules that perform large, multi-threaded tasks. The rules are usually designed to utilize all available processors on a node, and only a single task is executed on any given node at a time.

But what if I have thousands of single-threaded tasks that I want to distribute among the requested grid nodes? If the corresponding rule(s) are marked threads: 1, is Snakemake smart enough to run multiple tasks on each node? How does it know how many tasks can be run on an individual node—or how does the user specify a limit?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.