How to modify execution orders for Snakemake?

Question

My goal is to process two samples, S1 and S2, using two rules, step1, step2. The Snakemake file is like:

SAMPLES = ['S1', 'S2']

rule all:
  input: expand("{sample}.done", sample = SAMPLES)

rule step1:
  input: "{sample}"
  output: "{sample}.step1.done"
  shell: "touch {output}"

rule step2:
  input: "{sample}.step1.done"
  output: "{sample}.done"
  shell: "touch {output}"

The current execution order is:

step1 on S1
step1 on S2
step2 on S1
step2 on S2

Is there a way to change the order to:

step1 on S1
step2 on S1
step1 on S2
step2 on S2

Basically, I want to complete S1, then S2. Is there a way to specify the order of executions in Snakemake?

Answer 1

I think you should use the priority directive to give precedence to downstream rule(s).

Here below I give higher priority to step2 so when the first sample has completed step1 snakemake will run step2 on that sample rather than submitting another sample to step1.

SAMPLES = ['S1', 'S2', 'S3']

wildcard_constraints:
    sample= '|'.join([re.escape(x) for x in SAMPLES])

rule all:
    input: expand("{sample}.done", sample = SAMPLES)

rule step1:
    priority: 1
    output: "{sample}.step1.done"
    shell: 
        r"""
        sleep 5
        touch {output}
        """

rule step2:
    priority: 10
    input: "{sample}.step1.done"
    output: "{sample}.done"
    shell: 
        r"""
        sleep 5
        touch {output}
        """

(Note that without wildcard_constraints your/this code throws AmbiguousRuleException)

Answer 2

it seems cumbersome to create them to ensure execution order in my case

It took me a while to figure out, but it did not seem cumbersome to me.

You can re-use your rules, see below, and make Step 1 of S2 depend on Step 2 of S1 like so:

SAMPLES = ['S1', 'S2']


wildcard_constraints:
    sample = "S[12]"

rule all:
  input: expand("{sample}.done", sample = SAMPLES)



rule step1:
  input: data="{sample}"
  output: data="{sample}.step1.done"
  shell: "touch {output.data}"

rule step2:
  input: data="{sample}.step1.done"
  output: data="{sample}.done"
  shell: "touch {output.data}"


# step2 of S1 has the flag file as output

use rule step2 as step2ForS1 with:
    input: data="S1.step1.done"
    output: data="S1.done", flag=touch("flags/step2-s1-done")

# step1 of S2 needs the flag file as input

use rule step1 as step1ForS2 with:
    input: data="S2", flag="flags/step2-s1-done"
    output: data="S2.step1.done"

Answer 3

The problem that you are encountering, is that when snakemake determines the order to run a set of rules it first builds a Directed Acyclic Graph (DAG) of all the rules. This shows which rules are are dependant on each other, and what order they can be safely executed in, without breaking downstream dependencies.

In general, if you want tasks to execute one after another which do not share an explicit file dependency, you can use a Flag File as an ouput for rule 1 then reference it as an input in rule 2.

Unfortunately, I do not think this will help you in your case. You are expanding on the SAMPLES list and creating a wildcard. in the DAG this creates two competely separate execution branches which have no dependencies with each other. step1 and step2 are guaranteed to occur in order for each sample, however there is no guarantees between samples.

Is there a specific reason why you need all of S2 tasks to be run after S1? If so, then your best bet would be to create explicit rules for sample 1 and sample 2, using the Flag File to incidate when step2 for S1 is done.