6
$\begingroup$

My goal is to process two samples, S1 and S2, using two rules, step1, step2. The Snakemake file is like:

SAMPLES = ['S1', 'S2']

rule all:
  input: expand("{sample}.done", sample = SAMPLES)

rule step1:
  input: "{sample}"
  output: "{sample}.step1.done"
  shell: "touch {output}"

rule step2:
  input: "{sample}.step1.done"
  output: "{sample}.done"
  shell: "touch {output}"

The current execution order is:

step1 on S1
step1 on S2
step2 on S1
step2 on S2

Is there a way to change the order to:

step1 on S1
step2 on S1
step1 on S2
step2 on S2

Basically, I want to complete S1, then S2. Is there a way to specify the order of executions in Snakemake?

$\endgroup$
2
$\begingroup$

I think you should use the priority directive to give precedence to downstream rule(s).

Here below I give higher priority to step2 so when the first sample has completed step1 snakemake will run step2 on that sample rather than submitting another sample to step1.

SAMPLES = ['S1', 'S2', 'S3']

wildcard_constraints:
    sample= '|'.join([re.escape(x) for x in SAMPLES])

rule all:
    input: expand("{sample}.done", sample = SAMPLES)

rule step1:
    priority: 1
    output: "{sample}.step1.done"
    shell: 
        r"""
        sleep 5
        touch {output}
        """

rule step2:
    priority: 10
    input: "{sample}.step1.done"
    output: "{sample}.done"
    shell: 
        r"""
        sleep 5
        touch {output}
        """

(Note that without wildcard_constraints your/this code throws AmbiguousRuleException)

$\endgroup$
0
$\begingroup$

The problem that you are encountering, is that when snakemake determines the order to run a set of rules it first builds a Directed Acyclic Graph (DAG) of all the rules. This shows which rules are are dependant on each other, and what order they can be safely executed in, without breaking downstream dependencies.

In general, if you want tasks to execute one after another which do not share an explicit file dependency, you can use a Flag File as an ouput for rule 1 then reference it as an input in rule 2.

Unfortunately, I do not think this will help you in your case. You are expanding on the SAMPLES list and creating a wildcard. in the DAG this creates two competely separate execution branches which have no dependencies with each other. step1 and step2 are guaranteed to occur in order for each sample, however there is no guarantees between samples.

Is there a specific reason why you need all of S2 tasks to be run after S1? If so, then your best bet would be to create explicit rules for sample 1 and sample 2, using the Flag File to incidate when step2 for S1 is done.

$\endgroup$
3
  • $\begingroup$ In typical analysis, we would like to see some results for the first sample, and make sure it is correct. So I would like to see S1 processed all the way first. Thanks for offering the Flag File feature. But it seems cumbersome to create them to ensure execution order in my case. $\endgroup$
    – zhanxw
    May 4 '20 at 15:54
  • $\begingroup$ @zhanxw your need seems to mostly center around QA of the pipeline, and making sure that its producing the right outputs. I often have to validate a pipeline before running 10s or 1000s of samples. The best way I found is to not try to gate the pipeline like you are suggesting, but just run it once with a single sample, then run it again with all of the remaining samples once you are satisfied. $\endgroup$ May 5 '20 at 0:11
  • $\begingroup$ I see your points: run one sample at a time. I can do that, but that is not convenient as I need to write another script to run snakemake on each sample. I am curious if snakemake has already some solutions. Thanks. $\endgroup$
    – zhanxw
    May 5 '20 at 3:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.