4
$\begingroup$

I'm trying to use snakemake to replace some make workflows, but I'm finding that snakemake does not use the file modification time to trigger a rule.

Here is my snakefile

rule sample:
    input:
        "a"
    output:
        "b"
    shell:
        "touch {output}"

If I create a then run snakemake, b is created as expected:

$ touch a
$ snakemake -c1 b
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job      count    min threads    max threads
-----  -------  -------------  -------------
sample       1              1              1
total        1              1              1

Select jobs to execute...

[Sat Oct 29 14:36:43 2022]
rule sample:
    input: a
    output: b
    jobid: 0
    reason: Missing output files: b
    resources: tmpdir=/tmp

[Sat Oct 29 14:36:43 2022]
Finished job 0.
1 of 1 steps (100%) done
Complete log: .snakemake/log/2022-10-29T143643.334050.snakemake.log
$ ls -l a b
-rw-r--r-- 1 ... ... 0 Oct 29 14:36 a
-rw-r--r-- 1 ... ... 0 Oct 29 14:36 b

Now, if I touch a then ask snakemake to update b, nothing happens:

$ touch a
$ snakemake -c1 b
Building DAG of jobs...
Nothing to be done (all requested files are present and up to date).
Complete log: .snakemake/log/2022-10-29T144005.589943.snakemake.log
$ ls -l a b
-rw-r--r-- 1 ... ... 0 Oct 29 14:40 a
-rw-r--r-- 1 ... ... 0 Oct 29 14:36 b

Modifying the contents of a triggers an update as expected:

$ echo new > a
$ snakemake -c1 b
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job      count    min threads    max threads
-----  -------  -------------  -------------
sample       1              1              1
total        1              1              1

Select jobs to execute...

[Sat Oct 29 14:42:13 2022]
rule sample:
    input: a
    output: b
    jobid: 0
    reason: Updated input files: a
    resources: tmpdir=/tmp

[Sat Oct 29 14:42:13 2022]
Finished job 0.
1 of 1 steps (100%) done
Complete log: .snakemake/log/2022-10-29T144213.161985.snakemake.log
$ ls -l a b
-rw-r--r-- 1 ... ... 4 Oct 29 14:42 a
-rw-r--r-- 1 ... ... 0 Oct 29 14:42 b

I installed snakemake-minimal (version 7.17.1) from bioconda, am using python 3.9.13 from conda-forge, on a RHEL 7 system.

$ snakemake --version
7.17.1
$ python --version
Python 3.9.13
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.8 (Maipo)

I'm clearly misunderstanding something very basic about snakemake. Any suggestions on how to debug this, descriptions of how to have snakemake pay attention to file modification times, or just an explanation of why snakemake behaves differently from make would be much appreciated!

thanks in advance,

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

This is a Snakemake feature: For small files (less than 100000 bytes), checksums are compared to determine if a job needs to run if the input file is newer than output file:

https://github.com/snakemake/snakemake/pull/1568

With a size of 99999 bytes:

$ dd if=/dev/zero of=a bs=99999 count=1
1+0 records in
1+0 records out
99999 bytes (100 kB, 98 KiB) copied, 0.000502722 s, 199 MB/s
$ snakemake -c1 b
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job       count    min threads    max threads
------  -------  -------------  -------------
sample        1              1              1
total         1              1              1

Select jobs to execute...

[Mon Oct 31 14:38:00 2022]
rule sample:
    input: a
    output: b
    jobid: 0
    reason: Missing output files: b
    resources: tmpdir=/tmp

[Mon Oct 31 14:38:00 2022]
Finished job 0.
1 of 1 steps (100%) done
Complete log: .snakemake/log/2022-10-31T143800.486966.snakemake.log
$ dd if=/dev/zero of=a bs=99999 count=1
1+0 records in
1+0 records out
99999 bytes (100 kB, 98 KiB) copied, 0.000180857 s, 553 MB/s
$ snakemake -c1 b
Building DAG of jobs...
Nothing to be done (all requested files are present and up to date).
Complete log: .snakemake/log/2022-10-31T143813.285988.snakemake.log

With a size of 100000 bytes:

$ dd if=/dev/zero of=a bs=100000 count=1
1+0 records in
1+0 records out
100000 bytes (100 kB, 98 KiB) copied, 0.000173008 s, 578 MB/s
$ snakemake -c1 b
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job       count    min threads    max threads
------  -------  -------------  -------------
sample        1              1              1
total         1              1              1

Select jobs to execute...

[Mon Oct 31 14:41:14 2022]
rule sample:
    input: a
    output: b
    jobid: 0
    reason: Missing output files: b
    resources: tmpdir=/tmp

[Mon Oct 31 14:41:14 2022]
Finished job 0.
1 of 1 steps (100%) done
Complete log: .snakemake/log/2022-10-31T144114.211255.snakemake.log
$ dd if=/dev/zero of=a bs=100000 count=1
1+0 records in
1+0 records out
100000 bytes (100 kB, 98 KiB) copied, 5.35361 s, 18.7 kB/s
$ snakemake -c1 b
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job       count    min threads    max threads
------  -------  -------------  -------------
sample        1              1              1
total         1              1              1

Select jobs to execute...

[Mon Oct 31 14:41:26 2022]
rule sample:
    input: a
    output: b
    jobid: 0
    reason: Updated input files: a
    resources: tmpdir=/tmp

[Mon Oct 31 14:41:26 2022]
Finished job 0.
1 of 1 steps (100%) done
Complete log: .snakemake/log/2022-10-31T144125.949828.snakemake.log
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks! Do you know if there is a another mechanism that snakemake uses for "flag files" (snakemake.readthedocs.io/en/stable/snakefiles/…)? $\endgroup$
    – Jules B
    Nov 4, 2022 at 1:01
  • 1
    $\begingroup$ @JulesB Looks like they're just touched or created (see handle_touch()). Then touch() basically just calls os.utime() or touch --no-dereference to set the file access and modified times to the current time. $\endgroup$
    – Steve
    Nov 4, 2022 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.