3
$\begingroup$

I have the following two rules:

rule extract:
    input: "data/{dir}/{file}.gz"
    output: "data/{dir}/{file,^.*(?<!\.gz)$}"
    conda: "config/conda-extract-env.yml"
    shell: "cd 'data/{wildcards.dir}'; gunzip -k {wildcards.file}.gz"

rule extract_dot:
    input: "data/{dir}/wtdbg2.3.dot.gz"
    output: "data/{dir}/wtdbg2.3.dot"
    conda: "config/conda-extract-env.yml"
    shell: "cd 'data/{wildcards.dir}'; gunzip -k wtdbg2.3.dot.gz"

The second one is actually a hack because the first does not work on files named wtdbg2.3.dot. I have also tried giving the constraint using the wildcard_constraints member of the rule, and prefixed the string with an r or not.

Whatever I do, as soon as I uncomment the extract_dot rule, snakemake complains about the missing input file data/C.elegans/wtdbg2.3.dot in another rule. Note that there is a rule with output data/C.elegans/wtdbg2.3.dot.gz, and when changing this rule to output data/C.elegans/wtdbg2.3.dot instead, everything works as well.

So I conclude that the problem is in the regex. I have tested the regex on regex101.com, and it accepts wtdbg2.3.dot. But snakemake seems not to. Does anyone know if there is anything special about the snakemake regexes? Should they not be normal python regexes?

$\endgroup$
2
$\begingroup$

Does anyone know if there is anything special about the snakemake regexes? Should they not be normal python regexes?

No, nothing special. The Snakemake documentation just links to the Python re module documentation.

I think this is something strange in how the anchors ^ and $ behave when matching wildcards, and not anything else in your particular regular expression. Check out the pattern-matching with this dummy Snakefile:

rule all:
    input: ["data/subdir/example", "data/example", "example"]

rule anchored_example_of_subdir:
    """Can't add either ^ or $ or this will fail"""
    output: "data/{subdir,subdir}/example"

rule anchored_example_in_subdir:
    """Can't add ^ or this wil fail"""
    output: "data/{file,example$}"

rule anchored_example:
    """This works with both ^ and $"""
output: "{file,^example$}"

(There are no inputs or actions so it's just a matter of what filenames match what rules, and the regular expressions do nothing but match exact strings.) This "works" as is (snakemake -nrp shows what you expect) but if you add in anchors as described in the docstrings above the upper two rules no longer match the expected filenames. It's as if (and maybe is?) under the hood Snakemake is trying to apply the expression to the entire path.

Can you just drop your ^ and $? The snakemake documentation implies that it's using the equivalent of re.match rather than re.search already, with wording like "Here we could restrict the wildcard dataset to consist of digits only using \d+ as the corresponding regular expression" so I don't think it's necessary anyway.


Edit: I think I found the issue in the snakemake source. It munges your constraints together into one larger regular expression for the entire file path, so ^ and $ get embedded inside the resulting expression. This looks like something that should at least be clarified in the documentation.

$\endgroup$
1
  • $\begingroup$ Thank you for your answer! I will report this as an issue then. $\endgroup$ Dec 4 '20 at 6:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.