Snakemake: Accept any of a possible set of inputs (any one being sufficient)

Question

I have a little pipeline that I want to execute using Snakemake.

One of my rules (let's call it create_somefile) creates a file somefile_{n_dims}.json, and when running the pipeline can specify that I want to run the pipeline for a pre-defined set of values for n_dims, which is why I specified this as a wildcard.

Now I want to have another rule, that takes as input any of these files. It can take either somefile_1.json or somefile_2.json or somefile_32402348.json, it doesn't matter which one as long as any of these exists. The rule then creates a file that does not depend on the number of dimensions, and thus doesn't want and need the n_dims in its output.

When I try to have the wildcard only in the input, like this:

rule do_stuff:
    input:
        "somefile_{n_dims}.json"
    output:
        "general_file.json"
    shell:
        "echo {input} &&"
        "touch {output}"

Snakemake gives me the error Wildcards in input files cannot be determined from output files: n_dims. I don't want the output file to contain n_dims whatsoever, and unfortunately it's also impossible in snakemake to just create an additional throwaway temporary(..)-output that does depend on the n_dims just so snakemake is happy (error message then: ot all output, log and benchmark files of rule do_stuff contain the same wildcards. This is crucial though, in order to avoid that two or more jobs write to the same file..

Is there any way in snakemake to specify a rule's inputs as "take any of these", without wildcards?

Looking at the documentation, the only thing I found is the concept of checkpoints (https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#data-dependent-conditional-execution), however that doesn't seem to work as well - I tried making the rule create_somefile a checkpoint and changing the do_stuff rule to

rule do_stuff:
    input:
        lambda wildcards: checkpoints.create_somefile.get(**wildcards).output
    output:
        "general_file.json",
    shell:
        "echo {input} &&"
        "touch {output}"

however then I get the error WorkflowError: Missing wildcard values for n_dims.

Is there any concept in snakemake to allow me to do what I want? To have "wildcards" in inputs, that just stand for "execute this rule as soon as the input for any value of the wildcard is fulfilled, execute only once and be done with it"?

Answer 1

I don't think Snakemake can follow a "use any of these" pattern with a unified dependency graph and without additional assumptions, but you can do it if you break the workflow into two pieces.

This will take whatever somefile_*.json shows up first, or throw an exception if there aren't any:

from pathlib import Path

def input_for_do_stuff(wildcards):
    try:
        return next(Path(".").glob("somefile_*.json"))
    except StopIteration as err:
        raise Exception("No n_dims JSON files found") from err

rule do_stuff:
    input: input_for_do_stuff
    output:
        "general_file.json",
    shell:
        "echo {input} &&"
        "touch {output}"

rule create_somefile:
    output: "somefile_{n_dims}.json"
    shell: "touch {output}"

For example if you have somefile_2.json already there:

$ snakemake -nrp
Building DAG of jobs...
Job stats:
job         count    min threads    max threads
--------  -------  -------------  -------------
do_stuff        1              1              1
total           1              1              1


[Sun Oct 10 12:22:14 2021]
rule do_stuff:
    input: somefile_2.json
    output: general_file.json
    jobid: 0
    reason: Missing output files: general_file.json
    resources: tmpdir=/tmp

echo somefile_2.json &&touch general_file.json
Job stats:
job         count    min threads    max threads
--------  -------  -------------  -------------
do_stuff        1              1              1
total           1              1              1

This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

With that, you'll never be able to run the workflow by just requesting the final output, which kind of breaks Snakemake's way of thinking about things. But at least it'll work like you want if you run it in two pieces. If having a default makes sense for your case, you could also do something like this:

def input_for_do_stuff(wildcards):
    try:
        return next(Path(".").glob("somefile_*.json"))
    except StopIteration as err:
        return "somefile_1.json"

That way, even if no files exist to start with, Snakemake can come up with a way to make the output you ask for.