9

The problem is that you need a master rule that requires all of your desired outputs as inputs, in your case it would be : rule all: input: expand("data/{sp}/genome.fa.gz", sp=species.split(' ')) You'll also need separate download link inputs for each species. You could make a separate download_table.tsv for each species, but it would probably ...


9

You can omit --nodes, you need the following: #!/bin/bash #SBATCH --ntasks-per-node=1 #SBATCH -c threads_from_snakemake #SBATCH -p some_partition Commands go here For slurm, you may want to modify my SlurmEasy script. We use that with snakemake all the time, usually of the form: snakemake -c "SlurmEasy -t {threads} -n {rule} --mem-per-cpu {cluster.memory}...


8

you are starting snakemake the wrong way. snakemake snakemake means "Running the pipeline in Snakefile until you reach a rule named snakemake or a file name snakemake is created." If your snakefile is called snakemake and this is the pipeline you want to start, the correct syntax would be: snakemake -s snakemake


7

I think you are looking for --rulegraph instead of --dag --rulegraph Do not execute anything and print the dependency graph of rules in the dot language. This will be less crowded than above DAG of jobs, but also show less information. Note that each rule is displayed once, ...


5

This feature was added in version 3.6.0. Look at release notes. {wildcards} normally expands to a something that looks vaguely like a dictionary. In other words, something like key1=value1,key2=value2. Whether you really want to use that as a log file name is up to you. Let's use a very contrived example to demonstrate this. Here is the Snakefile: rule ...


5

I'd always kind of wondered how this worked too, so I took this as an excuse to look into the snakemake code. At the end of the day this becomes a question of (1) how are jobs actually submitted and (2) how is it determined if jobs are done (and then whether they failed)? For DRMAA, python has a module (appropriately named "drmaa") that wraps around the ...


5

A good practice is to provide yml file along with your Snakefile. This file should contain all the enviroment definitions that are needed to run your pipeline. The minimum content of this file would be to install snakemake itself. Any other program used in your pipeline should be included as well. For reproducibility version numbers must be used. So, if ...


5

In your submission script you have -n {-cluster.n}, hence the error from snakemake I guess. Are you sure it is not a typo and it should be -n {cluster.n} instead?


5

Snakemake gets wildcards from the parsing the input/output file names. For example, if you had rule sort: input: "{file}.bed" output: "{file}.sorted.bed" shell: "sort -k1,1 -k2,2n {input} > {output}" then the wildcards variable would be set as wildcards = {"file": ...}. In your multiqc rule, you have the {sample} variable within your expand(...


4

Using only the --use-conda directive, snakemake copies environments somewhere in the local .snakemake directory. This is repeated for each directory you run it in and replicates all environments effectively for each dataset. You can however tell snakemake to use a shared environment location using --conda-prefix as described here: Specify a directory in ...


4

As noted by the comments and the SLURM manual (not so much bioinformatics related?) You need the following terms in your batch/queuing command: --mail-type=ALL --mail-user=you@email.com I used "ALL" which will send you all reports. If you just want the finish message then you would use "END" and "FAIL" like such: --mail-type=END,FAIL Here are the ...


4

Could you give an example of names of input files and output files of normalize.py ? It seems to me that your case could be similar to the one described about the dynamic output: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#dynamic-files Here you would probably use it for the output of normalization and for the input of all (with a ...


4

DRMAA (Distributed Resource Management Application API) appears to be an open API that describes a specification for submission and management of work submitted to some grid/cluster. If your scheduler is DRMAA compliant, my guess would be that using snakemake's --drmaa flag will afford you the additional controls exposed by that API. As mentioned in your ...


4

Python (whether installed with conda or not) will prefer packages installed in your home directory unless you tell it not to. This is a little known secret about using a virtualenv, actually, since internally those tell python to ignore everything in your home directory. Within conda, there's long been a debate about whether python (and also R) should ...


4

This is an example of the conda solver behaving incorrectly. In this case, it's choosing to pick a version that minimizes the number of dependencies required. Version 5.3.0 has 25 fewer dependencies, so if you let conda choose the version that's what it will pick. There's nothing you can really do about this other than specifying the version of snakemake ...


4

You can run CI on a snakemake workflow when provoding a minimal example dataset. Maybe an example from the snakemake-workflows repo will be helpful: https://github.com/snakemake-workflows/dna-seq-varlociraptor


3

Given the Makefile you provided and which someone then deleted from the question, you should probably just add this line (this should be the first rule): rule all: input: ["data/{}/genome.fa.gz".format(x) for x in species.split()] This rule just specifies a list of expected output files and corresponds to the following lines from the original Makefile: ...


3

The following (or something close) should work: rule expression: input: "Align/{sample}/Aligned.toTranscriptome.out.bam" params: rsem_index = RSEM_INDEX output: "Expression/{sample}/rsem.genes.results" threads: 8 shell: """ rsem-calculate-expression --estimate-rspd --bam --no-bam-output -p {...


3

I'll fix the bamCoverage issue in the 2.6.0 release (it may already be fixed since I rewrite how bigWig files are made for that release). You can follow my status on the issue page. In principle you should be able to do this, it's just a deepTools limitation. This should work fine with computeMatrix, though keep in mind that you're just going to get a big ...


3

The solutions are either, upgrade snakemake or just take all wildcards out of cluster.json file and write them directly to snakemake execution command : snakemake -p --jobs 10 --cluster-config cluster.json --cluster "bsub -J "{rule}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host} -o {"logs/cluster/{rule}.{wildcards}.out" -e "logs/...


3

Each YAML file will always create its own environment. The packages in each aren't downloaded and installed each time, if they're already installed elsewhere they're just hard-linked in. You'll want to use both --use-conda and --conda-prefix. The latter will point to the base directory in which all conda environments will be stored. This has the benefit that ...


2

It seems that it is not possible to apply strip() directly on the wildcard placeholders. Luckily it is possible to executed functions in the input directive. So try changing your input of the complexity_20mer_counter rule to this: lambda wc: os.path.join(fastq_trimmed_dir, wc.sample + '_' + wc.read + fastq_extension.split('.')[0] + '_val_' + wc.read.strip(...


2

To quote from the snakemake documentation All paths in the snakefile are interpreted relative to the directory snakemake is executed in. This behaviour can be overridden by specifying a workdir in the snakefile: workdir: "path/to/workdir" Usually, it is preferred to only set the working directory via the command line, because above directive ...


2

A nice solution comes from a colleague of mine, as seen in this Snakemake workflow. The trick is to access the wildcards programmatically using an anonymous (lambda) function. In my example above, it would be implemented as follows. rule preprocess: input: lambda wildcards: glob('sample-{samp}/*.fastq'.format(samp=wildcards.samp)) output: '{samp}-...


2

I created a file .condarc in my home directory with the following content: channels: - conda-forge - bioconda - defaults and it works now. I don't understand why this is necessary though. I thought the channels in the .condarc file would be used if I did not specify them in my command.


2

You could query Python for the working directory within the Snakefile, since evidently Snakemake changes the actual working directory of the process to the one specified by --directory. For example, using a stub Snakefile with just these two lines: import os print("Current working directory: " + os.getcwd()) And on the command line: $ snakemake --...


2

If the environment already exists it will simply be used. You cannot, however, tell it to use an environment with a normal name (e.g., "my_env"). I assume you mean, "where the heck does the random looking conda environment name come from?!?", which is a very good question. The answer to that is that it's a hash of the --conda-prefix setting and the contents ...


2

I think you should use the priority directive to give precedence to downstream rule(s). Here below I give higher priority to step2 so when the first sample has completed step1 snakemake will run step2 on that sample rather than submitting another sample to step1. SAMPLES = ['S1', 'S2', 'S3'] wildcard_constraints: sample= '|'.join([re.escape(x) for x ...


2

The latest version solves the issue. Source : https://github.com/davidemms/OrthoFinder/issues/403 Latest Version: https://github.com/davidemms/OrthoFinder/releases/tag/2.4.0


2

Does anyone know if there is anything special about the snakemake regexes? Should they not be normal python regexes? No, nothing special. The Snakemake documentation just links to the Python re module documentation. I think this is something strange in how the anchors ^ and $ behave when matching wildcards, and not anything else in your particular regular ...


Only top voted, non community-wiki answers of a minimum length are eligible