1
$\begingroup$

I have some snakemake pipelines that have rules which run in different environments. Is it possible to run the pipeline without having to rebuild the environments from scratch every time when run in a different directory?

This is the way I would normally run snakemake:

snakemake --directory <my_output_dir> --use-conda

My snakefile might have rules that need to use different environments:

rule A:
   ....
   conda: envA.yml
   ....

rule B:
   ....
   conda: envB.yml
   ....

So if I wanted to do two different analyses with the same pipeline, is there anyway that I can point them to the same environments without having to install everything separately in each output directory?

The snakemake documentation makes it seem like they don't currently support this. You can either use 1) --conda-prefix when running snakemake and then point to a single env or 2) use the --use-conda directive and the env.yml files by rule, in which case it will create new envs any time you run it in a new output directory. So my case doesn't seem to be covered, but does anyone have a workaround to suggest? I often find conflicts between some of the R packages I use and python packages, so I tend to keep them in their own environments to avoid headache.

I'm starting to get neck-deep in conda environments at this point which is taking up space from my disk quota, but also I am bothered by the fact that I may actually want to run analysis of data from different projects with the same environments. If the environments are rebuilt at each execution, I do not know that they are the same, short of specifying the exact version info for everything I use in the env.yaml files (and even then, they could be slightly different if the repo has changed).

I have looked at this similar question: Running Snakemake in one single conda env in which it is suggested there is a way to do such a thing, but I'm afraid I don't understand how to implement it without an example.

Thanks

$\endgroup$

2 Answers 2

3
$\begingroup$

Each YAML file will always create its own environment. The packages in each aren't downloaded and installed each time, if they're already installed elsewhere they're just hard-linked in. You'll want to use both --use-conda and --conda-prefix. The latter will point to the base directory in which all conda environments will be stored. This has the benefit that running the same workflow in different places won't result in a new env being created (by default snakemake will do so inside a .snakemake directory otherwise).

$\endgroup$
1
  • $\begingroup$ Wow, thank you. I had no idea that these two flags could be used together (I thought --conda-prefix could only be used with a named pre-built single environment) -- that solves the problem. I see what you mean about the hard-linking but it still ends up looking quite messy if you have all these .snakemake directories everywhere and it also seems to slow down some conda commands (like conda env list), at least for me. $\endgroup$
    – vantom
    Apr 30, 2020 at 20:31
0
$\begingroup$

@vantom, while I am not very familiar with snakemake on a whole, I do have extensive knowledge writing workflows in WDL as well as CWL. The approach both of these languages take to overcome a similar issue that you are describing, is to use containerization through ie docker. This approach allows you to bundle the specific tools you need for a specific task, making different images for different tasks. There are several large repositories of images online ie: Docker Hub, or more biologically focused: Biocontainers

Looking through the documentation, it does appear that snakemake supports running your rule from within the context of a docker image. Without knowing how your compute environment is setup, I am not sure if there is any additional configuration you would need, however the documentation seems to suggest that it should be as easy as describing the name of the container. As long as you can run docker, (or an alternative like singularity) you should be able to avoid the current headache you are facing.

rule NAME:
    input:
        "table.txt"
    output:
        "plots/myplot.pdf"
    containers:
        "docker://joseespinosa/docker-r-ggplot2"
    script:
        "scripts/plot-stuff.R"
$\endgroup$
2
  • $\begingroup$ Thank you for the tip! I'm not currently using Docker, but it's useful to know this for the future. $\endgroup$
    – vantom
    Apr 30, 2020 at 20:33
  • $\begingroup$ For reproducibility, you should always used a tagged version of a container, that is: docker://joseespinosa/docker-r-ggplot2:v0.1 If you don't it will pull/build the :latest tag and you run the risk your pipeline could break on an update. See the tags menu in DockerHub; e.g. hub.docker.com/r/joseespinosa/docker-r-ggplot2/tags $\endgroup$ Sep 29, 2020 at 2:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.