I have some snakemake pipelines that have rules which run in different environments. Is it possible to run the pipeline without having to rebuild the environments from scratch every time when run in a different directory?
This is the way I would normally run snakemake:
snakemake --directory <my_output_dir> --use-conda
My snakefile might have rules that need to use different environments:
rule A: .... conda: envA.yml .... rule B: .... conda: envB.yml ....
So if I wanted to do two different analyses with the same pipeline, is there anyway that I can point them to the same environments without having to install everything separately in each output directory?
The snakemake documentation makes it seem like they don't currently support this. You can either use 1) --conda-prefix when running snakemake and then point to a single env or 2) use the --use-conda directive and the env.yml files by rule, in which case it will create new envs any time you run it in a new output directory. So my case doesn't seem to be covered, but does anyone have a workaround to suggest? I often find conflicts between some of the R packages I use and python packages, so I tend to keep them in their own environments to avoid headache.
I'm starting to get neck-deep in conda environments at this point which is taking up space from my disk quota, but also I am bothered by the fact that I may actually want to run analysis of data from different projects with the same environments. If the environments are rebuilt at each execution, I do not know that they are the same, short of specifying the exact version info for everything I use in the env.yaml files (and even then, they could be slightly different if the repo has changed).
I have looked at this similar question: Running Snakemake in one single conda env in which it is suggested there is a way to do such a thing, but I'm afraid I don't understand how to implement it without an example.