Bioinformatics Stack Exchange is a question and answer site for researchers, developers, students, teachers, and end users interested in bioinformatics. It only takes a minute to sign up.
Sign up to join this community
I'm going through an RNA-seq pipeline in R/Bioconductor and want to try multiple parameters at subsequent steps, for example, running clustering with different settings, running RegressOut or not on unwanted effects etc. That's a lot of "versions", even if I don't do combinations of these steps.
How can I keep track of this, and my conclusions? Not necessarily want to save the results.
The main purpose of git is to version code, which usually means sequential improvement of the codebase. While it is possible to use branches for multiple variants of the software, permanent branches are traditionally used for gradual integration of new features (i.e. dev/testing/master branches). Supporting multiple independent branches requires some investment, i.e. distributing common changes among branches via merge or cherry-pick. This is hard to manage when you have more than two-three branches.
If you compare different methods of analysis, you probably want to compare the results between methods. Having them on separate branches makes it hard.
In my opinion you should integrate all methods of analysis into the master branch. To avoid copy & paste, it is better to put common code in a library or an independent script. You can also specify a method as a run-time parameter of your pipeline, and create a meta-script which will execute all methods of interests.
Once you performed benchmarking, you shouldn't remove unused methods from you master branch. Having them is important for reproducible research, and your scripts could be used in the future for new datasets.
Save the different scripts with git (seems overkill)
Whoa. I did an actual double take when reading this:1 it’s the opposite of overkill.
Version controlling your scripts (using Git or something similar) is the absolute minimum, and should become completely automatic. For every new project I begin, one of the very first steps is to issue the git init command, and to set up a remote repository (on Github).
To keep track of different analyses, I use a combination of the following approaches:
In fact, I recommend creating a declarative workflow script for every analysis; I have found that this is the most practical, self-documenting way to run an analysis. It most closely resembles a wet-lab lab notebook. The advantage over a single R Markdown document is that rerunning just parts of the analysis can be completely automated, and dependencies in the workflow are apparent from the dependencies of the workflow rules. This is much harder in R Markdown.
Some time ago I create an example analysis workflow to show how this can be structured.
Regarding your other point:
Make notes in the script itself
“Notes” are a dangerous beasts: documentation is important, but experience shows that it’s sometimes very hard to keep documentation synchronised with the code. There is no mechanism that ensures that documentation and code actually agree. Differences between presumed analysis and actually executed analysis can become very problematic.
People therefore prefer using self-documenting code as much as possible; that is: writing code so that its meaning becomes immediately clear, without comments, even to somebody who hasn’t worked on the code before. Doing this well is hard and takes practice, but improves overall code quality. Once again, using a declarative workflow helps here because the dependencies between the rules are self-documenting the kind of analysis that was performed.
Jeff Atwood has written two seminal essays on this subject:
They are two of the best pieces of advice on programming that I could give.
1 To emphasise: take a look at the edit history of this answer.
2 I recommend ‘targets’ (for R), Nextflow or Snakemake. I used to use (and recommend) GNU make, but I no longer believe this to be an adequate tool.
I quite agree with this answer by @Konrad Rudolph.
I would like to emphasis that using parametrization for your scripts is what will help you avoid multiplying branches in git. So yes, use git, but you don't necessarily have to go "overkill" creating lots of branches.
Then, I would command these scripts from a workflow managment tool that will somehow take care of generating the branches of your analyses. If you use Snakemake, the various options taken along the path from your data to your results will be represented by the wildcards system, and this will be visible in the structure of your folders and file names, due to the fact that Snakemake works by inferring what should be done to produce a file based on its name.
This of course is not an excuse for not using other documentation approaches: Comments in the Snakefile and in the scripts, README files explaining how the workflow was run, using what configuration file. Put your scripts, the Snakefile, its configuration files and the README files under version control and document again using commit messages.
