8
$\begingroup$

I'm going through an RNA-seq pipeline in R/Bioconductor and want to try multiple parameters at subsequent steps, for example, running clustering with different settings, running RegressOut or not on unwanted effects etc. That's a lot of "versions", even if I don't do combinations of these steps.

How can I keep track of this, and my conclusions? Not necessarily want to save the results.

  • Save the different scripts with git (seems overkill)
  • Make notes in the script itself
$\endgroup$
5
$\begingroup$

The main purpose of git is to version code, which usually means sequential improvement of the codebase. While it is possible to use branches for multiple variants of the software, permanent branches are traditionally used for gradual integration of new features (i.e. dev/testing/master branches). Supporting multiple independent branches requires some investment, i.e. distributing common changes among branches via merge or cherry-pick. This is hard to manage when you have more than two-three branches.

If you compare different methods of analysis, you probably want to compare the results between methods. Having them on separate branches makes it hard.

In my opinion you should integrate all methods of analysis into the master branch. To avoid copy & paste, it is better to put common code in a library or an independent script. You can also specify a method as a run-time parameter of your pipeline, and create a meta-script which will execute all methods of interests.

Once you performed benchmarking, you shouldn't remove unused methods from you master branch. Having them is important for reproducible research, and your scripts could be used in the future for new datasets.

$\endgroup$
  • 1
    $\begingroup$ "You can also specify a method as a run-time parameter of your pipeline, and create a meta-script which will execute all methods of interests." This is the approach I use. Allows maximum flexibility for code reuse and keeps a straightforward record of everything that I tried. $\endgroup$ – heathobrien Jun 15 '17 at 12:36
  • $\begingroup$ Don't forget to store the output of the code (with the sessionInfo) if you run the meta-script.R $\endgroup$ – llrs Jun 15 '17 at 12:48
9
$\begingroup$

Save the different scripts with git (seems overkill)

Whoa. I did an actual double take when reading this:1 it’s the opposite of overkill.

Version controlling your scripts (using Git or something similar) is the absolute minimum, and should become completely automatic. For every new project I begin, one of the very first steps is to issue the git init command, and to set up a remote repository (on Github).

To keep track of different analyses, I use a combination of the following approaches:

  1. Write reusable functions/scripts and parametrise. The parameters are kept either inside the script itself (that then calls the relevant function repeatedly), or in a Makefile (I recommend Snakemake).
  2. Document the alternative analysis approaches; once again, this could be a Makefile with different rules for alternative analyses, or a set of notebooks (via R Markdown).
  3. Have different Git branches for mutually exclusive approaches. At the end of the analysis one of these branches gets merged into master, and published. If I want to publish several analysis approaches, I merge all these branches into master, and use approaches (1) or (2) enable them simultaneously.

In fact, I recommend creating a Makefile for every analysis; I have found that this is the most practical, self-documenting way to run an analysis. It most closely resembles a wet-lab lab notebook. The advantage over a single R Markdown document is that rerunning just parts of the analysis can be completely automated, and dependencies in the workflow are apparent from the dependencies of the Makefile rules. This is much harder in R Markdown.

Some time ago I create an example analysis workflow to show how this can be structured. Nowadays I would use Snakemake instead of GNU make.

Regarding your other point:

Make notes in the script itself

“Notes” are a dangerous beasts: documentation is important, but experience shows that it’s sometimes very hard to keep documentation synchronised with the code. There is no mechanism that ensures that documentation and code actually agree. Differences between presumed analysis and actually executed analysis can become very problematic.

People therefore prefer using self-documenting code as much as possible; that is: writing code so that its meaning becomes immediately clear, without comments, even to somebody who hasn’t worked on the code before. Doing this well is hard and takes practice, but improves overall code quality. Once again, using a Makefile helps here because the dependencies between the rules are self-documenting the kind of analysis that was performed.

Jeff Atwood has written two seminal essays on this subject:

They are two of the best pieces of advice on programming that I could give.


1 To emphasise: take a look at the edit history of this answer.

$\endgroup$
  • $\begingroup$ I appreciate the edit, but it does distort the meaning of my answer: The outrage implicit in the “What????!” is entirely intentional. I’ll change it to something else since people seem to disapprove. $\endgroup$ – Konrad Rudolph Jun 15 '17 at 15:43
  • $\begingroup$ +1 for the emphasis on makefiles. Makefile and version control make an awesome combo for reproducibility! $\endgroup$ – Kamil S Jaron Jun 15 '17 at 15:59
  • $\begingroup$ Maybe I should have stated that I meant branching. That seemed to be a lot of overhead for running an R function with a different parameter setting. I'd like to accept your answer too.. but can accept only one. $\endgroup$ – Peter Jul 4 '17 at 9:56
1
$\begingroup$

I quite agree with this answer by @Konrad Rudolph.

I would like to emphasis that using parametrization for your scripts is what will help you avoid multiplying branches in git. So yes, use git, but you don't necessarily have to go "overkill" creating lots of branches.

Then, I would command these scripts from a workflow managment tool that will somehow take care of generating the branches of your analyses. If you use Snakemake, the various options taken along the path from your data to your results will be represented by the wildcards system, and this will be visible in the structure of your folders and file names, due to the fact that Snakemake works by inferring what should be done to produce a file based on its name.

This of course is not an excuse for not using other documentation approaches: Comments in the Snakefile and in the scripts, README files explaining how the workflow was run, using what configuration file. Put your scripts, the Snakefile, its configuration files and the README files under version control and document again using commit messages.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.