1 of 6
Konrad Rudolph
  • 4.8k
  • 14
  • 45

Save the different scripts with git (seems overkill)

What????!

No. Version controlling your scripts (using Git or something similar) is the absolute minimum, and should become completely automatic. For every new project I begin, one of the very first steps is to issue the git init command, and to set up a remote repository (on Github).

To keep track of different analyses, I use a combination of the following approaches:

  1. Write reusable functions/scripts and parametrise. The parameters are kept either inside the script itself (that then calls the relevant function repeatedly), or in a Makefile (I recommend Snakemake).
  2. Document the alternative analysis approaches; once again, this could be a Makefile with different rules for alternative analyses, or a set of notebooks (via R Markdown).
  3. Have different Git branches for mutually exclusive approaches. At the end of the analysis one of these branches gets merged into master, and published. If I want to publish several analysis approaches, I merge all these branches into master, and use approaches (1) or (2) enable them simultaneously.

In fact, I recommend creating a Makefile for every analysis; I have found that this is the most practical, self-documenting way to run an analysis. It most closely resembles a wet-lab lab notebook. The advantage over a single R Markdown document is that rerunning just parts of the analysis can be completely automated, and dependencies in the workflow are apparent from the dependencies of the Makefile rules. This is much harder in R Markdown.

Some time ago I create an example analysis workflow to show how this can be structured. Nowadays I would use Snakemake instead of GNU make.

Konrad Rudolph
  • 4.8k
  • 14
  • 45