Save the different scripts with git (seems overkill)
Whoa. I did an actual double take when reading this:1 it’s the opposite of overkill.
Version controlling your scripts (using Git or something similar) is the absolute minimum, and should become completely automatic. For every new project I begin, one of the very first steps is to issue the
git init command, and to set up a remote repository (on Github).
To keep track of different analyses, I use a combination of the following approaches:
- Write reusable functions/scripts and parametrise. The parameters are kept either inside the script itself (that then calls the relevant function repeatedly), or in a declarative workflow.2
- Document the alternative analysis approaches; once again, this could be a declarative workflow with different rules for alternative analyses, or a set of notebooks (via R Markdown).
- Have different Git branches for mutually exclusive approaches. At the end of the analysis one of these branches gets merged into the main branch, and published. If I want to publish several analysis approaches, I merge all these branches into the main branch, and use approaches (1) or (2) enable them simultaneously.
In fact, I recommend creating a declarative workflow script for every analysis; I have found that this is the most practical, self-documenting way to run an analysis. It most closely resembles a wet-lab lab notebook. The advantage over a single R Markdown document is that rerunning just parts of the analysis can be completely automated, and dependencies in the workflow are apparent from the dependencies of the workflow rules. This is much harder in R Markdown.
Some time ago I create an example analysis workflow to show how this can be structured.
Regarding your other point:
Make notes in the script itself
“Notes” are a dangerous beasts: documentation is important, but experience shows that it’s sometimes very hard to keep documentation synchronised with the code. There is no mechanism that ensures that documentation and code actually agree. Differences between presumed analysis and actually executed analysis can become very problematic.
People therefore prefer using self-documenting code as much as possible; that is: writing code so that its meaning becomes immediately clear, without comments, even to somebody who hasn’t worked on the code before. Doing this well is hard and takes practice, but improves overall code quality. Once again, using a declarative workflow helps here because the dependencies between the rules are self-documenting the kind of analysis that was performed.
Jeff Atwood has written two seminal essays on this subject:
They are two of the best pieces of advice on programming that I could give.
1 To emphasise: take a look at the edit history of this answer.
2 I recommend ‘targets’ (for R), Nextflow or Snakemake. I used to use (and recommend) GNU make, but I no longer believe this to be an adequate tool.