Ultimate reproducibility in R?

Question

I'm looking for a convenient and reliable way to make an R analysis reproducible, either at different times or across collaborators.

Listing the package versions or a sessionInfo() output is not very helpful, as it leaves the work of re-creating the environment entirely to you.

As I see, in order to make analysis reproducible, the following has to be the same:

• the R environment (R, repos (Bioconductor), packages, and maybe RStudio)
• data
• code
• seed for random processes: set.seed()

I'll post an answer, but would like to hear about other options.

Answer 1

You're right about trying to get the R environments to be as similar as possible, as easily as possible. It's not always necessary to have the exact environment to get the same results, but it definitely helps if you find yourself struggling to replicate results.

General tips for ensuring reproducibility, not specific to R

See Sandve et al., PLoS Comp Bio, 2013. They mention a number of things that you do, like recording seed values and session info. I'll highlight a few points:

1. Environment: Include a small script that lists all the necessary dependencies
   • This should include all version numbers of languages and packages used
   • Better implementation: make it a script that automatically installs dependencies
2. Data: Share raw and/or processed data where possible
   • At the very least, details on how to obtain the data you used
   • This can be an accession number in a database, or a link to the source material
   • Better implementation: again, make it a script that automatically downloads the raw data you used
3. Code: Comment and document your code
   • People (yourself included) need to understand what you've done and why
   • Additionally, your work may be reproducible, but it may not be correct. If you (or anyone else) attempts to fix a problem in your code, they will have a difficult time doing so if they can't understand what your code is doing
4. Code: Use version control
   • Services like GitHub and BitBucket make it simple to share your exact work with collaborators
   • Version control also ensures that you have a history of all the changes done to your work
5. No manual manipulation of data
   • If something is done to your data, write it down

R-specific tips

1. The packrat package, as you mentioned in your answer
2. Using a number of custom functions for analyzing your data? Compile those functions into an R package
   • Allows others to easily load/unload the exact functions you used through tools like CRAN, BioConductor, or devtools::install_github()
   • Makes it easy to see whether these functions are working as intended, since they are separated from other parts of the analysis
3. Make unit tests for your work with the testthat package
   • Helps ensure that your code is working as intended, and gives others confidence in your work as well
   • Write the tests using testthat::test_that(), and check that they pass using R CMD check or devtools::check()
   • This works especially well with R packages
   • This also works especially well with TravisCI, since you can automatically have your code get tested after you change it and make a commit
4. Consider Docker images for your environment
   • This bundles everything about the R environment together into one easily executable image (and it's not just limited to R)
   • These images can be easily shared through Docker Hub
   • Note this might not always be the best solution, since few people know Docker and it's not always simple to use. Docker also requires admin privileges, which may not be possible for work on computer clusters
   • There is lots of current discussion about Docker in science. See Boettiger, 2014, for an example, using R

Answer 2

Another option I'm trying at the moment is the combination of snakemake, conda.io and knitr.

• Conda is a "Package, dependency and environment management for any language" and could be used to set up the R-environment as well as other software. Note, most bioinformatics software can be found in the related bioconda project.

• Snakemake is a workflow management system, which can be used to automate the preprocessing of your data in a reproducible manner (also it works nicely with conda since a conda environments can be specified in the Snake-Makefile)

• Knitr for dynamic report generation with R. Well suited to document and reproduce your R workflow into reports, ticking your code and seed boxes.

Answer 3

One way to achieve this is via packrat, a dependency management system developed by RStudio.

Setup:

1) Create project

2) Create script in project directory to start RStudio with specific R version:

export RSTUDIO_WHICH_R=/path/to/R
rstudio

3) Bring project under packrat control. Install bioconductor, install packages...

---

Every time you work on the project, run the above script in project directory. https://github.com/rstudio/packrat/issues/342#issuecomment-315059897

Answer 4

checkpoint is a system for ensuring you get packages of the same time / version. However, I confess that I haven't used it and thus can't vouch for it.

My feeling is that the rest will have to be handled outside this: code and data can be versioned in git (and dat). A whole solution might have use something like docker as others have suggested.