Solutions for managing data in a small bioinformatics / 'omics lab?

Question

A different sort of problem: even a small 'omics lab generates a lot of data, raw, intermediate and processed. What (software) solutions exist for managing this data, such that "old" data can be retrieved and checked or re-analysed, even after people have left the lab? Important points would be:

• ease of installation
• ease of putting data in, in an appropriately tagged / labeled fashion (it's no good if the repository is just a centralised bad mess)
• useful search and exploration
• security (i.e. restricted to members of the lab)
• previews / summaries of the data
• can accommodate any dataset
• local, not SaaS

It seems from my explorations of the topic that there's very little between the primitive (e.g. handrolled Access databases) and big industrial solutions. Things I have looked at include:

• Dataverse: very popular, installation seems complex and unclear if uploading is that easy
• DSpace: mostly for publications and documents
• CKAN
• OSF: used this for a while, integrates with a lot of services but uploading data seemed awkward

Answer 1

I don't know of any prebuilt products, but I can describe how we managed this in my postdoc lab, and how I plan to manage it in my newly-started group.

Rule 1: All work happens in the projects project directory on the central filestore, not on your desktop or laptop.

Rule 2: Heavy computational work is done by the groups standard analysis pipelines. Interpretation is done in jupyter notebooks or Rmarkdown.

A project has a directory on the group's filestore. That directory has a fixed structure:

proj001----raw_data             *
       |
        --external_datasets
       |
        --src                  *
       |
        --notebooks            *
       |
        --pipeline1
       |
        --pipeline2
       | 
        -- etc
 * link to a seperate, backed up filesystem.

Pipelines are where heavy duty analysis happens and everyone uses the same standard pipelines, producing the same standard output files and database structure.

So common pipelines might be the mapping pipeline, the readqc pipeline, the differential expression pipeline, the exome pipeline, the chip-seq pipeline etc...

Pipelines have three important outputs: an automatically generated html report, an sqlite database and files in the export directory.

When we archive a project, this is what we save, along with the pipeline's configuration file and log file.

So if I know that Jane Bloggs did an RNAseq 5 years ago on a cell type i'm interested in, if I konw that that was project 5, I know that in the project 5 directory there will be a diff_expression_pipeline directory, and that it will contain an sqlite database called csvdb and that that will have a table called refcoding_deseq_gene_diff and that table will follow a known format. There will be bigwigs in the export directory. Or the BAMs will be in the mapping_pipeline directory.

Of course the problem remains knowing that Jane Bloggs did this RNAseq and what the project was called. We use a Wiki for this, but its not ideal.

Answer 2

We have been organizing data on a "per project" basis, using a GitHub repository for each project. Each project ends up being a paper (written in R Markdown), so you could have multiple "projects" based on the same datasets. See this as an example:

https://github.com/SchlossLab/new_project

The analyses are all managed with a Makefile that goes from loading the sequence data from the SRA, to performing analyses, to rendering the final manuscript. This way it is always clear how things were done. Here is an example of what that can look like:

https://github.com/SchlossLab/new_project/blob/master/Makefile

As far as organizing the raw data and whatnot, I have just used a simple MySQL database hosted on MAMP (Mac version, Linux is LAMP, Windows is WAMP). It has a GUI and everything, and is pretty easy to use. Here is a resource for getting started on that:

http://microbiology.github.io/PDFs/MySQL-Intro-Workshop-Hannigan.pdf

All of this stuff is free to use, which is awesome! Hope this helps! :)