I am currently looking for a system which will allow me to version both the code and the data in my research.
I think my way of analyzing data is not uncommon, and this will be useful for many people doing bioinformatics and aiming for the reproducibility.
Here are the requrements:
- Analysis is performed on multiple machines (local, cluster, server).
- All the code is transparently synchronized between the machines.
- Source code versioning.
- Generated data versioning.
- Support for large number of small generated files (>10k). These also could be deleted.
- Support for large files (>1Gb). At some point old generated files can permanently deleted. It would be insane to have transparent synchronization of those, but being able to synchronize them on demand would be nice.
So far I am using git + rsync/scp. But there are several downsides to it.
- Synchronization between multiple machines is a bit tedious, i.e. you have to git pull before you start working and git push after each update. I can live with that.
- You are not supposed to store large generated data files or large number of files inside your repository.
- Therefore I have to synchronize data files manually using rsync, which is error prone.
There is something called git annex. It seems really close to what I need. But:
- A bit more work than git, but that's ok.
- Unfortunately it seems it does not work well with the large number of files. Often I have more that 10k small files in my analysis. There are some tricks to improve indexing, but it doesn't solve the issue. What I need is one symlink representing the full contents of directory.
One potential solution is to use Dropbox or something similar (like syncthing) in combination with git. But the downside is there will be no connection between the source code version and the data version.
Is there any versioning system for the code and the data meeting the requirements you can recommend?