15
votes
How to version the code and the data during the analysis?
There is a couple of points to consider here, which I outline below. The goal here should be to find a workflow that is minimally intrusive on top of already using ...
- 300
14
votes
Accepted
The state, limitations and comparisons of large variant stores
An epic question. Unfortunately, the short answer is: no, there are no widely used solutions.
For several thousand samples, BCF2, the binary representation of VCF, should work well. I don't see the ...
- 6,605
10
votes
Accepted
How to merge .fastq.qz files into a single .fastq.gz with their same id without losing any content in parallel
All you need is cat. You won't find any better tool for a simple job like this. Just run:
...
- 10.6k
10
votes
How to version the code and the data during the analysis?
Your question is somewhat open, but I think it could prove an interesting discussion. I don't believe in many cases it is worth storing the data you have created in ...
- 812
6
votes
How to version the code and the data during the analysis?
First of all, kudos to you for taking versioning seriously. The fact that you're mindful of this issue is a good sign that you want to do responsible research!
For many bioinformatics projects, data ...
- 5,110
5
votes
How to version the code and the data during the analysis?
The Open Science Framework uses versioning for all files and is free to use: https://osf.io
You can integrate data or code from various sources such as github, dropbox, google drive, figshare or ...
- 439
4
votes
How to version the code and the data during the analysis?
The way we deal with this is:
All work is done in a single filesystem mounted on the cluster
This file system is mounted on local machines via sshfs/samba (depending on the location of the current "...
- 3,341
4
votes
Solutions for managing data in a small bioinformatics / 'omics lab?
I don't know of any prebuilt products, but I can describe how we managed this in my postdoc lab, and how I plan to manage it in my newly-started group.
Rule 1: All work happens in the projects ...
- 3,341
3
votes
How to merge .fastq.qz files into a single .fastq.gz with their same id without losing any content in parallel
We sequence DNA on daily basis. Like terdon said, you can use cat to group all the FastQ of all your sample whether they are ...
- 1,135
3
votes
Copying files from multiple directories to a single
Tinkered a bit and realized the solution did not neccistate a for loop.
mkdir ~/ST_0075/alignment; cp ~/ST_0075/annotations/*/*.gff ~/ST_0075/alignment
- 63
3
votes
how do I convert a vcf file for a GWAS study (using R package vcfR)
If you want to do GWAS or mendelian randomisation, you can do it with Plink (v2) which should be faster than R.
- 203
3
votes
Appropriate value to replace outliers in clinical studies
In my experience, it is more common to use all clinical data as-is for clinical studies. And if data is missing, either omit the sample or omit the variable with missing data.
If your classifier can'...
- 367
3
votes
How to version the code and the data during the analysis?
This answer will sort of only cover the big data parts, i.e. things > 100MB ish, and only if your analysis pipeline ties in with the Python ecosystem. It will require a bit of learning
Try using ...
- 131
3
votes
How to version the code and the data during the analysis?
Using Git for version-controlling code is a good practice, but it does not lend itself well to versioning large data files. Manually syncing data across multiple nodes is asking for trouble, you want ...
- 449
2
votes
Accepted
How to version the code and the data during the analysis?
I'm a bit late to answer this question, but we've developed pretty much exactly the system you describe, based on Git and git-annex.
Its called DataLad, free and open source software, and currently ...
- 36
2
votes
Solutions for managing data in a small bioinformatics / 'omics lab?
We have been organizing data on a "per project" basis, using a GitHub repository for each project. Each project ends up being a paper (written in R Markdown), so you could have multiple "projects" ...
- 131
2
votes
Calculate average parental genotype
Beside the correct question by @Kamil, this sounds like a job for datamash.
First let's get the number of columns in the file
...
- 1,352
2
votes
Impute phenotype under some constraints
Question 1: Is there a tool for imputing phenotypes while also using genetic data?
Yes, try Andy Dahl's Phenix from the Marchini lab.
Disclosure: I did my PhD in the Marchini lab, although I was ...
- 2,286
2
votes
Accepted
Getting a stretch of genomic ranges from a dataframe/granges object based on metadata column
The GenomicRanges package in bioconductor will do this. It lets you do 'set' operations on genomic ranges, such as concatenating them, finding overlaps, etc. Hopefully something in the following ...
- 1,012
1
vote
Appropriate value to replace outliers in clinical studies
Removing outliers is common practice in statistical modeling and perfectly acceptable. However, with regards 1.5 IQR I am far from certain about this approach. Normally, if you want to be conservative ...
M__♦
- 13k
1
vote
Calculate average parental genotype
These are unusual parents because they all appear haploid. Anyway, if I was doing this I would calculate the allele frequency per progeny such as makefreq in adegenet in R-cran (just my personal ...
M__♦
- 13k
1
vote
How to split multifasta based on partial fasta header
I had a similar problem and found a solution using awk. Since this is the first time I use awk, the syntax might not be perfect, but you should get an idea of what is possible.
...
1
vote
How to split multifasta based on partial fasta header
The SeqBuddy --pull_records command can grab sequences based on a regular expression against record IDs and/or metadata. Given your example:
...
- 176
1
vote
Is Wikidata a reliable KB for genomic and drugs annotations?
The question is what you mean with "reliable". For bacterial genome annotations there is one Wikidata based platform called WikiGenomes (publications).
- 111
Only top scored, non community-wiki answers of a minimum length are eligible