1
$\begingroup$

Note: this question can also be found on Biostars

I need to perform a stress test in a GWAS tool and the duty demands a dataset (plink format) having 100 thousand samples, having 40 million SNPs in a proportion of case:control=30:70. I'm performing the command:

plink1.9 --simulate ds1.sim --make-bed --out ds1 --simulate-ncases 30000 --simulate-ncontrols 70000

But after about 3 hours execution, I'm facing Error: File write failure., even though my environment has 256T of disk. (It's a container with a S3 (Object store) mounted through s3fs).

I was wondering if there's a clever way to do that? Perhaps breaking it into small chunks and merging up later somehow? It would prevent me to be waiting hours till an error happens and I loose either time and the sample generated so far.

Thanks in advance!

$\endgroup$
0
1
$\begingroup$

First comment: you should be using real data, rather than simulated data, as we are not very good at modelling genetic variation and recombination at a genome-wide scale. A way that this can be done with GCTA is mentioned in the PLINK manual.

I'll do some calculations to estimate the size of data you need to store:

Assume that this plink-formatted file is in tped format (one row per SNP), and the entire genome is diploid. That means that a single SNP/individual pair will take up three bytes. You want 30,000 cases and 70,000 controls, so each line from each sample will take up 300kB (plus a bit of change for the header). With 1 thousand SNPs, that's 300MB per sample; with 1 million SNPs, that's 300GB per sample; with 10 million SNPs, that's 3TB per sample; with 40 million SNPs, that's 12 TB for one sample.

Reading the plink manual, I see that the BED format is a two-bit compression for each SNP/individual pair, so one byte per 4 individuals, or about 12x compression (i.e. 1TB for one sample).

Okay, that's a lot. It's well within a 256TB storage limit, but might hit file size limits depending on the file system.

If there is a data generation issue with PLINK, it might be worth trying (as you suggest) splitting the input for simulation then merging afterwards. My guess is that doing the simulation on fewer SNPs (rather than fewer individuals) would work better. Plink has a --bmerge option for merging the compressed binary file formats.

I usually approach problems like these by working my way up from something that I know works. Try first merging two datasets of 1000 SNPs. If that works, try larger datasets. Rinse and repeat until the computer crashes, you get bored, or you reach your goal.

$\endgroup$
2
  • $\begingroup$ Hi @gringer thanks for the answer! Do you know where can I find free and open license real data with this amount of samples? I have been using the 1000 genome phase 3, but they have only 2504 samples. For this one, I managed to create synthetic phenotypes through some scripts I put together (github.com/bambrozio/bioinformatics/blob/master/utils/…), but GCTA --simu-cc/--simu-qt probably would address it in a more reliable way. Thanks for that! $\endgroup$ Feb 15 '20 at 11:26
  • 1
    $\begingroup$ For now, I managed to create a 20k samples with 8mi SNPs with Plink (1/5 from my final goal). My limitations is actually the S3 file size limitations. Also, your maths make sense, and suggests my scenarios doesn't. Thanks for that! Btw - loved your TEDxWellington talk :) $\endgroup$ Feb 15 '20 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.