I am trying to check my variant generation pipeline for humans. The pipeline is for NGS. I am searching for any dataset(.fastq+.bam+.vcf) for any specific sample. Now, GIAB has a nice dataset for that. However, the problem is that it's not a tiny dataset. And it's not feasible to test my pipeline with this data every time. I tried to generate a synthetic dataset with some read simulators(NEAT, MASON2). However, their aligned read files(.bam) files seem not very good. You can check my Issue for NEAT in link. So, my question is, Is there any publicly available tiny dataset(.fastq,.bam, .vcf) with which I can validate my pipeline fast? Or, is there any simulator with which I can simulate .bam, .fastq files from a given input .vcf? I am using fastqc,bwa,samtools,GATK4 mainly and GRCH38 reference in my pipeline.


1 Answer 1


I would suggest subsampling an existing dataset such as the GIAB resource that you've already indicated.

Use a utility such as seqtk to subsample the appropriate number of reads from the dataset, and use that.

Example with seqtk to get 1 million reads each from paired fastqs while preserving pairing:

  # note -s100 random seed which should be used for both to keep pairs
  seqtk sample -s100 giab_read1.fq 1000000 > giab_sub_r1.fq
  seqtk sample -s100 giab_read2.fq 1000000 > giab_sub_r2.fq
  • $\begingroup$ Ok, that's actually a good suggestion. However, how to subsample gold standard fastq, .bam, and .vcf? So, that I know my pipeline steps/changes are okay in all three steps? $\endgroup$ Commented Oct 30, 2023 at 2:46
  • 1
    $\begingroup$ @ShafayetRahat just subsample the fastq, and then run the small fastq files through your pipeline to get bam abd vcf. I use a similar approach, and it only takes about 5 minutes to run through the pipeline with these small files. $\endgroup$
    – terdon
    Commented Oct 30, 2023 at 9:31
  • $\begingroup$ @ShafayetRahat terdon is right, unless you're doing something odd mapping should be unaffected by depth. And you will not be able to subsample a VCF to only represent certain reads, I am not sure how you would do that. (though in principle you can do it with a BAM file, it would just be a lot of work compared to re-mapping). $\endgroup$ Commented Oct 30, 2023 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.