How to simulate NGS reads, controlling sequence coverage?

I have a FASTA file with 100+ sequences like this:

>Sequence1
GTGCCTATTGCTACTAAAA ...
>Sequence2
GCAATGCAAGGAAGTGATGGCGGAAATAGCGTTA
......


I also have a text file like this:

Sequence1 40
Sequence2 30
......


I would like to simulate next-generation paired-end reads for all the sequences in my FASTA file. For Sequence1, I would like to simulate at 40x coverage. For Sequence2, I would like to simulate at 30x coverage. In other words, I want to control my sequence coverage for each sequence in my simulation.

Q: What is the simplest way to do that? Any software I should use? Bioconductor?

• What is the read length you're using? How long are the sequences? Do you need to hit the coverage target exactly or with some probability?
– Greg
May 17, 2017 at 17:48
• I would add a few more questions to greg's. Do I understand right that you would like to simulate sequencing of the template sequence from the file? So do the sequences represent genomes? Would you like to consider amplification bias? Sequencing errors? What sequencing platform you would like to simulate? May 30, 2017 at 15:34

10 Answers

I am not aware of any software that can do this directly, but I would split the fasta file into one sequence per file, loop over them in BASH and invoke ART the sequence simulator (or another) on each sequence.

For more information about ART, please see their paper here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3278762/

• Could you expand on this? This really isn't a very useful answer in this state. What is ART? Where can I find it? How do I use it? How does iot work? Does it introduce errors? Can we control the error rate? Also, does it really not take multifasta files? If not, looping over several thousand files in bash will be very, very slow. There may be better ways.
– terdon
May 29, 2017 at 22:38
• >Also, does it really not take multifasta files? If not, looping over several thousand files in bash will be very, very slow. There may be better ways Where did anyone state that it does not take multifasta? OP wanted a very specific coverage per fasta record. May 30, 2017 at 10:49
• No, I didn't mean explain it to me. I meant please edit your answer so it provides this information. We try to have answers as comprehensive as possible and written so they can stand alone without external references on the Stack Exchange sites. Comments are hard to read, easy to miss and result in the horrible mess of biostars where you have no idea where the actual answer is. Please edit your answer to clarify.
– terdon
May 30, 2017 at 10:54

I am working on a Illumina sequencing simulator for metagenomics: InSilicoSeq

It is still in alpha release and very experimental, but given a multi-fasta and an abundance file, it will generate reads from your input genomes with different coverages.

From the documentation:

iss generate --genomes genomes.fasta --abundance abundance_file.txt \
--model_file HiSeq2500 --output HiSeq_reads


Where:

# multi-fasta file
>genome_A
ATGC...
>genome_B
CCGT...
...

# abundance file (total abundance must be 1!)
genome_A    0.2
genome_B    0.4
...


I didn't design it to work with coverage but rather abundance of the genome in a metagenome, so you might have to do a tiny bit of math ;)

The polyester bioconductor package can do this. It says it simulates RNA-seq reads, but I don't know if that's really any different from other NGS reads.

It can use a range of error and bias models, or learn them from a dataset.

• Could you expand on this a little? How would the OP use it on their files? What range of error and bias? How would we use them? What would be a reasonable default to use? Perhaps a minimal working example would be helpful.
– terdon
May 29, 2017 at 22:40

The wgsim package by Heng Li (of BWA and samtools fame) is my go-to tool for simulating Illumina reads. It doesn't provide any convenient way to simulate differential coverage across different sequences, but it shouldn't be to hard to run wgsim multiple times, generating the desired level of coverage for each sequence of interest.

I would implement a Python script to slurp up your test file, and call wgsim (using the subprocess module) for each sequence. This will probably require you to have each sequence in a separate file. :-(

• Could you expand this by adding an example command showing how to take an input fasta file and produce the fastq? This feels more like (helpful) directions to where an answer can be found than an answer.
– terdon
May 29, 2017 at 22:41

This python script takes a fasta file and tsv file with counts and prints the sequences in the fasta files that many times as it is specified in the tsv file (assuming the format in the question). So if bar.tsv and foo.fasta will be your files:

from Bio import SeqIO

repeat = {}
for line in open("bar.tsv"):
seq_id, coverage = line.split()
repeat[seq_id] = int(coverage)

for seq_record in SeqIO.parse(foo.fasta, "fasta"):
for i in range(repeat.get(seq_record.name, 0)):
print(">",seq_record.name,"_",i,sep='')
print(seq_record.seq)

• Could you explain what this does and how it works? It looks like you will just be printing the original sequence $coverage times, and that can't be right. The OP needs to simulate reads, not just repeat the same sequence N times. – terdon May 29, 2017 at 22:49 • Ouch, then I misunderstood the question. I though that the sequences are the reads that should be simulated and 40x is laterally I would like to have this sequence 40 times. I edited the answer to at least avoid confusion. May 30, 2017 at 15:27 • I don't know if you misunderstood, or I did. But one of us did :) Thanks for the edit, at least now the OP can see what this does and decide for themselves. – terdon May 30, 2017 at 15:38 If you're ok with some randomness you can generate reads from your sequence file using a Poisson random variable. You'll need to do some math to figure out what value of lambda to use in order for the expected coverage at each base pair in your read to match what you set in your text file. For example you have a sequence S of length 1,000, a read length of 50, and an insert size of 100. For each base b in S generate a Poisson random variable p. You will then generate p reads from base b to b+50. Then, generate the paired read starting at b+50+100. Again, you would have to play with it to figure out what lambda to use but this would give you basically what you want, as long as you're ok with not having exactly the coverage you're targeting for each read. • And also will be more realistic to simulate insert distance from Poisson-like distribution! The thing is this is a reinvention of a bycicle. There are a lot of simulation tools. May 23, 2017 at 8:18 • Wouldn't pulling from a normal distribution be better for the insertion size? And yeah, I know there are lots of tools but this would be one way of doing it yourself if you were interested. – Greg May 23, 2017 at 15:09 • from my experience, normal is a bit less flexible for fitting - the density is skewed, eg bioinformatics.ucdavis.edu/docs/2016-june-workshop/_images/… . So probably a Negative Binomial will be a good choice, but I never tried to fit it to the actual insert size densities... May 23, 2017 at 15:35 Simulating NGS reads while controlling sequence coverage is now easy with RNFtools (from version 0.3.1). See the tutorial, especially section Sequence extraction. Environment preparation First, install BioConda and add the required channels. Then either install RNFtools in the default Conda environment conda install rnftools  or create and activate a separate Conda environment (preferable) conda create -n rnftools rnftools source activate rnftools  Simulation Assume that you have a reference file ref.fa and a tab-separated coverage file coverage.tsv (e.g., those from your example). Then the following RNFtools Snakefile will do the job you want: import rnftools import csv rnftools.mishmash.sample("simulation_with_coverage_control", reads_in_tuple=1) fa = "ref.fa" tsv = "coverage.tsv" with open(tsv) as f: table = csv.reader(f, delimiter='\t') for seqname, cov in table: rnftools.mishmash.DwgSim( fasta=fa, sequences=[seqname], coverage=float(cov), read_length_1=10, # quick test with supershort reads read_length_2=0, ) include: rnftools.include() rule: input: rnftools.input()  When you save this file (Snakefile) and run snakemake, RNFtools will simulate reads using DWGsim with the coverages defined your text file, and save all the simulated reads in simulation_with_coverage_control.fq. You can play with all the parameters. In particular, you can use a different simulator (e.g., Art-Illumina using rnftools.mishmash.ArtIllumina). See the RNFtools documentation for more information. Another next-gen read simulation tool is gemsim. I haven't tested it, but I would be interested if anyone has had any experience with it. Go to ART website. Download your package, binary or source: Uncompress the .gz file, open the folder "art_bin_MountRainier", then "examples". Open the text file "run_test_examples_illumina.sh". This file contains numerous example of samples codes. That will give you an idea on how you can use the software. Here is an example: # 4) simulation of paired-end reads of 100bp with the mean fragment size 500 and standard deviation 10 # using the built-in seperated quality profiles for A, C, G, and T$art -ss HS20 -i ./testSeq.fa -o ./paired_end_sep -l 100 -f 10 -p -m 500 -s 10 -sp -sam
#convert both aln files to a bed file
../aln2bed.pl paired_end_sep.bed paired_end_sep1.aln paired_end_sep2.aln


You can split your FASTA file sequence-by-sequence using

split -a 6 -p '^>' your_file.fa seq_


and then use any existing read simulator supporting coverage (ART, DWGsim, etc.). If you want to have all the reads mixed (not ordered by the original sequence), you can use RNFtools.

Edit 1:

As @terdon pointed out, the previous command works on OS X only. An analogical one liner for Linux (but with a slightly different naming scheme using numbers rather than letters) can be

csplit -f seq_ -n 6 your_file.fa '/^>/' {*}


To make this command work also on OS X, one needs to install coreutils (e.g., using brew) and then use gcsplit instead of csplit.

Edit 2:

Once FASTA is split by sequences, the simulation becomes straightforward and many different approaches can be used. My favorite one is using GNU Parallel. Imagine that you have your coverages in a text file called covs.txt on separate lines and in the same order as the sequences in your_file.fa, e.g.,

40
30
...


Then you can simulate reads from the original sequences using DWGsim by

ls -1 seq_* | paste covs.txt - \
| parallel -v --colsep '\t' dwgsim -1 100 -2 100 -C {1} {2} sim_{2}


and merge the obtained FASTQ files using:

cat sim_seq_*.bwa.read1.fastq > reads.1.fq
cat sim_seq_*.bwa.read2.fastq > reads.2.fq


One possible danger of this approach is that we have assumed that the number of seq_* files is the same as the number of lines in covs.txt, which might not be true (by mistake). We should check this prior to the simulation step, e.g., by:

[[ "$(ls -1 seq_* | wc -l)" == "$(cat covs.txt | wc -l)" ]] \
|| echo "Incorrent number of lines in covs.txt"


Another caveat is that the simulated reads are not in a random order (they are grouped by source sequences).

• I am afraid you're not answering the question. First of all, you are using non-standard sort options. I assume you use a Mac? The -p is a BSD-only option and not available on other *nix systems AFAIK. That said, while this will probably work to split the file on a MacOS system, splitting it is trivial and there are many, many ways of doing so. The hard bit is simulating the reads and you're not actually explaining how the OP can do that.
– terdon
May 29, 2017 at 22:45
• Thanks for the edit. However, the main issue here is that your answer is not answering the question asked. You are answering "how can I split a multifasta file into many single-sequence files" but the question is about simulating reads. Splitting the file might be part of an answer, or it might not, but it certainly isn't an answer.
– terdon
May 30, 2017 at 14:19
• Oh, and for a portable way without using any of the many specialized tools for this, you can use awk. Something like awk -vRS=">" -F'\n' 'NR>1{print ">"$0 >>$1".fa"}' file.fa. Your csplit (and your split) approach is indeed simpler though.
– terdon
May 30, 2017 at 14:31