Let's say we have a metagenomic sample that is paired-end FASTQ files including 10,000,000 DNA reads collected using shotgun sequencing.
How would one make a random subsample of the mentioned metagenomic sample with for example 1,000,000 reads? I know about the seqtk function that could be used in such cases, but my question is more about whether it is meaningful to represent a metagenomic sample with a small subsample generated using this way or it has to satisfy certain criteria, e.g. in terms of the number of reads and diversity to be a valid representative of the metagenomic sample?
Knowing that some species might be very rare in the sample, would it ever be proper to do subsampling on metagenomic samples?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ Why do you want to subsample in the first place? $\endgroup$– user3051Mar 9, 2020 at 11:56
-
$\begingroup$ To make more samples to be used for machine learning $\endgroup$– RemyMar 9, 2020 at 12:00
-
$\begingroup$ Ah ... data augmentation, yep thats a common approach. I personally can't quite see what teh problem is, if you can perform ML surely you can use a randomisation module, e.g. Python's random or sample? $\endgroup$– M__ ♦Mar 13, 2020 at 4:35
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
I am not NGS expert, I known ML. Essentially you are performing data augmentation which is essential for ML. What I question is whether a replicate of e6 from a population of e7 would really be sufficient for ML because you need sample sizes (replicates in this case) of >100000.
The answer is really simple you bootstrap the data (sample with replacement), to obtain your replicates. Numpy has a random module that will do this,
p.random.seed(345)
pop = np.random.randint(0,10000000 , size=1000000)
Use pop to select from fastaq. The other way in is to define each fastq sequence within pop then simply,
sample = np.random.choice(pop, size=1000000)
Or a combination of the two. If you can do ML you can do Python coding.
-
$\begingroup$ Dear Michael, thank you for the answer, but what I am asking is not how to do random subsampling but the idea whether it makes sense to do it for metagenomic data where it may remove key information of rare taxons. I don't think that subsampling should be done with replacement anyway since the subsamples would not be considered independent for training and testing. $\endgroup$– RemyMar 15, 2020 at 21:45
-
$\begingroup$ Ok, in that case I don't understand the rationale. Sampling is used heavily in machine learning, it is also used in statistical analysis such as Monte Carlo and null distributions. Generally it is used to ensure the observation is an outlier (except for ML) $\endgroup$– M__ ♦Mar 15, 2020 at 23:56