# how to repeat records in fastq n times efficiently?

How to iterate/repeat a record n times in a fastq file using bioawk? I wrote a python code using biopython, but it is very very slow. So, I am wondering if I can get some help by using bioawk. Thank you so much!

for example, in the file.fq file, I'd like to repeat read_id1 for 100times, read_id2 for 100times, ..., and write them to a new file.

• Hi and welcom to BS, could you please a little bit more what you problem you want to solve? Do you want only specific record or all records of the file? Maybe you can show your python code for clarity. – Mr_Z Jan 17 at 12:43
• You might want to show your biopython implementation. While biopython can be slow, your solution may be sub-optimal – Chris_Rands Jan 17 at 18:29
• There are quite a few people complaining about the speed of Biopython... I think pysam is much faster. – Kamil S Jaron Jan 22 at 19:03

Not sure why you would want to do this, but if the aim is to repeat all records in the fastq/fasta files n times, the most efficient way I can think of is just to concatenate the orginal file n times, similar to this thread.

i.e adapting the accepted answer would look something like this (I have not tested this):

for i in {1..n};do cat file.fq >> file_ntimes.fq ; done


Update based on comments:

To replicate each fastq entry back to back, maybe this hacky unix command can be adjusted (say for n=4):

n=10
paste - - - - <file.fq | awk '{for(i=0;i<$n;i++)print}' | tr '\t' '\n' > file_10times.fq  The idea is to paste blocks of 4 lines (i.e. 1 fastq entry) into one line and replicate this line n times. tr is reversing the paste action (since the separation by paste is a tab character). This won't work if fastq file for some reason has tabs in it which I don't expect but didn't test either. Note, for fasta file this would be paste - - rather than paste - - - - I haven't tested this, but if it works for you I could polish it up if needed. Update2 Based on Xioafeis comment this is solution seems to work as well: bioawk -cfastx '{for(i=1; i<=10; i++) {print "@"$name"\n"$seq"\n+\n"$qual}}' file.fq > file_10times.fq

• Thank you so much for your reply! But I don't think I made my question clear. I want to repeat rd_1 for 100 times, rd_2 for 100 times, ... in this order, not (rd_1, rd_2, ...) x 100 times. – Xiaofei Wang Jan 17 at 13:28
• My goal is to pairwise each of read record in file_1.fq with each of record in file_2.fq into 2 new fastq files and get overlap between them. Actually, for the one that you suggested that is what I want to for my file_new_2.fq, which I have already figured out. But not the one that I am asking. I will edit my question with a figure. Really appreciate for your reply! – Xiaofei Wang Jan 17 at 13:36
• I did wrote a python code using biopython, but it is very slow I guess it slurped whole file into dictionary, so very very slow and seems running forever – Xiaofei Wang Jan 17 at 13:43
• Thank you so much! I figured it out by using bioawk. – Xiaofei Wang Jan 18 at 12:05
• bioawk -cfastx '{for(i=1; i<=10; i++) {print "@"$name"\n"$seq"\n+\n"\$qual}}' test_repeat.fq > test_repeat_10times.fq – Xiaofei Wang Jan 18 at 12:08