How to merge .fastq.qz files into a single .fastq.gz with their same id without losing any content in parallel

Question

I have a large number of .fastq.gz files of different lane and reads. I have to merge them each reads group files into single .fastq.gz files.

**eg:

1st type
NA24694_GCCAAT_L001_R1_001.fastq.gz
NA24694_GCCAAT_L001_R1_002.fastq.gz
…....
…....
NA24694_GCCAAT_L001_R1_033.fastq.gz

2nd type
NA24694_GCCAAT_L001_R2_001.fastq.gz
NA24694_GCCAAT_L001_R2_002.fastq.gz
…....
…....
NA24694_GCCAAT_L001_R2_027.fastq.gz

3rd type
NA24694_GCCAAT_L002_R1_001.fastq.g
NA24694_GCCAAT_L002_R1_002.fastq.g
…....
…....
NA24694_GCCAAT_L002_R1_040.fastq.g

4th type
NA24694_GCCAAT_L002_R2_001.fastq.gz
NA24694_GCCAAT_L002_R2_002.fastq.gz
…....
…....
NA24694_GCCAAT_L002_R2_040.fastq.gz

so now i need to merge all the corresponding files into single file.

Output:
EA00694_GCCAAT_L001_R1.fastq.gz
EA00694_GCCAAT_L001_R2.fastq.gz
EA00694_GCCAAT_L002_R1.fastq.gz
EA00694_GCCAAT_L002_R2.fastq.gz

I tried ""cat'" but its not possible to do for large files. Could any one help me to run in script parallel in perl or python or shell. or any linux command which can be used for large file.

Thanks all.

Answer 1

All you need is cat. You won't find any better tool for a simple job like this. Just run:

cat NA24694_GCCAAT_L001_R1*fastq.gz > EA00694_GCCAAT_L001_R1.fastq.gz
cat NA24694_GCCAAT_L001_R2*fastq.gz > EA00694_GCCAAT_L001_R2.fastq.gz
cat NA24694_GCCAAT_L001_R3*fastq.gz > EA00694_GCCAAT_L002_R1.fastq.gz
cat NA24694_GCCAAT_L001_R4*fastq.gz > EA00694_GCCAAT_L002_R2.fastq.gz

That should work just fine. You say you tried cat and it "didn't work" but since you don't tell us how it failed, I can't really help. The only issues I can think of is that either there are too many files (as in several hundred thousand, whatever the value returned by getconf ARG_MAX on your system which is 2097152 on mine) or, more likely, you are running out of disk space.

If it's a disk space issue, you might be able to get around it by adding each file and then deleting it:

for i in 1 2 3 4; do 
    for file in NA24694_GCCAAT_L001_R${i}_*fastq.gz; do
        cat "$file" >> EA00694_GCCAAT_L001_R${i}.fastq.gz && rm "$file"
    done
done

IMPORTANT: the command above is destructive. It will delete each file after it has been added to the new one. If everything works fine that's not a problem, but if I've made a mistake or your file names are slightly different you might lose your data. So I strongly urge you to simply run the cat commands in the first section on a machine with enough disk space to deal with them.

Answer 2

We sequence DNA on daily basis. Like terdon said, you can use cat to group all the FastQ of all your sample whether they are bgzipped or not like so:

root@slurm1: ~  # ls
am224_S6_L001_R1_001.fastq.gz
am224_S6_L001_R2_002.fastq.gz
am224_S6_L002_R1_003.fastq.gz
am224_S6_L002_R2_004.fastq.gz
am232_S10_L001_R1_001.fastq.gz
am232_S10_L001_R2_002.fastq.gz
am232_S10_L002_R1_003.fastq.gz
am232_S10_L002_R2_004.fastq.gz

You could do:

# initiate am empty array
listSamples=()

# loop to add the unique samples to an array
for i in *.fastq.gz;
do
    if [[ "$sampleName" =~ "${listSamples[@]}" ]]; then
        sampleName=$(echo $i | cut -d'_' -f1)
        listSamples+=("$sampleName")
    fi
done

# loop in the newly created array of samples and group them by reads number
for i in "${listSamples[@]}"
do
    cat "$i"_*_R1_* > "$i"_1.fastq.gz
    cat "$i"_*_R2_* > "$i"_2.fastq.gz
done

However, I highly suggest you to compress your files as .bz2 to save storage as much as possible. You can achieve this using the multithreaded tool lbzip2. Assuming that you have 10 CPUs available, you can do:

for i in "${listSamples[@]}"
do
    bzcat "$i"_*_R1_* | lbzip2 -n 10 --best > "$i"_1.fastq.bz2
    bzcat "$i"_*_R2_* | lbzip2 -n 10 --best > "$i"_2.fastq.bz2
done

Hope this helps !