fuse fastq files with multiple records

Question

I'd like to fuse ~50 pairs of corresponding fastq files on per sequence bases. I'd be happy for a solution as a Linux script, R script (if feasible with huge files), or any free software that runs preferably under Debian Linux (otherwise Windows). Aim is preprocessing for a Linux pipeline for NGS data. Ideally both the sequence and quality data are fused.

The aim of this request is the following: there is an existing pipeline for downstream mapping of immunoreceptor sequences (migec/mixcr/vdjtools etc.). Previously we got Illumina reads (read 1, read 2 = reverse), with the index and UMI sequences attached to one of the reads => can be alligned and processed to habe 1 biological useful sequence with UMI and index at one end. NOW we get read 1, read 2 and seperately "read 3" which is just the UMI+Index reads, and which was attached by the core facility to read 1 using software we do not have access to. Since the previous pipeline works well, we prefere to use it in the future, but are open for suggestions how to use the index reads with migec/mixcr/vdjtools otherwise.

Each of the 2 files contains the same number (millions) of short (~300 bp) and corresponding DNA reads:

File_A.fastq:

• @record1 / aaa... / +... / ...
• @record2 / ...
• @record3 / ...
• ... (millions)

File_B.fastq:

• @record1 / ttt... / +... / ...
• @record2 / ...
• @record3 / ...
• ... (millions)

The target file should look like:

• @record1 / aaa...ttt... / +... / ...
• @record2 / ...
• @record3 / ...
• ... (millions)

(of course the aaa...ttt... is just an example)

I was looking for a solution on Stackexchange and Google, but the solutions, I found do not work for me because

• manual processing is not an option given the number of sequences
• I need to fuse the files on a per sequence basis, i.e. a simple Linux command like cat file1 file2 is not what I need (as far as I understand, this would give me (record1_A record2_A.... record1B record2_B).
• R, package shortread, readFastq + writeFastq would be perfect, but cannot handle such large numbers of input lines

Maybe I did not use the right search terms, but in any case I'd be happy for a solution

Answer 1

Based on the comments of the question I thought to put together this answer using GNU coreutils which should exist on pretty much all systems.

The idea is to paste the 2 files together with paste (assuming they are in the same order and each line correspond to the same line in the other file).

Since only the sequence and quality of both files should be kept, the lines starting with @ and + need to be made empty for one of the files. This can be achived with sed, e.g. sed 's/^@.*//' empties all lines starting with @ but keeps the empty line.

To modify the second file on the fly, one can using process substitution <(). Everything strung together should be something like this:

file1=File_A.fastq
file2=File_B.fastq
paste -d "" $file1 <(cat $file2 | sed 's/^@.*//' | sed 's/^+.*//') > target.fastq

I haven't tested it thoroughly, but could be a good starting point.

If you have a lot of files to paste, you might however want to process them first (not on the fly) using:

for file in $(ls *.fastq); do 
  cat $file | sed 's/^@.*//' | sed 's/^+.*//' > ${file%.fastq}_cleaned.fastq
done

This creates File_A_mod.fastq from File_A.fastq etc. Best to move them in a separate directory (make sure the first one is the original instead of the _cleaned one since it serves as the base). You should have the following in the folder: File_A.fastq File_B_cleaned.fastq File_C_cleaned.fastq .... Those can be than pasted with paste -d "" *.fastq > target.fastq.

Answer 2

What you actually want to do is overlap the reads, for which you can use tools like FLASH or BBMerge (I've had better luck with FLASH).