I am currently working with fastq files which have hundreds of thousands of lines of text. However, not all of them are sequences I am interested in. My sequences are in one line and have a fixed length of 272 characters.
I possess a list of smaller sequences, around 2 million text strings in the following formats (which are fragments from the various fastq files I have):
Format 1
ATCGATCG
ATCGATCGATCG
ACGATCGTGTTACG
ATTTACGTACGTA
AATCG
ACGATACGATACG
Or I also have them in Format 2
ATCGATCGATACGAT|ATACGTGTTACGAT|ATCGATACG|ATACGATGA
(I just wrote random sequences just to give an idea of how the pattern.txt files look like, I know Format 1 and 2 are not the same in this post, however, the files are in the same order - only the syntax is changed.).
What I've been trying to do is use grep in a Linux environment and use the pattern.txt files to search for the bigger sequences in my fastq files where the pattern is present, and then output it into a .txt file to further work on.
I've used this command so far (with the Format 2 of patterns):
grep -E -f patterns.txt target.fastq > output.txt
Now, this works.
However, I have to limit the number of patterns in my key file and the number of fastq files I can search at once, otherwise I get "grep: memory exhausted" error.
I also want to print out all the sequences up to and including my pattern, but not past it (I don't care what's after my pattern sequence).
And for that I have successfully used the following:
grep -oE "\S+singlepattern" target.fastq > output.txt
However, I can only manage to get this to work if I manually write in the command the pattern sequence I want to look for (thus I can only use a couple). I can't seem to get it to work with the file list.
So I thought of using a while loop. This is what I've written so far in the terminal (it's one continuous line, but I formatted it here for simplicity on the eyes):
while read line; do
grep $line target.fastq > output.txt;
done < patterns.txt
However, this does absolutely nothing and just spits out a blank output.txt
I have tried changing the syntax, with quotations on arguments, no quotations, I tried the various -E -f -o -F, egrep, etc but it's all the same. No error message, just blank file.
I've also tried implementing the \S to the loop as follows:
while read line; do
grep -oE "\S+$line" target.fastq > output.txt;
done < patterns.txt
Yet, the result is the same.
Any suggestions on how I can fix this? Or even other methods to get the cut-down sequences (up to and including my pattern) from the fastq files into an output.txt file?
Thanks in advance!