How can I delete these lines in my fastq file?

Question

Hi guys =) I'm sorry if this is a repeat but I haven't been able to find an answer using the search field or google.

I am trying to edit a fastq file that has a corrupt line. The read in question is VH00362:5:AAAV5KTHV:1:1210:64850:6869 where the read is just a massive repeat – not the expected 110bp. I am trying to remove two lines from this file and asking around I've been told to use sed as in

gzip -c file.fastq.gz | sed "45500973,45500974d" > new.fastq.gz

but when I run wc -l to check the number of lines the results are incoherent, as if it has deleted much more than the two lines I asked. Another issue that I'm seeing with this file is that every time I run gzip I get a invalid compressed data--format violated error, which tells me the file is corrupted but I'm doing my best to salvage whatever I can from the file as it is an irreplaceable sample.

Is there another way to simply delete these two lines? Any tool or command you could recommend? Any help is deeply appreciated.

Answer 1

This is not the thing I do these days but ...

gunzip -c file.fastq.gz | sed "45500973,45500974d" > new.fastq

You were using gzip instead of gunzip, and that the source of the error. Finally the out file isn't compressed, so you probably want:

gunzip -c file.fastq.gz | sed "45500973,45500974d" | gzip > new.fastq.gz

That should work (I tested it and yes it works).

Answer 2

A fastq file has four line-separated fields per sequence. If you suspect that a sequence is corrupted, then you would want to delete all four lines related to that sequence. If you just delete the first two lines (sequence identifier and raw sequence letters), the file will most probably throw an error down the road. In your case, the sequence would look something like this -

@VH00362:5:AAAV5KTHV:1:1210:64850:6869
GATTTGGGGTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTGTTCAACTCACAGTTT
+
!''*((((***+))%%%++)(%%%%).1***-+*''))**55CCF>>>>>>CCCCCCC65

So, the code to delete these four lines from a fastq.gz file would be -

gunzip -c file.fastq.gz | sed "45500973,45500976d" | gzip > new.fastq.gz

This sed command specifies a range of lines and says that delete lines from line number 45500973 to 45500976.