1
$\begingroup$

Hi guys =) I'm sorry if this is a repeat but I haven't been able to find an answer using the search field or google.

I am trying to edit a fastq file that has a corrupt line. The read in question is VH00362:5:AAAV5KTHV:1:1210:64850:6869 where the read is just a massive repeat – not the expected 110bp. I am trying to remove two lines from this file and asking around I've been told to use sed as in

gzip -c file.fastq.gz | sed "45500973,45500974d" > new.fastq.gz

but when I run wc -l to check the number of lines the results are incoherent, as if it has deleted much more than the two lines I asked. Another issue that I'm seeing with this file is that every time I run gzip I get a invalid compressed data--format violated error, which tells me the file is corrupted but I'm doing my best to salvage whatever I can from the file as it is an irreplaceable sample.

Is there another way to simply delete these two lines? Any tool or command you could recommend? Any help is deeply appreciated.

$\endgroup$
1
  • $\begingroup$ Does gzip -k file.fastq.gz report an error? If so, I would first try to gunzip the file, or zcat file.fastq.gz > file.fastq and then operate on the decompressed one. I am not sure, but I suspect that a corrupted archive might not cat as expected and I would feel safer getting as much of the file decompressed as I can first. I am not even sure you will get anything, please try and report back. $\endgroup$
    – terdon
    May 23 at 11:19

2 Answers 2

2
$\begingroup$

This is not the thing I do these days but ...

gunzip -c file.fastq.gz | sed "45500973,45500974d" > new.fastq

You were using gzip instead of gunzip, and that the source of the error. Finally the out file isn't compressed, so you probably want:

gunzip -c file.fastq.gz | sed "45500973,45500974d" | gzip > new.fastq.gz

That should work (I tested it and yes it works).

$\endgroup$
1
$\begingroup$

A fastq file has four line-separated fields per sequence. If you suspect that a sequence is corrupted, then you would want to delete all four lines related to that sequence. If you just delete the first two lines (sequence identifier and raw sequence letters), the file will most probably throw an error down the road. In your case, the sequence would look something like this -

@VH00362:5:AAAV5KTHV:1:1210:64850:6869
GATTTGGGGTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTGTTCAACTCACAGTTT
+
!''*((((***+))%%%++)(%%%%).1***-+*''))**55CCF>>>>>>CCCCCCC65

So, the code to delete these four lines from a fastq.gz file would be -

gunzip -c file.fastq.gz | sed "45500973,45500976d" | gzip > new.fastq.gz

This sed command specifies a range of lines and says that delete lines from line number 45500973 to 45500976.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.