Batch alignment of inconsistently named Fastq files

Question

I have numerous gzip-compressed paired-end Fastq files with ChIP-seq data that I would like to align with bowtie2. I confirmed that bowtie2 takes .gz files, although I do see this warning when I run for two samples:

Warning: gzbuffer added in zlib v1.2.3.5. Unable to change buffer size from default of 8192.

My files are named like so.

10_S10_L002_R1_001.fastq.gz  12_S12_L002_R1_001.fastq.gz  2_S2_L001_R1_001.fastq.gz  4_S4_L001_R1_001.fastq-003.gz  6_S6_L001_R1_001.fastq-002.gz  8_S8_L002_R1_001.fastq.gz  
10_S10_L002_R2_001.fastq.gz  12_S12_L002_R2_001.fastq.gz  2_S2_L001_R2_001.fastq.gz  4_S4_L001_R2_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz  8_S8_L002_R2_001.fastq.gz  
11_S11_L002_R1_001.fastq.gz  1_S1_L001_R1_001.fastq.gz    3_S3_L001_R1_001.fastq.gz  5_S5_L001_R1_001.fastq-002.gz  7_S7_L002_R1_001.fastq.gz      9_S9_L002_R1_001.fastq.gz
11_S11_L002_R2_001.fastq.gz  1_S1_L001_R2_001.fastq.gz    3_S3_L001_R2_001.fastq.gz  5_S5_L001_R2_001.fastq-003.gz  7_S7_L002_R2_001.fastq.gz      9_S9_L002_R2_001.fastq.gz

The bash commands I tried are below. When I run this test command:

for i in $(ls *.fastq*.gz | grep -v 6 | rev | cut -c 16- | rev | uniq); do 
    echo $i; 
done

...I get this output.

12_S12_L002_R1_001.fastq.gz  12_S12_L002_R2_001.fastq.gz 

When I run this:

for i in $(ls *.fastq*.gz | rev | cut -c 16- | rev | uniq); do 
    echo $i 
    bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 -1 ${i}R1_001.fastq.gz -2 ${i}R2_001.fastq.gz | samtools view -bS - >${i%}.bam
done

I do see one BAM for my paired end files. But as I see the second set of files which are named:

6_S6_L001_R1_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz

if I run the same script over them, I would get an error as the loop can't read the paired end file.

So the tedious way is I can make a separate folder for these sets of data with fastq-002.gz and fastq-003.gz and can run the script by tweaking the cut parameter.

But I would like to know what is the programmatic way of resolving the issue.

Answer 1

A very simple approach, but one which will work with the names you show, is to remove any occurrence of 00N (where N is any number) from right before the .gz. To do so, edit your script to:

for i in $(ls *.fastq*.gz | sed 's/00[0-9]\.gz/.gz/' | rev | cut -c 16- | rev | uniq); do 
  bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 \
    -1 ${i}R1_001.fastq.gz \
    -2 ${i}R2_001.fastq.gz | 
        samtools view -bS - >${i%}.bam"
done