How can I give different names to files in a directory with a for loop in a bash script?

Question

I'm expecting to get 17 different paired-end fastq files (34 in total), so I want to make a bash script to just run my code through all the fastq files in a directory at once. How can I change the name of the input and output files each time the script runs through each file? So when it moves to the file_002, all names have file_002 at the beginning instead of file_001, and so on. And also, when merging the R1 and R2 reads how can I make that it only merges the correspondant files with a loop? for examples merging only file_001_R1 with file_001_R2, file_002_R1 with file_002_R2, file_003_R1 with file_003_R2, and so on.

This is my code:

for file in directory_name
do
pear -f file_001_R1.fastq.gz -r file_001_R2.fastq.gz -o file_001.fastq
cutadapt -g TGATAACAATTGGAGCAGCCTC...GGATCGACCAAGAACCAGCA -o file_001_barcode.fastq file_001.fastq
cutadapt -g GTGTACAAATAATTGTCAAC...CTGTCTCTTATACACATCTC -o file_001_UMI.fastq file_001.fastq
seqkit concat file_001_barcode.fastq file_001_UMI.fastq > file_001_concatenation.fastq
seqkit rmdup -s file_001_concatenation.fastq -o file_001_unique_pairs.fastq
seqkit subseq -r file_001_unique_pairs.fastq > file_001_unique_barcodes.fasta
bowtie -q --suppress 1,2,4,6,7,8 -x ref_index file_001_unique_barcodes.fasta > file_001_barcodes_allignment.bowtie
sort file_001_barcodes_allignment.bowtie | uniq -c > file_001_barcode_counts.txt
awk 'BEGIN{print "Barcode,TF_variant,Code"}{print $3","$2","$1}' file_001_barcode_counts.txt > file_001_barcode_counts.csv
done

Thank you.

Answer 1

Here's what I use - along with an example you can modify as needed:

FASTQ_DIR="/directory/containing/FASTQs/"

for R1_FASTQ in "$FASTQ_DIR"*R1*.fastq.gz
do
                         BASE=$(basename "$R1_FASTQ" _R1.fastq.gz)
                         R2_FASTQ="${R1_FASTQ/R1/R2}"
                         R3_FASTQ="${R1_FASTQ/R1/R3}"
                         echo "BASE: $BASE"
                         echo "R1: $R1_FASTQ"
                         echo "R2: $R2_FASTQ"
                         echo "R3: $R3_FASTQ"
                         # process R1/R2 here using "$R1_FASTQ" "$R2_FASTQ" for input, and "${BASE}_XXXXXXX.fastq" for output
done

Basically, the strategy is to iterate over all the R1 FASTQs in a directory, calculate a BASE name that can be used to rename output files with arbitrary extensions (like .sam, .bam, etc...), and use BASH shell parameter expansion to calculate and set variables to the R2 (and R3 if necessary) reads present in an Illumina run that uses standard Illumina naming conventions.

This may need modification depending on the naming conventions you are using.

Answer 2

Something like this? This uses printf to create a zero-padded 3-digit number corresponding to 17 numbers from 1 to 17:

for fileBase in $(printf 'file_%03d\n' $(seq 1 17));
   do echo ${fileBase}_R1.fq.gz ${fileBase}_R2.fq.gz;
   pear -f ${fileBase}_R1.fastq.gz -r ${fileBase}_R2.fastq.gz -o ${fileBase}.fastq
   ...
done