Script to run everything in a loop for extracting tar.gz files into fastq and to bam with alignment?

Question

Original tar.gz files look like below:

UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz
UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz
UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz
UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz

Before extracting the tar.gz files, I changed their names based on names.text file as below and using the command:

xargs -n2 mv -i <<<"$(sed '1d; s/$/.tar.gz/' names.txt)"

Tar.gz_filenames       Sample_Names
UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz                Sample2
UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz                Sample1
UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz                Sample3
UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz                       Sample4

After changing the names, looks like below:

Sample1.tar.gz
Sample2.tar.gz
Sample3.tar.gz
Sample4.tar.gz

Using the tar.gz files after changing names I used the below code:

for i in /doc/*.tar.gz
do
    tar xvzf $i -C $TMPDIR
    for sample in $TMPDIR/*1.fastq
    do
        dir2="/destination/directory"
        base=$(basename $sample ".1.fastq")
        module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq | samtools view -Sb - > ${dir2}/${base}.bam
    done
done

When I run the above code, the output is like below:

150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq
150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_2.fastq
150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq
150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_2.fastq
140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_2.fastq
140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq
110608_UNC11-SN627_0103_BD0CVVABXX.7_1.fastq
110608_UNC11-SN627_0103_BD0CVVABXX.7_2.fastq
Warning: Could not open read file "/scratch/username/slurm-job.36320423/140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq.1.fastq" for reading; skipping...
Error: No input read files were valid
(ERR): hisat2-align exited with value 1
Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq.1.fastq" for reading; skipping...
Error: No input read files were valid
(ERR): hisat2-align exited with value 1
Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq.1.fastq" for reading; skipping...
Error: No input read files were valid
(ERR): hisat2-align exited with value 1

I dont mins whatever the name of fastq's in $TMPDIR but from $TMPDIR hisat2 should use those fastq's and give output bam files in /destination like below:

Sample1.bam
Sample2.bam
Sample3.bam
Sample4.bam

I would like to run a script doing tar.gz files -> fastq -> bam (everything in loop)

Answer 1

This should work for you:

for i in *.tar.gz
do
    dir2="/destination/directory"
    tar xvzf $i -C $TMPDIR
    files=($TMPDIR/*)
    module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2 $files[1]| samtools view -Sb - > ${dir2}/$i.bam
done

(If this is the sort of thing you'll be doing often, it would be worth investing the time to learn a more robust approach such as Snakemake or Nextflow).

Edit:

If you want to run on all files at the same time, the easiest way is find / xargs

Change you script to this:

for i in $@
do
    dir2="/destination/directory"
    tar xvzf $i -C $TMPDIR
    files=($TMPDIR/*)
    module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2 $files[1]| samtools view -Sb - > ${dir2}/$i.bam
done

Then run it like this:

find . -d 1 -name '*.tar.gz' | xargs -n 1 bash NAMEOFSCRIPT

This works for SGE. I'm not sure about the details of SLURM. You might have to figure out how to run each job in the background or use gnu parallel or something. As @Ian Sudbury says, pipeline software can take care of all these details for you.

Answer 2

Your samples end with _1.fastq and _2.fastq, so you need to change this:

-1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq

to this:

-1 $TMPDIR/${base}_1.fastq -2 $TMPDIR/${base}_2.fastq