# Script to run everything in a loop for extracting tar.gz files into fastq and to bam with alignment?

Original tar.gz files look like below:

UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz
UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz
UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz
UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz


Before extracting the tar.gz files, I changed their names based on names.text file as below and using the command:

xargs -n2 mv -i <<<"$(sed '1d; s/$/.tar.gz/' names.txt)"

Tar.gz_filenames       Sample_Names
UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz                Sample2
UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz                Sample1
UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz                Sample3
UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz                       Sample4


After changing the names, looks like below:

Sample1.tar.gz
Sample2.tar.gz
Sample3.tar.gz
Sample4.tar.gz


Using the tar.gz files after changing names I used the below code:

for i in /doc/*.tar.gz
do
tar xvzf $i -C$TMPDIR
for sample in $TMPDIR/*1.fastq do dir2="/destination/directory" base=$(basename $sample ".1.fastq") module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1$TMPDIR/${base}.1.fastq -2$TMPDIR/${base}.2.fastq | samtools view -Sb - >${dir2}/{base}.bam done done  When I run the above code, the output is like below: 150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq 150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_2.fastq 150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq 150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_2.fastq 140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_2.fastq 140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq 110608_UNC11-SN627_0103_BD0CVVABXX.7_1.fastq 110608_UNC11-SN627_0103_BD0CVVABXX.7_2.fastq Warning: Could not open read file "/scratch/username/slurm-job.36320423/140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq.1.fastq" for reading; skipping... Error: No input read files were valid (ERR): hisat2-align exited with value 1 Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq.1.fastq" for reading; skipping... Error: No input read files were valid (ERR): hisat2-align exited with value 1 Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq.1.fastq" for reading; skipping... Error: No input read files were valid (ERR): hisat2-align exited with value 1  I dont mins whatever the name of fastq's in TMPDIR but from $TMPDIR hisat2 should use those fastq's and give output bam files in /destination like below: Sample1.bam Sample2.bam Sample3.bam Sample4.bam  I would like to run a script doing tar.gz files -> fastq -> bam (everything in loop) ## 2 Answers This should work for you: for i in *.tar.gz do dir2="/destination/directory" tar xvzf$i -C $TMPDIR files=($TMPDIR/*)
module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2$files[1]| samtools view -Sb - > ${dir2}/$i.bam
done


(If this is the sort of thing you'll be doing often, it would be worth investing the time to learn a more robust approach such as Snakemake or Nextflow).

Edit:

If you want to run on all files at the same time, the easiest way is find / xargs

Change you script to this:

for i in $@ do dir2="/destination/directory" tar xvzf$i -C $TMPDIR files=($TMPDIR/*)
module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2$files[1]| samtools view -Sb - > ${dir2}/$i.bam
done


Then run it like this:

find . -d 1 -name '*.tar.gz' | xargs -n 1 bash NAMEOFSCRIPT


This works for SGE. I'm not sure about the details of SLURM. You might have to figure out how to run each job in the background or use gnu parallel or something. As @Ian Sudbury says, pipeline software can take care of all these details for you.

• Definately +1 for putting all this in some sort of pipeline. Many of these will now handle automated distribution of many files to your SLURM cluster. Job tracking. Restarts after errors etc. – Ian Sudbery Apr 23 '18 at 9:49
• @heathobrien Hi, The above code uses one after another tar.gz file. It is taking huge time. How should I make the code work for all the tar.gz files at a time? – stack_learner Apr 30 '18 at 6:26
• see updated answer – heathobrien Apr 30 '18 at 7:12
• Even I haven't worked on SLURM before. Do you think if I give #SBATCH --array=1-6%3 will work? lets say I have 6 tar.gz files so 3 jobs at a time. – stack_learner Apr 30 '18 at 11:48

Your samples end with _1.fastq and _2.fastq, so you need to change this:

-1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq


to this:

-1 $TMPDIR/${base}_1.fastq -2 $TMPDIR/${base}_2.fastq

• Thanq. ya this was the only error in my code. – stack_learner Apr 23 '18 at 11:51
• Hi, The above code uses one after another tar.gz file. It is taking huge time. How should I make the code work for all the tar.gz files at a time? – stack_learner Apr 30 '18 at 6:27
• Untar the files and use a workflow manager like snakemake. BTW, make sure you have sufficient resources to process many samples in parallel. – Devon Ryan Apr 30 '18 at 6:36
• But as soon as Untar done for one file it should be the input for hisat2. Yes, I'm using 20 cpu cores, 1 week time and tmp 500G, memory usage 4G. – stack_learner Apr 30 '18 at 6:43
• Then put the untar command into something like Snakemake too. – Devon Ryan Apr 30 '18 at 6:44