-1
$\begingroup$

Original tar.gz files look like below:

UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz
UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz
UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz
UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz

Before extracting the tar.gz files, I changed their names based on names.text file as below and using the command:

xargs -n2 mv -i <<<"$(sed '1d; s/$/.tar.gz/' names.txt)"

Tar.gz_filenames       Sample_Names
UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz                Sample2
UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz                Sample1
UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz                Sample3
UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz                       Sample4

After changing the names, looks like below:

Sample1.tar.gz
Sample2.tar.gz
Sample3.tar.gz
Sample4.tar.gz

Using the tar.gz files after changing names I used the below code:

for i in /doc/*.tar.gz
do
    tar xvzf $i -C $TMPDIR
    for sample in $TMPDIR/*1.fastq
    do
        dir2="/destination/directory"
        base=$(basename $sample ".1.fastq")
        module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq | samtools view -Sb - > ${dir2}/${base}.bam
    done
done

When I run the above code, the output is like below:

150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq
150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_2.fastq
150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq
150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_2.fastq
140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_2.fastq
140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq
110608_UNC11-SN627_0103_BD0CVVABXX.7_1.fastq
110608_UNC11-SN627_0103_BD0CVVABXX.7_2.fastq
Warning: Could not open read file "/scratch/username/slurm-job.36320423/140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq.1.fastq" for reading; skipping...
Error: No input read files were valid
(ERR): hisat2-align exited with value 1
Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq.1.fastq" for reading; skipping...
Error: No input read files were valid
(ERR): hisat2-align exited with value 1
Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq.1.fastq" for reading; skipping...
Error: No input read files were valid
(ERR): hisat2-align exited with value 1

I dont mins whatever the name of fastq's in $TMPDIR but from $TMPDIR hisat2 should use those fastq's and give output bam files in /destination like below:

Sample1.bam
Sample2.bam
Sample3.bam
Sample4.bam

I would like to run a script doing tar.gz files -> fastq -> bam (everything in loop)

$\endgroup$
7
$\begingroup$

This should work for you:

for i in *.tar.gz
do
    dir2="/destination/directory"
    tar xvzf $i -C $TMPDIR
    files=($TMPDIR/*)
    module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2 $files[1]| samtools view -Sb - > ${dir2}/$i.bam
done

(If this is the sort of thing you'll be doing often, it would be worth investing the time to learn a more robust approach such as Snakemake or Nextflow).

Edit:

If you want to run on all files at the same time, the easiest way is find / xargs

Change you script to this:

for i in $@
do
    dir2="/destination/directory"
    tar xvzf $i -C $TMPDIR
    files=($TMPDIR/*)
    module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2 $files[1]| samtools view -Sb - > ${dir2}/$i.bam
done

Then run it like this:

find . -d 1 -name '*.tar.gz' | xargs -n 1 bash NAMEOFSCRIPT

This works for SGE. I'm not sure about the details of SLURM. You might have to figure out how to run each job in the background or use gnu parallel or something. As @Ian Sudbury says, pipeline software can take care of all these details for you.

$\endgroup$
  • $\begingroup$ Definately +1 for putting all this in some sort of pipeline. Many of these will now handle automated distribution of many files to your SLURM cluster. Job tracking. Restarts after errors etc. $\endgroup$ – Ian Sudbery Apr 23 '18 at 9:49
  • $\begingroup$ @heathobrien Hi, The above code uses one after another tar.gz file. It is taking huge time. How should I make the code work for all the tar.gz files at a time? $\endgroup$ – user3351523 Apr 30 '18 at 6:26
  • $\begingroup$ see updated answer $\endgroup$ – heathobrien Apr 30 '18 at 7:12
  • $\begingroup$ Even I haven't worked on SLURM before. Do you think if I give #SBATCH --array=1-6%3 will work? lets say I have 6 tar.gz files so 3 jobs at a time. $\endgroup$ – user3351523 Apr 30 '18 at 11:48
1
$\begingroup$

Your samples end with _1.fastq and _2.fastq, so you need to change this:

-1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq

to this:

-1 $TMPDIR/${base}_1.fastq -2 $TMPDIR/${base}_2.fastq
$\endgroup$
  • $\begingroup$ Thanq. ya this was the only error in my code. $\endgroup$ – user3351523 Apr 23 '18 at 11:51
  • $\begingroup$ Hi, The above code uses one after another tar.gz file. It is taking huge time. How should I make the code work for all the tar.gz files at a time? $\endgroup$ – user3351523 Apr 30 '18 at 6:27
  • $\begingroup$ Untar the files and use a workflow manager like snakemake. BTW, make sure you have sufficient resources to process many samples in parallel. $\endgroup$ – Devon Ryan Apr 30 '18 at 6:36
  • $\begingroup$ But as soon as Untar done for one file it should be the input for hisat2. Yes, I'm using 20 cpu cores, 1 week time and tmp 500G, memory usage 4G. $\endgroup$ – user3351523 Apr 30 '18 at 6:43
  • $\begingroup$ Then put the untar command into something like Snakemake too. $\endgroup$ – Devon Ryan Apr 30 '18 at 6:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.