Hi im doing Variant calling on fastq files for which i have 4000 fq files and the variant calling are done in different 9 steps. Each step generate different files that are output to next steps.

So can anybody help me with this to do it in a loop it pick up names from previous steps for all files. Step 1 command is here:

bwa mem -M -R '@RG\tID:ERR040140\tLB:2517707\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR040140' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR040140_* > ERR040140.sam

So i made an excel file to break the command in columns so that i can change the file names keeping remaining command same just like this:


So changes are occurring 5th, 7th and 9th column. which are:


Can anyone help me with loop or something?

Thank you!

  • $\begingroup$ Please edit your question and replace the image of the excel file with a copy/paste of the same file exported to tsv (save as 'tab separated text file'). That way we can actually copy it and use it to test our examples. $\endgroup$ – terdon Dec 11 '18 at 11:13
  • $\begingroup$ Please remember to confirm an answer once you've received one. $\endgroup$ – Stephopolis Dec 11 '18 at 18:38
  • 1
    $\begingroup$ I have already given the excel file command above. And im unable to paste it here the page is not letting me paste it that is why i quoted 1 example and than added a picture with all other repeating commands with different file names. $\endgroup$ – Safina A.R Dec 12 '18 at 4:23

Another solution for repeating one command with different parameters is to use parallel.

Create a tab-delimited file samples.txt, which contains the sample names and the information for your read groups, e.g.:

Sample1 Lib1
Sample2 Lib2
Sample3 Lib3

Then you can use parallel like this:

parallel --colsep "\t" 'bwa mem -M -R "@RG\tID:{1}\tLB:{2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:{1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta {1}_* > {1}.sam' :::: samples.txt

For each line in samples.txt a new process is started, replacing {1} and {2} with the values in column 1 and 2 of this line.

fin swimmer

  • $\begingroup$ wow that's look much easier i will try and update you. $\endgroup$ – Safina A.R Dec 11 '18 at 13:15
  • $\begingroup$ swimmer thank you for this simplest code. I want to know whether this parallel produce log? because i cant see anything on the screen as the bwa command produce screen messages while running. Secondly, if i don't want to run these commands parallel than what should i used instead of parallel as while running it parallel my computer got stuck. $\endgroup$ – Safina A.R Dec 12 '18 at 7:57
  • $\begingroup$ By default parallel prints to the screen after a job has finished. Doing this makes sure that you can e.g. concatenate outputs of parallel jobs without mixing them. You can use the option --ungroup to get the output immediately. But use this with care! Without any parameter the number of parallel jobs is the same as the number of CPU cores. You can set it to an explicit number using -j, e.g. -j 1 $\endgroup$ – finswimmer Dec 12 '18 at 8:57
  • $\begingroup$ Ok. Thank you. It is running for this command i will modified it accordingly for other steps and if i got stuck i will update here. $\endgroup$ – Safina A.R Dec 12 '18 at 12:35

Something like this might do it:


# make an array to hold your list of samples/base filenames
samples=( S1 S2 S3 S4 S5 )

# make a loop to go through the sample IDs
for sample in "${samples[@]}"
  # do something with each sample/file, e.g. make a directory for it
  mkdir "${sample}"_home

  # do something else with the sample, e.g. unzip the fastq.gz file
  gunzip "${sample}".fq.gz > "${sample}"_home/"${sample}".fq

One disadvantage with this is that it will process the files one after the other. If you have a cluster, you might prefer to run them all at the same time to make the whole thing quicker. If you have a SLURM cluster, you can use job array to do something similar.

If you have your list of samples in a text file, you could try replacing the samples array command in the script above with something like this:

samples=( `cat samples_list.txt` )

This will only work if the samples are listed on a single line, space-separated. You may need to process your samples file further to get them in this format, or use the bash readarray command.

  • $\begingroup$ I have a list of sample ids store in a txt so is it possible that it make a list using that file? $\endgroup$ – Safina A.R Dec 11 '18 at 13:14
  • $\begingroup$ @SafinaA.R please remember to actually show your data. Edit your question and show us this text file. That way, we will know the format and can use it to give you example answers. $\endgroup$ – terdon Dec 11 '18 at 13:26
  • $\begingroup$ This solution requires saving the sample names directly into the bash script. This sounds a little difficult since OP said she has a file with 4000 sample names. $\endgroup$ – conchoecia Dec 11 '18 at 16:55
  • $\begingroup$ hi @terdon The data is same as shown by fin swimmer in the above example. It is a text file tab separated having two col. Sample Ids and Library Ids $\endgroup$ – Safina A.R Dec 12 '18 at 4:16
  • $\begingroup$ I haven't tried this, but I added a comment to my script above about how to get the sample IDs out of a text file. $\endgroup$ – Jonathan Moore Jan 4 at 16:30

if you have a repeating workflow, I strongly recommend to have a look at workflow management systems like snakemake. I've also wrote a little tutorial on biostars about this topic, which might be useful for you.

fin swimmer

  • $\begingroup$ That's a bit complicated can there be simple for loop trick? $\endgroup$ – Safina A.R Dec 11 '18 at 11:04

You can create a script to pass arguments to as follows:

#! /bin/bash

bwa mem -M -R "@RG\tID:${1}\tLB:${2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:${1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ${1}_* > ${1}.sam

Now if you run bash script.sh sample lib then sample and lib will be the script as arguments ${1} and ${2} respectively. Then all you need is a loop to pass arguments to the script:

for sample in sample1 sample2 sample3
   for lib in lib1 lib2 lib3
     bash script.sh $sample $lib

You can use nohup to run each sample in the background and proceed with next one. This will in essence, submit each command as a separate job to run on a different core. These will print to separate log files sample.out instead of to the terminal. It will also continue to run, even if you disconnect from SSH or close the terminal.

for sample in sample1 sample2 sample3
   for lib in lib1 lib2 lib3
     nohup bash script.sh $sample $lib > ${sample}.out &

You should only do this for single-threaded processes. For multi-threaded processes or memory-intensive jobs, this will demand a lot from the machine. Even if this isn’t a shared server, it will perform poorly if you run out of memory and have to “swap” memory by writing to disk. So it’s best to submit them one-by-one (above) or use a cluster with a job scheduler.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.