How to run same command on multiple files?

Question

Hi im doing Variant calling on fastq files for which i have 4000 fq files and the variant calling are done in different 9 steps. Each step generate different files that are output to next steps.

So can anybody help me with this to do it in a loop it pick up names from previous steps for all files. Step 1 command is here:

bwa mem -M -R '@RG\tID:ERR040140\tLB:2517707\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR040140' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR040140_* > ERR040140.sam

So i made an excel file to break the command in columns so that i can change the file names keeping remaining command same just like this:

So changes are occurring 5th, 7th and 9th column. which are:

tID:ERR040140
tLB:2517707
tSM:ERR040140
ERR040140_*
ERR040140.sam

Can anyone help me with loop or something?

Thank you!

Answer 1

Another solution for repeating one command with different parameters is to use parallel.

Create a tab-delimited file samples.txt, which contains the sample names and the information for your read groups, e.g.:

Sample1 Lib1
Sample2 Lib2
Sample3 Lib3

Then you can use parallel like this:

parallel --colsep "\t" 'bwa mem -M -R "@RG\tID:{1}\tLB:{2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:{1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta {1}_* > {1}.sam' :::: samples.txt

For each line in samples.txt a new process is started, replacing {1} and {2} with the values in column 1 and 2 of this line.

fin swimmer

Answer 2

Something like this might do it:

#!/bin/bash

# make an array to hold your list of samples/base filenames
samples=( S1 S2 S3 S4 S5 )

# make a loop to go through the sample IDs
for sample in "${samples[@]}"
do
  :
  # do something with each sample/file, e.g. make a directory for it
  mkdir "${sample}"_home

  # do something else with the sample, e.g. unzip the fastq.gz file
  gunzip "${sample}".fq.gz > "${sample}"_home/"${sample}".fq
done

One disadvantage with this is that it will process the files one after the other. If you have a cluster, you might prefer to run them all at the same time to make the whole thing quicker. If you have a SLURM cluster, you can use job array to do something similar.

If you have your list of samples in a text file, you could try replacing the samples array command in the script above with something like this:

samples=( `cat samples_list.txt` )

This will only work if the samples are listed on a single line, space-separated. You may need to process your samples file further to get them in this format, or use the bash readarray command.

Answer 3

if you have a repeating workflow, I strongly recommend to have a look at workflow management systems like snakemake. I've also wrote a little tutorial on biostars about this topic, which might be useful for you.

fin swimmer

Answer 4

You can create a script to pass arguments to as follows:

#! /bin/bash

bwa mem -M -R "@RG\tID:${1}\tLB:${2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:${1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ${1}_* > ${1}.sam

Now if you run bash script.sh sample lib then sample and lib will be the script as arguments ${1} and ${2} respectively. Then all you need is a loop to pass arguments to the script:

for sample in sample1 sample2 sample3
   do
   for lib in lib1 lib2 lib3
     do
     bash script.sh $sample $lib
     done
   done

You can use nohup to run each sample in the background and proceed with next one. This will in essence, submit each command as a separate job to run on a different core. These will print to separate log files sample.out instead of to the terminal. It will also continue to run, even if you disconnect from SSH or close the terminal.

for sample in sample1 sample2 sample3
   do
   for lib in lib1 lib2 lib3
     do
     nohup bash script.sh $sample $lib > ${sample}.out &
     done
   done

You should only do this for single-threaded processes. For multi-threaded processes or memory-intensive jobs, this will demand a lot from the machine. Even if this isn’t a shared server, it will perform poorly if you run out of memory and have to “swap” memory by writing to disk. So it’s best to submit them one-by-one (above) or use a cluster with a job scheduler.