Changing this code in a way to work for my files

Question

I have a bash script which extracts some information from a .vcf file but one .vcf file at each time. How I can change this script in a way to work on a bunch of .vcf files and the output is a .txt file named by the name of original .vcf

This is my script

help please

---

EDITED

 #!/usr/bash

#outfilename
outname=$".parsed.txt"

#Header for output file
echo -e "$outname\t$chrom\t$Pos\t$Ref\t$Alt\t$ReadCount\t$VariantAlleleCount\t$ReferenceAlleleCount\t$ReadCountControl\t$VariantAlleleCountControl\t$ReferenceAlleleCountControl\t$VAF" >> $outname ;
while read -r line ; 
do

#Basic information
chrom=$(echo "$line" | sed 's/ /\t/g' | cut -f 1) #&& echo $chrom;
Pos=$(echo "$line" | sed 's/ /\t/g' |  cut -f 2) #&& echo $Pos;
Ref=$(echo "$line" | sed 's/ /\t/g' | cut -f 4)
Alt=$(echo "$line" | sed 's/ /\t/g' | cut -f 5)

#Tumor sample read, variant and reference information
ReadCount=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 13 | sed 's/ReadCount=//' )
VariantAlleleCount=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 26| sed 's/VariantAlleleCount=//')
ReferenceAlleleCount=$(echo $ line | awk -v rc="$ReadCount" -v vac="$VariantAlleleCount" '{print rc-vac}')


#Control or Normal read, variant, reference information
ReadCountControl=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 14 | sed 's/ReadCountControl=//')
VariantAlleleCountControl=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 27 | sed 's/VariantAlleleCountControl=//')   

ReferenceAlleleCountControl=$(echo "$line" | awk -v rcc="$ReadCountControl" -v vacc="$VariantAlleleCountControl" '{print rcc-vacc}')


VAF=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 28 | sed 's/VariantAlleleFrequency=//')
#Print output
echo -e "$outname\t""$chrom"'\t'"$Pos"'\t'"$Ref"'\t'"$Alt"'\t'"$ReadCount"'\t'"$VariantAlleleCount"'\t'"$ReferenceAlleleCount"'\t'"$ReadCountControl"'\t'"$VariantAlleleCountControl"'\t'"$ReferenceAlleleCountControl"'\t'"$VAF" >> $outname ;

#Remove info tags from VCF
done; < <( egrep -v '#' "$1")

---

And the error I obtained

[fi1d18@cyan01 new]$ bash vcf_parasing.sh
vcf_parasing.sh: line 7: syntax error near unexpected token `echo'
vcf_parasing.sh: line 7: `echo -e $outname'\t' "Chrom"'\t'"Position"'\t'"Ref"'\t'"Alt"'\t'"TumorReadCount"'\t'"TumorVariantAlleleCount"'\t'"TumorReferenceAlleleCount"'\t'"NormalReadCount"'\t'"NormalVariantAlleleCount"'\t'"NormalReferenceAlleleCount"'\t'"VAF" > $outname'
[fi1d18@cyan01 new]$

My script

https://www.dropbox.com/s/di49mij2a43i43s/vcf_parasing.sh?dl=0

Two samples of .vcf

https://www.dropbox.com/s/ryk7q0aybeo5cy2/LP6008202-DNA_B03_vs_LP6008201-DNA_B03.snp.pass.vcf?dl=0

https://www.dropbox.com/s/u44s6d54r5d57bu/LP6008336-DNA_H02_vs_LP6008333-DNA_H02.snp.pass.vcf?dl=0

Answer 1

Using shell loops for text practice is considered bad practice. It is exceedingly slow, the syntax is complicated so it's very easy to get it wrong and it's just painful. The shell isn't designed as a proper scripting language, so while it can be (ab)used that way, you really should avoid it whenever possible.

Here's a Perl script that should do exactly what your bash script does but will be several times faster and can deal with multiple input files:

#!/usr/bin/env perl

## Iterate over all files given at the command line
foreach my $inFile (@ARGV) {
  print STDERR "Processing $inFile...\n";
  my $outFile = "$inFile.parsed.txt";
  $outFile =~ m|([^/]+)$|;
  #/#Useless comment to fix SE syntax highlighting
  my $outFileName = $1;
  open(my $outFileHandle, '>', $outFile) or
      die("Couldn't open '$outFile' for writing: $!");
  open(my $inFileHandle, '<', $inFile) or
      die("Couldn't open '$inFile' for writing: $!");

  ## Print header
  print $outFileHandle "$outFileName\tChrom\tPosition\tRef\tAlt\tTumorReadCount\t" .
      "TumorVariantAlleleCount\tTumorReferenceAlleleCount\tNormalReadCount" .
          "\tNormalVariantAlleleCount\tNormalReferenceAlleleCount\tVAF\n";
  ## Read all lines of the input file
  while (<$inFileHandle>) {
    ## Skip headers
    next if /^#/;
    ## Get VCF fields
    my ($chrom, $pos, $name, $ref, $alt, $qual,
        $filter, $info, $format, $values) = split(/\t/);
    my @infoFields = split(/;/, $info);
    my $readCount = $infoFields[12];
    $readCount =~ s/ReadCount=//;
    #/#Useless comment to fix SE syntax highlighting

    my $variantAlleleCount = $infoFields[25];
    $variantAlleleCount =~ s/VariantAlleleCount=//;
    #/#Useless comment to fix SE syntax highlighting
    my $referenceAlleleCount= $readCount - $variantAlleleCount;

    my $readCountControl = $infoFields[13];
    $readCountControl =~ s/ReadCountControl=//;
    #/#Useless comment to fix SE syntax highlighting

    my $variantAlleleCountControl = $infoFields[26];
    $variantAlleleCountControl =~ s/VariantAlleleCountControl=//;
    #/#Useless comment to fix SE syntax highlighting

    my $referenceAlleleCountControl = $readCountControl - $variantAlleleCountControl;
    my $VAF = $infoFields[27];
    $VAF =~ s/VariantAlleleFrequency=//;
    #/#Useless comment to fix SE syntax highlighting

    my $outString = join("\t", $outFileName, $chrom, $pos, $ref, $alt, $readCount,
                         $variantAlleleCount, $referenceAlleleCount, $readCountControl,
                         $variantAlleleCountControl, $referenceAlleleCountControl, $VAF);
    print $outFileHandle "$outString\n";
  }
  close($outFileHandle);
  close($inFileHandle);
}

Save that file as parser.pl and then run it on all your vcf files with:

perl parser.pl /path/to/vcfs/*vcf

That should create a .parsed.txt file for each input file in the target directory.

---

If you just have to use the shell for this, you can try with this improved version of your script:

#!/usr/bin/env bash

for file in "$@"; do
  outFile="${file}.parsed.txt"
  outFileName=$(basename "$outFile")
  (
    printf '%s\tChrom\tPosition\tRef\tAlt\tTumorReadCount\t' "$outFileName"
    printf 'TumorVariantAlleleCount\tTumorReferenceAlleleCount\tNormalReadCount\t'
    printf 'NormalVariantAlleleCount\tNormalReferenceAlleleCount\tVAF\n'
  ) > "$outFile"

  echo "Reading $file"
  grep -v '^#' "$file" |
    while read -r line; do
      read chrom Pos Ref Alt rest < <(awk -F"\t" '{print $1,$2,$4,$5}' <<<"$line")
      ReadCount=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 13 | sed 's/ReadCount=//' )
      VariantAlleleCount=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 26| sed 's/VariantAlleleCount=//')
      ReferenceAlleleCount=$(($ReadCount - $VariantAlleleCount))

      ReadCountControl=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 14 | sed 's/ReadCountControl=//')
      VariantAlleleCountControl=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 27 | sed 's/VariantAlleleCountControl=//')   

      ReferenceAlleleCountControl=$(($ReadCountControl - $VariantAlleleCountControl))

      VAF=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 28 | sed 's/VariantAlleleFrequency=//')
      printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$outFileName" "$chrom" "$Pos" "$Ref" "$Alt" \
             "$ReadCount" "$VariantAlleleCount" "$ReferenceAlleleCount" "$ReadCountControl" "$VariantAlleleCountControl" \
             "$ReferenceAlleleCountControl" "$VAF" >> "$outFile" 
    done
done

---

Just to illustrate what I mean about the shell being slow, I ran both scripts above on the example files you provided:

$ time foo.pl *vcf 2>/dev/null

real    0m0.051s
user    0m0.047s
sys     0m0.003s

$ time bar.sh *vcf 2>/dev/null

real    2m28.842s
user    3m33.648s
sys     1m12.456s

As you can see above, the perl script took less than a second, while the bash script took almost two and a half minutes!

---

Finally, if you find the perl script too long, here's the same thing in a more condensed style:

#!/usr/bin/env perl

## Iterate over all files given at the command line
foreach my $inFile (@ARGV) {
  print STDERR "Processing $inFile...\n";
  my $outFile = "$inFile.parsed.txt";
  $outFile =~ m|([^/]+)$|;
  #/#Useless comment to fix SE syntax highlighting
  my $outFileName = $1;
  open(my $outFileHandle, '>', $outFile);
  open(my $inFileHandle, '<', $inFile);

  ## Print header
  print $outFileHandle "$outFileName\tChrom\tPosition\tRef\tAlt\tTumorReadCount\t" .
      "TumorVariantAlleleCount\tTumorReferenceAlleleCount\tNormalReadCount" .
          "\tNormalVariantAlleleCount\tNormalReferenceAlleleCount\tVAF\n";
  ## Read all lines of the input file
  while (<$inFileHandle>) {
    ## Skip headers
    next if /^#/;
    ## Get VCF fields
    my ($chrom, $pos, $name, $ref, $alt, $qual,
        $filter, $info, $format, $values) = split(/\t/);
    $info=~s/[^;]+=//g;
    my @infoFields = split(/;/, $info);
    print $outFileHandle (join "\t", $outFileName, $chrom, $pos, $ref, $alt,
                               $infoFields[12],$infoFields[25],
                               $infoFields[12]-$infoFields[25],
                               $infoFields[13],$infoFields[26],
                               $infoFields[13]-$infoFields[26],
                               $infoFields[27]) . "\n";
  }
}

Answer 2

Here is a quick and kind of dirty solution with minimum intervention to the code:

• Add $outname'\t' right after echo -e to two two lines, the lines after #Header for output file and #Print output. The former will alter your header and the latter your actual output: you will have a first column with file name information.
• Then you can run this script in a for loop to generate multiple output files which then can be merged into a single file with cat.

Note that with this workaround, you will have multiple headers in the merged file. If you need a clean output, you will need to put your while loop in a for loop.