0
$\begingroup$

I have a bash script which extracts some information from a .vcf file but one .vcf file at each time. How I can change this script in a way to work on a bunch of .vcf files and the output is a .txt file named by the name of original .vcf

This is my script

help please


EDITED

 #!/usr/bash

#outfilename
outname=$".parsed.txt"

#Header for output file
echo -e "$outname\t$chrom\t$Pos\t$Ref\t$Alt\t$ReadCount\t$VariantAlleleCount\t$ReferenceAlleleCount\t$ReadCountControl\t$VariantAlleleCountControl\t$ReferenceAlleleCountControl\t$VAF" >> $outname ;
while read -r line ; 
do

#Basic information
chrom=$(echo "$line" | sed 's/ /\t/g' | cut -f 1) #&& echo $chrom;
Pos=$(echo "$line" | sed 's/ /\t/g' |  cut -f 2) #&& echo $Pos;
Ref=$(echo "$line" | sed 's/ /\t/g' | cut -f 4)
Alt=$(echo "$line" | sed 's/ /\t/g' | cut -f 5)

#Tumor sample read, variant and reference information
ReadCount=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 13 | sed 's/ReadCount=//' )
VariantAlleleCount=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 26| sed 's/VariantAlleleCount=//')
ReferenceAlleleCount=$(echo $ line | awk -v rc="$ReadCount" -v vac="$VariantAlleleCount" '{print rc-vac}')


#Control or Normal read, variant, reference information
ReadCountControl=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 14 | sed 's/ReadCountControl=//')
VariantAlleleCountControl=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 27 | sed 's/VariantAlleleCountControl=//')   

ReferenceAlleleCountControl=$(echo "$line" | awk -v rcc="$ReadCountControl" -v vacc="$VariantAlleleCountControl" '{print rcc-vacc}')


VAF=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 28 | sed 's/VariantAlleleFrequency=//')
#Print output
echo -e "$outname\t""$chrom"'\t'"$Pos"'\t'"$Ref"'\t'"$Alt"'\t'"$ReadCount"'\t'"$VariantAlleleCount"'\t'"$ReferenceAlleleCount"'\t'"$ReadCountControl"'\t'"$VariantAlleleCountControl"'\t'"$ReferenceAlleleCountControl"'\t'"$VAF" >> $outname ;

#Remove info tags from VCF
done; < <( egrep -v '#' "$1")

And the error I obtained

[fi1d18@cyan01 new]$ bash vcf_parasing.sh
vcf_parasing.sh: line 7: syntax error near unexpected token `echo'
vcf_parasing.sh: line 7: `echo -e $outname'\t' "Chrom"'\t'"Position"'\t'"Ref"'\t'"Alt"'\t'"TumorReadCount"'\t'"TumorVariantAlleleCount"'\t'"TumorReferenceAlleleCount"'\t'"NormalReadCount"'\t'"NormalVariantAlleleCount"'\t'"NormalReferenceAlleleCount"'\t'"VAF" > $outname'
[fi1d18@cyan01 new]$

My script

https://www.dropbox.com/s/di49mij2a43i43s/vcf_parasing.sh?dl=0

Two samples of .vcf

https://www.dropbox.com/s/ryk7q0aybeo5cy2/LP6008202-DNA_B03_vs_LP6008201-DNA_B03.snp.pass.vcf?dl=0

https://www.dropbox.com/s/u44s6d54r5d57bu/LP6008336-DNA_H02_vs_LP6008333-DNA_H02.snp.pass.vcf?dl=0

$\endgroup$
3
  • 1
    $\begingroup$ Doing this sort of thing with a bash loop over every line is a really, really bad idea. It will be slower than just about any other method, the syntax is hard to read (and I say this as someone with considerable experience with shell scripts) and the whole thing will be more painful than needed. If you could edit your question and explain exactly what information you are trying to extract, I'm sure we could help you find a better way. $\endgroup$
    – terdon
    Jan 1 '20 at 13:30
  • $\begingroup$ Thank you @terdon. This script extract some fields from .vcf file output of Strelka mutation caller. the problem is that except for the some columns, the location of the other information is of easily detecting $\endgroup$
    – Exhausted
    Jan 1 '20 at 14:46
  • 1
    $\begingroup$ Note that you don't need all of those quotes for the echo. Just use echo -e "$outname\t$chrom\t$Pos\t$Ref\t$Alt\t$ReadCount\t$VariantAlleleCount\t$ReferenceAlleleCount\t$ReadCountControl\t$VariantAlleleCountControl\t$ReferenceAlleleCountControl\t$VAF" >> $outname ; $\endgroup$
    – terdon
    Jan 1 '20 at 14:59
5
$\begingroup$

Using shell loops for text practice is considered bad practice. It is exceedingly slow, the syntax is complicated so it's very easy to get it wrong and it's just painful. The shell isn't designed as a proper scripting language, so while it can be (ab)used that way, you really should avoid it whenever possible.

Here's a Perl script that should do exactly what your bash script does but will be several times faster and can deal with multiple input files:

#!/usr/bin/env perl

## Iterate over all files given at the command line
foreach my $inFile (@ARGV) {
  print STDERR "Processing $inFile...\n";
  my $outFile = "$inFile.parsed.txt";
  $outFile =~ m|([^/]+)$|;
  #/#Useless comment to fix SE syntax highlighting
  my $outFileName = $1;
  open(my $outFileHandle, '>', $outFile) or
      die("Couldn't open '$outFile' for writing: $!");
  open(my $inFileHandle, '<', $inFile) or
      die("Couldn't open '$inFile' for writing: $!");

  ## Print header
  print $outFileHandle "$outFileName\tChrom\tPosition\tRef\tAlt\tTumorReadCount\t" .
      "TumorVariantAlleleCount\tTumorReferenceAlleleCount\tNormalReadCount" .
          "\tNormalVariantAlleleCount\tNormalReferenceAlleleCount\tVAF\n";
  ## Read all lines of the input file
  while (<$inFileHandle>) {
    ## Skip headers
    next if /^#/;
    ## Get VCF fields
    my ($chrom, $pos, $name, $ref, $alt, $qual,
        $filter, $info, $format, $values) = split(/\t/);
    my @infoFields = split(/;/, $info);
    my $readCount = $infoFields[12];
    $readCount =~ s/ReadCount=//;
    #/#Useless comment to fix SE syntax highlighting

    my $variantAlleleCount = $infoFields[25];
    $variantAlleleCount =~ s/VariantAlleleCount=//;
    #/#Useless comment to fix SE syntax highlighting
    my $referenceAlleleCount= $readCount - $variantAlleleCount;

    my $readCountControl = $infoFields[13];
    $readCountControl =~ s/ReadCountControl=//;
    #/#Useless comment to fix SE syntax highlighting

    my $variantAlleleCountControl = $infoFields[26];
    $variantAlleleCountControl =~ s/VariantAlleleCountControl=//;
    #/#Useless comment to fix SE syntax highlighting

    my $referenceAlleleCountControl = $readCountControl - $variantAlleleCountControl;
    my $VAF = $infoFields[27];
    $VAF =~ s/VariantAlleleFrequency=//;
    #/#Useless comment to fix SE syntax highlighting

    my $outString = join("\t", $outFileName, $chrom, $pos, $ref, $alt, $readCount,
                         $variantAlleleCount, $referenceAlleleCount, $readCountControl,
                         $variantAlleleCountControl, $referenceAlleleCountControl, $VAF);
    print $outFileHandle "$outString\n";
  }
  close($outFileHandle);
  close($inFileHandle);
}

Save that file as parser.pl and then run it on all your vcf files with:

perl parser.pl /path/to/vcfs/*vcf

That should create a .parsed.txt file for each input file in the target directory.


If you just have to use the shell for this, you can try with this improved version of your script:

#!/usr/bin/env bash

for file in "$@"; do
  outFile="${file}.parsed.txt"
  outFileName=$(basename "$outFile")
  (
    printf '%s\tChrom\tPosition\tRef\tAlt\tTumorReadCount\t' "$outFileName"
    printf 'TumorVariantAlleleCount\tTumorReferenceAlleleCount\tNormalReadCount\t'
    printf 'NormalVariantAlleleCount\tNormalReferenceAlleleCount\tVAF\n'
  ) > "$outFile"

  echo "Reading $file"
  grep -v '^#' "$file" |
    while read -r line; do
      read chrom Pos Ref Alt rest < <(awk -F"\t" '{print $1,$2,$4,$5}' <<<"$line")
      ReadCount=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 13 | sed 's/ReadCount=//' )
      VariantAlleleCount=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 26| sed 's/VariantAlleleCount=//')
      ReferenceAlleleCount=$(($ReadCount - $VariantAlleleCount))

      ReadCountControl=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 14 | sed 's/ReadCountControl=//')
      VariantAlleleCountControl=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 27 | sed 's/VariantAlleleCountControl=//')   

      ReferenceAlleleCountControl=$(($ReadCountControl - $VariantAlleleCountControl))

      VAF=$(echo "$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 28 | sed 's/VariantAlleleFrequency=//')
      printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$outFileName" "$chrom" "$Pos" "$Ref" "$Alt" \
             "$ReadCount" "$VariantAlleleCount" "$ReferenceAlleleCount" "$ReadCountControl" "$VariantAlleleCountControl" \
             "$ReferenceAlleleCountControl" "$VAF" >> "$outFile" 
    done
done

Just to illustrate what I mean about the shell being slow, I ran both scripts above on the example files you provided:

$ time foo.pl *vcf 2>/dev/null

real    0m0.051s
user    0m0.047s
sys     0m0.003s

$ time bar.sh *vcf 2>/dev/null

real    2m28.842s
user    3m33.648s
sys     1m12.456s

As you can see above, the perl script took less than a second, while the bash script took almost two and a half minutes!


Finally, if you find the perl script too long, here's the same thing in a more condensed style:

#!/usr/bin/env perl

## Iterate over all files given at the command line
foreach my $inFile (@ARGV) {
  print STDERR "Processing $inFile...\n";
  my $outFile = "$inFile.parsed.txt";
  $outFile =~ m|([^/]+)$|;
  #/#Useless comment to fix SE syntax highlighting
  my $outFileName = $1;
  open(my $outFileHandle, '>', $outFile);
  open(my $inFileHandle, '<', $inFile);

  ## Print header
  print $outFileHandle "$outFileName\tChrom\tPosition\tRef\tAlt\tTumorReadCount\t" .
      "TumorVariantAlleleCount\tTumorReferenceAlleleCount\tNormalReadCount" .
          "\tNormalVariantAlleleCount\tNormalReferenceAlleleCount\tVAF\n";
  ## Read all lines of the input file
  while (<$inFileHandle>) {
    ## Skip headers
    next if /^#/;
    ## Get VCF fields
    my ($chrom, $pos, $name, $ref, $alt, $qual,
        $filter, $info, $format, $values) = split(/\t/);
    $info=~s/[^;]+=//g;
    my @infoFields = split(/;/, $info);
    print $outFileHandle (join "\t", $outFileName, $chrom, $pos, $ref, $alt,
                               $infoFields[12],$infoFields[25],
                               $infoFields[12]-$infoFields[25],
                               $infoFields[13],$infoFields[26],
                               $infoFields[13]-$infoFields[26],
                               $infoFields[27]) . "\n";
  }
}
$\endgroup$
6
  • $\begingroup$ Oh my God, this is amazing. I am wondering when some people put their time and efforts to helping the other people this nicely . The perl done that in a second $\endgroup$
    – Exhausted
    Jan 1 '20 at 17:16
  • $\begingroup$ Sorry @terdon your code works on Single Nucleotide variants (SNVs) calls very well but on INDELs returns empty fields that may be because for example the position of ReferenceAlleleCount in SNV and INDEL are not the same. This is link to one of my INDEL files coming from the same Strelka variant caller sotonac-my.sharepoint.com/personal/fi1d18_soton_ac_uk/Documents/… I am not sure even I can post this comment or not because surely I see myself too annoying $\endgroup$
    – Exhausted
    Jan 3 '20 at 4:56
  • $\begingroup$ @Angel I can't access that file. It's probably best if you ask a new question, and post some lines from the bad files. Or, if the position is different, just try modifying the script and changing the numbers in the various $infofields[N]. $\endgroup$
    – terdon
    Jan 3 '20 at 9:15
  • $\begingroup$ Thank you @terdon, I have asked this as a new question here bioinformatics.stackexchange.com/questions/11074/… as I found the script specific to INDELs differs to what works for SNVs. If possible have a look please; thank you in advance $\endgroup$
    – Exhausted
    Jan 4 '20 at 16:40
  • $\begingroup$ @Angel you will need to explain what information you need. Since your script doesn't work, that won't help us understand your objective. So, show us your expected output from a few lines of example input. $\endgroup$
    – terdon
    Jan 4 '20 at 16:43
1
$\begingroup$

Here is a quick and kind of dirty solution with minimum intervention to the code:

  • Add $outname'\t' right after echo -e to two two lines, the lines after #Header for output file and #Print output. The former will alter your header and the latter your actual output: you will have a first column with file name information.
  • Then you can run this script in a for loop to generate multiple output files which then can be merged into a single file with cat.

Note that with this workaround, you will have multiple headers in the merged file. If you need a clean output, you will need to put your while loop in a for loop.

$\endgroup$
3
  • $\begingroup$ Thank you so much @haci . I edited the script as you suggested adding for loop to that in my main post but I obtained the same error posted in my main post $\endgroup$
    – Exhausted
    Jan 1 '20 at 14:43
  • 1
    $\begingroup$ What happens when you get rid of the whitespace in the $outname'\t' $chrom bit (remember there are two of these for the two lines that were altered from the original code) and the quotes as @terdon suggested in their comment? $\endgroup$
    – haci
    Jan 1 '20 at 15:10
  • $\begingroup$ Sorry I tried all of your comments but I failed I have attached my original script and two vcf files here maybe it is better than pasting the code here $\endgroup$
    – Exhausted
    Jan 1 '20 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.