# Changing this code in a way to work for my files

I have a bash script which extracts some information from a .vcf file but one .vcf file at each time. How I can change this script in a way to work on a bunch of .vcf files and the output is a .txt file named by the name of original .vcf

This is my script

EDITED

 #!/usr/bash

#outfilename
outname=$".parsed.txt" #Header for output file echo -e "$$outname\t$$chrom\t$$Pos\t$$Ref\t$$Alt\t$$ReadCount\t$$VariantAlleleCount\t$$ReferenceAlleleCount\t$$ReadCountControl\t$$VariantAlleleCountControl\t$$ReferenceAlleleCountControl\t$$VAF" >>$outname ;
do

#Basic information
chrom=$$(echo "$$line" | sed 's/ /\t/g' | cut -f 1) #&& echo $$chrom; Pos=$$(echo "$$line" | sed 's/ /\t/g' | cut -f 2) #&& echo$$Pos;
Ref=$$(echo "$$line" | sed 's/ /\t/g' | cut -f 4)
Alt=$$(echo "$$line" | sed 's/ /\t/g' | cut -f 5)

#Tumor sample read, variant and reference information
ReadCount=$$(echo "$$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 13 | sed 's/ReadCount=//' )
VariantAlleleCount=$$(echo "$$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 26| sed 's/VariantAlleleCount=//')
ReferenceAlleleCount=$$(echo$$ line | awk -v rc="$$ReadCount" -v vac="$$VariantAlleleCount" '{print rc-vac}')

#Control or Normal read, variant, reference information
ReadCountControl=$$(echo "$$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 14 | sed 's/ReadCountControl=//')
VariantAlleleCountControl=$$(echo "$$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 27 | sed 's/VariantAlleleCountControl=//')

ReferenceAlleleCountControl=$$(echo "$$line" | awk -v rcc="$$ReadCountControl" -v vacc="$$VariantAlleleCountControl" '{print rcc-vacc}')

VAF=$$(echo "$$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 28 | sed 's/VariantAlleleFrequency=//')
#Print output
echo -e "$$outname\t""$$chrom"'\t'"$$Pos"'\t'"$$Ref"'\t'"$$Alt"'\t'"$$ReadCount"'\t'"$$VariantAlleleCount"'\t'"$$ReferenceAlleleCount"'\t'"$$ReadCountControl"'\t'"$$VariantAlleleCountControl"'\t'"$$ReferenceAlleleCountControl"'\t'"$$VAF" >> $outname ; #Remove info tags from VCF done; < <( egrep -v '#' "$1")


And the error I obtained

[fi1d18@cyan01 new]$$bash vcf_parasing.sh vcf_parasing.sh: line 7: syntax error near unexpected token echo' vcf_parasing.sh: line 7: echo -e$$outname'\t' "Chrom"'\t'"Position"'\t'"Ref"'\t'"Alt"'\t'"TumorReadCount"'\t'"TumorVariantAlleleCount"'\t'"TumorReferenceAlleleCount"'\t'"NormalReadCount"'\t'"NormalVariantAlleleCount"'\t'"NormalReferenceAlleleCount"'\t'"VAF" > $$outname' [fi1d18@cyan01 new]$$


My script

https://www.dropbox.com/s/di49mij2a43i43s/vcf_parasing.sh?dl=0

Two samples of .vcf

https://www.dropbox.com/s/ryk7q0aybeo5cy2/LP6008202-DNA_B03_vs_LP6008201-DNA_B03.snp.pass.vcf?dl=0

https://www.dropbox.com/s/u44s6d54r5d57bu/LP6008336-DNA_H02_vs_LP6008333-DNA_H02.snp.pass.vcf?dl=0

• Doing this sort of thing with a bash loop over every line is a really, really bad idea. It will be slower than just about any other method, the syntax is hard to read (and I say this as someone with considerable experience with shell scripts) and the whole thing will be more painful than needed. If you could edit your question and explain exactly what information you are trying to extract, I'm sure we could help you find a better way. Jan 1 '20 at 13:30
• Thank you @terdon. This script extract some fields from .vcf file output of Strelka mutation caller. the problem is that except for the some columns, the location of the other information is of easily detecting Jan 1 '20 at 14:46
• Note that you don't need all of those quotes for the echo. Just use echo -e "$outname\t$chrom\t$Pos\t$Ref\t$Alt\t$ReadCount\t$VariantAlleleCount\t$ReferenceAlleleCount\t$ReadCountControl\t$VariantAlleleCountControl\t$ReferenceAlleleCountControl\t$VAF" >> $outname ; Jan 1 '20 at 14:59 ## 2 Answers Using shell loops for text practice is considered bad practice. It is exceedingly slow, the syntax is complicated so it's very easy to get it wrong and it's just painful. The shell isn't designed as a proper scripting language, so while it can be (ab)used that way, you really should avoid it whenever possible. Here's a Perl script that should do exactly what your bash script does but will be several times faster and can deal with multiple input files: #!/usr/bin/env perl ## Iterate over all files given at the command line foreach my $$inFile (@ARGV) { print STDERR "Processing inFile...\n"; my outFile = "inFile.parsed.txt"; outFile =~ m|([^/]+)|; #/#Useless comment to fix SE syntax highlighting my outFileName = 1; open(my outFileHandle, '>', outFile) or die("Couldn't open 'outFile' for writing: !"); open(my inFileHandle, '<', inFile) or die("Couldn't open 'inFile' for writing:$$!"); ## Print header print $$outFileHandle "$$outFileName\tChrom\tPosition\tRef\tAlt\tTumorReadCount\t" . "TumorVariantAlleleCount\tTumorReferenceAlleleCount\tNormalReadCount" . "\tNormalVariantAlleleCount\tNormalReferenceAlleleCount\tVAF\n"; ## Read all lines of the input file while (<$$inFileHandle>) { ## Skip headers next if /^#/; ## Get VCF fields my (chrom, pos, name, ref, alt, qual, filter, info, format, values) = split(/\t/); my @infoFields = split(/;/, info); my readCount = infoFields[12];$$readCount =~ s/ReadCount=//; #/#Useless comment to fix SE syntax highlighting my $$variantAlleleCount =$$infoFields[25]; $$variantAlleleCount =~ s/VariantAlleleCount=//; #/#Useless comment to fix SE syntax highlighting my$$referenceAlleleCount= $$readCount -$$variantAlleleCount; my $$readCountControl =$$infoFields[13];$readCountControl =~ s/ReadCountControl=//;
#/#Useless comment to fix SE syntax highlighting

my $$variantAlleleCountControl =$$infoFields[26];
$variantAlleleCountControl =~ s/VariantAlleleCountControl=//; #/#Useless comment to fix SE syntax highlighting my $$referenceAlleleCountControl =$$readCountControl - $$variantAlleleCountControl; my$$VAF = $$infoFields[27];$$VAF =~ s/VariantAlleleFrequency=//; #/#Useless comment to fix SE syntax highlighting my $$outString = join("\t",$$outFileName, $$chrom,$$pos, $$ref,$$alt, $$readCount,$$variantAlleleCount, $$referenceAlleleCount,$$readCountControl, $$variantAlleleCountControl,$$referenceAlleleCountControl, $$VAF); print$$outFileHandle "$$outString\n"; } close($$outFileHandle); close($inFileHandle);
}


Save that file as parser.pl and then run it on all your vcf files with:

perl parser.pl /path/to/vcfs/*vcf


That should create a .parsed.txt file for each input file in the target directory.

If you just have to use the shell for this, you can try with this improved version of your script:

#!/usr/bin/env bash

for file in "$$@"; do outFile="$${file}.parsed.txt"
outFileName=$$(basename "$$outFile")
(
printf '%s\tChrom\tPosition\tRef\tAlt\tTumorReadCount\t' "$$outFileName" printf 'TumorVariantAlleleCount\tTumorReferenceAlleleCount\tNormalReadCount\t' printf 'NormalVariantAlleleCount\tNormalReferenceAlleleCount\tVAF\n' ) > "$$outFile"

echo "Reading $$file" grep -v '^#' "$$file" |
read chrom Pos Ref Alt rest < <(awk -F"\t" '{print $$1,$$2,$$4,$$5}' <<<"$$line") ReadCount=$$(echo "$$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 13 | sed 's/ReadCount=//' ) VariantAlleleCount=$$(echo "$$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 26| sed 's/VariantAlleleCount=//') ReferenceAlleleCount=$$(($$ReadCount -$$VariantAlleleCount))

ReadCountControl=$$(echo "$$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 14 | sed 's/ReadCountControl=//')
VariantAlleleCountControl=$$(echo "$$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 27 | sed 's/VariantAlleleCountControl=//')

ReferenceAlleleCountControl=$$(($$ReadCountControl - $VariantAlleleCountControl)) VAF=$$(echo "$$line" | cut -f 8 | sed 's/;/\t/g' | cut -f 28 | sed 's/VariantAlleleFrequency=//') printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$$outFileName" "$$chrom" "$$Pos" "$$Ref" "$$Alt" \ "$$ReadCount" "$$VariantAlleleCount" "$$ReferenceAlleleCount" "$$ReadCountControl" "$$VariantAlleleCountControl" \ "$$ReferenceAlleleCountControl" "$$VAF" >> "$outFile"
done
done


Just to illustrate what I mean about the shell being slow, I ran both scripts above on the example files you provided:

$time foo.pl *vcf 2>/dev/null real 0m0.051s user 0m0.047s sys 0m0.003s$ time bar.sh *vcf 2>/dev/null

real    2m28.842s
user    3m33.648s
sys     1m12.456s


As you can see above, the perl script took less than a second, while the bash script took almost two and a half minutes!

Finally, if you find the perl script too long, here's the same thing in a more condensed style:

#!/usr/bin/env perl

## Iterate over all files given at the command line
foreach my $$inFile (@ARGV) { print STDERR "Processing inFile...\n"; my outFile = "inFile.parsed.txt"; outFile =~ m|([^/]+)|; #/#Useless comment to fix SE syntax highlighting my outFileName = 1; open(my outFileHandle, '>', outFile); open(my inFileHandle, '<',$$inFile);

print $$outFileHandle "$$outFileName\tChrom\tPosition\tRef\tAlt\tTumorReadCount\t" .
"\tNormalVariantAlleleCount\tNormalReferenceAlleleCount\tVAF\n";
## Read all lines of the input file
while (<$$inFileHandle>) { ## Skip headers next if /^#/; ## Get VCF fields my (chrom, pos, name, ref, alt, qual, filter, info, format, values) = split(/\t/); info=~s/[^;]+=//g; my @infoFields = split(/;/, info); print outFileHandle (join "\t", outFileName, chrom, pos, ref, alt, infoFields[12],infoFields[25], infoFields[12]-infoFields[25], infoFields[13],infoFields[26], infoFields[13]-infoFields[26],$$infoFields[27]) . "\n";
}
}

• Oh my God, this is amazing. I am wondering when some people put their time and efforts to helping the other people this nicely . The perl done that in a second Jan 1 '20 at 17:16
• Sorry @terdon your code works on Single Nucleotide variants (SNVs) calls very well but on INDELs returns empty fields that may be because for example the position of ReferenceAlleleCount in SNV and INDEL are not the same. This is link to one of my INDEL files coming from the same Strelka variant caller sotonac-my.sharepoint.com/personal/fi1d18_soton_ac_uk/Documents/… I am not sure even I can post this comment or not because surely I see myself too annoying Jan 3 '20 at 4:56
• @Angel I can't access that file. It's probably best if you ask a new question, and post some lines from the bad files. Or, if the position is different, just try modifying the script and changing the numbers in the various $infofields[N]. Jan 3 '20 at 9:15 • Thank you @terdon, I have asked this as a new question here bioinformatics.stackexchange.com/questions/11074/… as I found the script specific to INDELs differs to what works for SNVs. If possible have a look please; thank you in advance Jan 4 '20 at 16:40 • @Angel you will need to explain what information you need. Since your script doesn't work, that won't help us understand your objective. So, show us your expected output from a few lines of example input. Jan 4 '20 at 16:43 Here is a quick and kind of dirty solution with minimum intervention to the code: • Add $outname'\t' right after echo -e to two two lines, the lines after #Header for output file and #Print output. The former will alter your header and the latter your actual output: you will have a first column with file name information.
• Then you can run this script in a for loop to generate multiple output files which then can be merged into a single file with cat.

Note that with this workaround, you will have multiple headers in the merged file. If you need a clean output, you will need to put your while loop in a for loop.

• Thank you so much @haci . I edited the script as you suggested adding for loop to that in my main post but I obtained the same error posted in my main post Jan 1 '20 at 14:43
• What happens when you get rid of the whitespace in the $outname'\t'$chrom bit (remember there are two of these for the two lines that were altered from the original code) and the quotes as @terdon suggested in their comment?
– haci
Jan 1 '20 at 15:10
• Sorry I tried all of your comments but I failed I have attached my original script and two vcf files here maybe it is better than pasting the code here Jan 1 '20 at 15:40