# bcftools filtering all files in a directory

Probably a silly oversight on my part, but I'm trying to filter all the vcfs in a directory with bcftools using a simple loop. My basic command is working fine:

bcftools filter -i 'QUAL > 1000' -o filter/file1out.vcf file1in.vcf


but when I try to loop it, it echoes through each filename, but generates no output files or error messages.

for f in *.vcf; do echo "filtering $$f"; bcftools filter -i 'QUAL > 1000' -o filter/$$f_out.vcf $f; done  Am I not handling the output direction properly? Any help much appreciated! ## 3 Answers I assume your shell is looking for the variable $f_out, but can't find it because you did not define it. It probably generated a hidden file with the name .vcf in your output directory.

Instead, use the following: -o $${f}_out.vcf$$. This will "protect/limit" the bash variable to just $f rather than $f_out.

When encountering issues like this it's worth putting echo in your loop right before your command, in this case bcftools (adding quotes whenever necessary). That would show you the command as how it is going to be executed and would show you, in this case, that the output name is incorrect.

• This works thanks! But I wonder why I've had a similar loop working fine with $f.SnpSift.vcf as my output? Does it understand the . differently than the _ after $f? Dec 3 '18 at 17:44
• Yes that's right, and easy to test using a dummy variable and echo Dec 3 '18 at 19:14
• Alternatively, instead of (or in addition to) echo-ing you can use set -u which will give an error on undefined variables (this assumes you use Bash or Zsh as your shell (not sure about others)). Dec 6 '18 at 9:14

Using a for loop for this is by far the safer option. However, you need to quote your variables and also make sure you're passing the right variable name. While you're at it, you can also remove the extension so that your filtered file name can be foo_out.vcf instead of foo.vcf_out.vcf (which is what your loop would create. Putting all that together:

for f in *.vcf; do
echo "filtering $$f"; bcftools filter -i 'QUAL > 1000' -o filter/"$${f//.vcf}"_out.vcf "$f"; done fi  • I like the "${f//.vcf}", thanks - my filenames have been growing unnecessarily long! Dec 3 '18 at 19:52

For one-liners, xargs is often more convenient than the for loop:

ls *.vcf | sed s,.vcf$,, \ | xargs -i echo 'bcftools filter -i "QUAL > 1000" -o "filter/{}_out.vcf" "{}.vcf"' \ | parallel  Note that your command line (with @WouterDeCoster's fix) writes output to foo.vcf_out.vcf. My command line writes to foo_out.vcf. • Xargs is indeed convenient, but parsing ls is very fragile and will break on any non-standard file name. Why not for f in *vcf; do printf 'bcftools filter -i "QUAL > 1000" -o filter/"%s" "%s"\n' "${f//.vcf}_out.vcf" "\$f"; done | parallel? That seems as simple as xargs but can deal with arbitrary file names. Dec 3 '18 at 16:17
• @terdon I don't see that is an issue. You can simply add quotation marks. Dec 3 '18 at 17:19
• Yes, that will protect from spaces, but not from newlines. And OK, having newlines in filenames isn't something that happens very often but it is possible. I freely admit, however, that I may just be nitpicking because I'm too used to hanging out on Unix & Linux where this sort of thing is considered very important and any example of parsing ls is met with horror. Still, since the shell loop is safe and painless, it just seems better to use that instead. Dec 3 '18 at 17:27