bcftools filtering all files in a directory

Question

Probably a silly oversight on my part, but I'm trying to filter all the vcfs in a directory with bcftools using a simple loop. My basic command is working fine:

bcftools filter -i 'QUAL > 1000' -o filter/file1out.vcf file1in.vcf 

but when I try to loop it, it echoes through each filename, but generates no output files or error messages.

for f in *.vcf; do echo "filtering $f"; bcftools filter -i 'QUAL > 1000' -o filter/$f_out.vcf $f; done

Am I not handling the output direction properly? Any help much appreciated!

Answer 1

I assume your shell is looking for the variable $f_out, but can't find it because you did not define it. It probably generated a hidden file with the name .vcf in your output directory.

Instead, use the following: -o ${f}_out.vcf. This will "protect/limit" the bash variable to just $f rather than $f_out.

When encountering issues like this it's worth putting echo in your loop right before your command, in this case bcftools (adding quotes whenever necessary). That would show you the command as how it is going to be executed and would show you, in this case, that the output name is incorrect.

Answer 2

Using a for loop for this is by far the safer option. However, you need to quote your variables and also make sure you're passing the right variable name. While you're at it, you can also remove the extension so that your filtered file name can be foo_out.vcf instead of foo.vcf_out.vcf (which is what your loop would create. Putting all that together:

for f in *.vcf; do 
    echo "filtering $f"; 
bcftools filter -i 'QUAL > 1000' -o filter/"${f//.vcf}"_out.vcf "$f"; done
fi

Answer 3

For one-liners, xargs is often more convenient than the for loop:

ls *.vcf | sed s,.vcf$,, \
  | xargs -i echo 'bcftools filter -i "QUAL > 1000" -o "filter/{}_out.vcf" "{}.vcf"' \
  | parallel

Note that your command line (with @WouterDeCoster's fix) writes output to foo.vcf_out.vcf. My command line writes to foo_out.vcf.