Filtering file with AWK and writing output to new file

Question

Hi I'm trying to filter a column in a PAF file with the AWK command and write it to a new file:

awk '$12>=20 {print $0}' /PATH/FILE.paf > /OUT/PATH/FITLERED_FILE.paf

This command makes sure the 12th column has a value of at least 20 and then prints the entire contents of the file - with the filtered column - to a new file.

This works for one file but I have 90+ I need to filter through. I tried using a for loop:

for i in /PATH/TO/FILES/*_mapped.paf; do awk '$12>=20 {print $0 > "awk_{}.out"}' $i; done

Which didn't work. And then I tried using a pipe command:

cat /Users/rimo/Desktop/Pat_Assemblies/flye_mapped.txt | parallel "awk '{$12>=20 print $0}' /Users/rimo/Desktop/Pat_Assemblies/Assemblies_Flye/{} > awk_{}"

Which also didn't work. I think the problem I am having is format-based but I would appreciate it if someone could steer me in the right direction!

Answer 1

The {} isn't in any way special to the shell or awk. It is used as a replacement by certain other programs, off the top of my head I can only think of find and xargs where it is used by convention, but it isn't a shell thing or an awk thing. Here, you are using it inside the awk command so it just breaks the awk's format because { } introduces a command block and this one is empty.

Your file name is $i, so if you want to make foo_mapped.paf into awk_foo_mapped.paf.out, you need awk_$i.out. Something like this:

for file in /PATH/TO/FILES/*_mapped.paf; do 
    fileName="${file##*/}"
    awk '$12>=20' "$file" > awk_"$fileName".out
done

Note how I removed the print $0. This is because the default action for awk when something evaluates to true is to print the current line, so awk '$12>=20 {print $0 } is equivalent to awk '$12>=20.

You can also use your original approach of having awk deal with the file name by using awk's internal FILENAME variable. Like this:

awk '$12>=20{ print > "awk_"FILENAME".out" }' /PATH/TO/FILES/*_mapped.paf

Depending on the number of files, you might need to close the file handle after writing. If you get an error about too many open files, try:

mawk '$12>=20{print > "awk_"FILENAME".out"; close("awk_"FILENAME".out")}' /PATH/TO/FILES/*_mapped.paf