2
$\begingroup$

Hi I'm trying to filter a column in a PAF file with the AWK command and write it to a new file:

awk '$12>=20 {print $0}' /PATH/FILE.paf > /OUT/PATH/FITLERED_FILE.paf

This command makes sure the 12th column has a value of at least 20 and then prints the entire contents of the file - with the filtered column - to a new file.

This works for one file but I have 90+ I need to filter through. I tried using a for loop:

for i in /PATH/TO/FILES/*_mapped.paf; do awk '$12>=20 {print $0 > "awk_{}.out"}' $i; done

Which didn't work. And then I tried using a pipe command:

cat /Users/rimo/Desktop/Pat_Assemblies/flye_mapped.txt | parallel "awk '{$12>=20 print $0}' /Users/rimo/Desktop/Pat_Assemblies/Assemblies_Flye/{} > awk_{}"

Which also didn't work. I think the problem I am having is format-based but I would appreciate it if someone could steer me in the right direction!

$\endgroup$
5
  • $\begingroup$ What operating system are you using? Do you have access to GNU awk (gawk on some systems, just awk on most Linuxes except recent Ubuntu and maybe some others). $\endgroup$
    – terdon
    Commented Oct 21, 2022 at 16:20
  • $\begingroup$ I'm using a mac OS terminal $\endgroup$
    – rimo
    Commented Oct 21, 2022 at 16:22
  • $\begingroup$ OK. Do you have gawk? No need to install it if you don't, it would just make some things easier if you already have it. $\endgroup$
    – terdon
    Commented Oct 21, 2022 at 16:26
  • $\begingroup$ I don't have gawk but I could install it if it's easier! What would you suggest as an alternative? $\endgroup$
    – rimo
    Commented Oct 21, 2022 at 16:28
  • $\begingroup$ You don't really need gawk here. I was mistakenly thinking you wanted to edit the original file which is made easier by gawk (you can use gawk -i inplace) and gawk won't choke if you have too many open files, but all you need to avoid that is to close the file handle after writing (see the last command in my answer), so you really don't need to worry. Just use the default awk. $\endgroup$
    – terdon
    Commented Oct 21, 2022 at 16:32

1 Answer 1

3
$\begingroup$

The {} isn't in any way special to the shell or awk. It is used as a replacement by certain other programs, off the top of my head I can only think of find and xargs where it is used by convention, but it isn't a shell thing or an awk thing. Here, you are using it inside the awk command so it just breaks the awk's format because { } introduces a command block and this one is empty.

Your file name is $i, so if you want to make foo_mapped.paf into awk_foo_mapped.paf.out, you need awk_$i.out. Something like this:

for file in /PATH/TO/FILES/*_mapped.paf; do 
    fileName="${file##*/}"
    awk '$12>=20' "$file" > awk_"$fileName".out
done

Note how I removed the print $0. This is because the default action for awk when something evaluates to true is to print the current line, so awk '$12>=20 {print $0 } is equivalent to awk '$12>=20.

You can also use your original approach of having awk deal with the file name by using awk's internal FILENAME variable. Like this:

awk '$12>=20{ print > "awk_"FILENAME".out" }' /PATH/TO/FILES/*_mapped.paf

Depending on the number of files, you might need to close the file handle after writing. If you get an error about too many open files, try:

mawk '$12>=20{print > "awk_"FILENAME".out"; close("awk_"FILENAME".out")}' /PATH/TO/FILES/*_mapped.paf
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.