Edit column in multiple GFF3 file using parallel and awk?

Question

I have multiple gff files that I want to edit the start and end by 5 bp in batch. I found a relevant answer here for single file, but I could not add comment there. The awk line works for my files if processing only one file, but when I try to do the same thing in batch, it returns syntax error. How should I do it with parallel?

> ls ./*ratio.gff.gz | parallel "zcat {} | head | awk -F'\t' -v OFS='\t' '/^[^#]/{$5-=5}{$4+=5}1' "
awk: line 1: syntax error at or near -=
awk: line 1: syntax error at or near +=
awk: line 1: syntax error at or near -=
awk: line 1: syntax error at or near +=
awk: line 1: syntax error at or near -=
awk: line 1: syntax error at or near +=
awk: line 1: syntax error at or near -=
awk: line 1: syntax error at or near +=
awk: line 1: syntax error at or near -=
awk: line 1: syntax error at or near +=
awk: line 1: syntax error at or near -=
awk: line 1: syntax error at or near +=
awk: line 1: syntax error at or near -=
awk: line 1: syntax error at or near +=


> zcat xxxx_ratio.gff.gz | head | awk -F'\t' -v OFS='\t' '/^[^#]/{$5-=5}{$4+=5}1'
##gff-version   3               5
# biweight_mean=-0.22621330627900907                    5
chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5136    5179    0.60    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005131;count=2
chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5181    5229    0.65    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005176;count=1
chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5241    5287    0.41    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005236;count=1
chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5286    5327    0.09    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005281;count=1
chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5406    5452    0.92    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005401;count=1
chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5466    5519    0.98    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005461;count=1
chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5506    5559    0.26    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005501;count=1
chr2L   NimbleScan      31945102:MA_larvae_Exp1/MA_larvae_Ctrl1:BLOCK1  5576    5628    0.41    .       .       seq_id=chr2L:1-23011544;probe_id=CHR02LFS000005571;count=1

Thank you!

Answer 1

You are getting a syntax error because your command is in double quotes. This means that the awk variables you are using, $4 and $5, are being expanded by the shell before launching awk, and since they are not defined, they are passed to awk as empty, making your command:

awk -F'\t' -v OFS='\t' '/^[^#]/{-=5}{+=5}1' "

And that is what gives you the syntax error. If you are 100% sure your file names contain no whitespace and no globbing characters, so you can use ls as you have done (which is generally a very bad idea), then this should work:

 ls ./*ratio.gff.gz | 
   parallel "zcat {} | head | awk -F'\t' -v OFS='\t' '/^[^#]/{\\\$5-=5}{\\\$4+=5}1' "

However, that won't change the original files and it isn't saving the output anywhere, so I doubt it's what you really want to do. This command will edit each file, and save the result into a new file with the same name plus edited.gz. So if you start with foo.gff.gz, you will get a new file called foo.gff.gz.edited.gz:

parallel "awk -F'\t' -v OFS='\t' '/^[^#]/{\\\$5-=5}{\\\$4+=5}1' <(zcat {}) | gzip > {}.edited.gz" ::: *gff.gz

Once you are sure this works as you want, you can rename all the edited.gz files back to the original name, overwriting the originals:

for f in *edited.gz; do mv -- "$f" "${f//.edited.gz}"; done

---

IMPORTANT: The solution above is affected by a markdown rendering bug. There should only ever be one \, as can be seen in the source of this answer, and not three. Make sure to replace all occurrences of \\\ in this answer with \.