I have a text file as follows

a   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz
b   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz
c   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz

The file is tab delimited and has 8 columns with 8th column being the url link I need to download the files one by one using wget and changing the name of the downloaded file with -O option in wget. The argument I want to give to the -O option is combination col1 and col2 separated by _ so that for first line I get the downloaded file as a_k.fastq.gz and so on. For example, the first line of the file above should give:

wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz

Following the post from here I realized looping through the file is slow and sometimes difficult. How can I download these files using wget with awk or cut as the tool for fixed fields

  • $\begingroup$ Can you post up an example wget command? $\endgroup$
    – Bioathlete
    Dec 13, 2017 at 19:20
  • $\begingroup$ wget encodeproject.org/files/ENCFF001JPX/@@download/… -O a_k.fastq.gz $\endgroup$ Dec 13, 2017 at 19:23
  • $\begingroup$ how can I put it into code in the comment section?? $\endgroup$ Dec 13, 2017 at 19:23
  • 2
    $\begingroup$ edit the original post $\endgroup$
    – Bioathlete
    Dec 13, 2017 at 19:25
  • $\begingroup$ @Llopis we've decided that basic things that bioinformaticians need to do in their daily work are on topic even when the issue could be expressed as a general text parsing question. $\endgroup$
    – terdon
    Dec 15, 2017 at 10:35

2 Answers 2

awk '{print "wget " $9 " -O " $1 "_" $2 ".fastq.gz"}' tmp 

will give you

wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz
wget https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz -O b_k.fastq.gz
wget https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz -O c_k.fastq.gz

If you add a | bash to the command it will then execute all the wget commands.

  • $\begingroup$ This works great. Can you tell what |bash is doing ?? $\endgroup$ Dec 13, 2017 at 19:37
  • 1
    $\begingroup$ @user3138373 the output of the awk command is a set of shell commands, essentially a mini script. By piping it to bash with | bash you're telling it to execute each input line as a command. So it will run the wget calls and download the sequences. $\endgroup$
    – terdon
    Dec 13, 2017 at 19:43

A variation of Bioathlete’s answer that doesn't suffer from escaping issues:

awk 'BEGIN{ FS="\t" } { printf("%s\t%s_%s.fastq.gz\n", $8, $1, $2) }' input |
while IFS=$'\t' read -r url filename; do
    wget -O "$filename" -- "$url"

Or entirely with shell parsing:

while IFS=$'\t' read -r c1 c2 c3 c4 c5 c6 c7 url; do
    wget -O "${c1}_${c2}.fastq.gz" -- "$url"
done < input

Or entirely without shell parsing but with GNU xargs:

awk 'BEGIN{ FS="\t" } { printf("%s_%s.fastq.gz\n%s\n", $1, $2, $8) }' input |
xargs -r -d '\n' -n 2 -- wget -O

… you could even run multiple wget instances in parallel from GNU xargs with the -P option, though you should set the -q option on wget to disable progress bars overwriting each other on the same terminal. Then again it would be better to use a real command-line download manager like Aria or Axel for that task.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.