Error after trimming illumina adapters

Question

I am removing illumina adapters of the NGS data with a loop. My NGS data is storage in /data/HTS_seq/. I used this function:

 for infile in /data/HTS_seq/*_read1.fastq; do outfile=$infile\_trim.fastq cutadapt -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC -e 0.1 -O 3 -m 30 --max-n 2 -o $outfile $infile; done

 for infile in /data/HTS_seq/*_read2.fastq; do outfile=$infile\_trim.fastq cutadapt -a AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA -e 0.1 -O 3 -m 30 --max-n 2 -o $outfile $infile; done

and It gives to me this error:

  IOError: [Errno 20] Not a directory: '/data/HTS_seq/PV589_03664AAB_AGGACT_read1.fastq/_trim.fastq'
  Traceback (most recent call last):
  File "/usr/local/bin/cutadapt", line 10, in <module>
cutadapt.main()
  File "/usr/local/lib/python2.7/dist-packages/cutadapt/scripts/cutadapt.py", line 652, in main
options.discard_untrimmed)
  File "/usr/local/lib/python2.7/dist-packages/cutadapt/scripts/cutadapt.py", line 327, in trimmed_and_untrimmed_files
trimmed = untrimmed = xopen(output_path, 'w')
  File "/usr/local/lib/python2.7/dist-packages/cutadapt/xopen.py", line 178, in xopen
return open(filename, mode)
      IOError: [Errno 20] Not a directory: '/data/HTS_seq/PV589_03664AAB_AGGACT_read1.fastq/_trim.fastq'

Which is my error? Thank you in advance.

Answer 1

First of all, you aren't actually setting the $outfile variable. Your loop has this basic format:

for var in *; do newVar="something" command $newVar; done

For example:

$ for var in 1 2 3 4; do newVar="new:$var" echo "$var: $newVar"; done
1: 
2: 
3: 
4: 

As you can see, newVar is never actually being set. That's because setting a variable is its own operation. So you need to do that, end it with a control character like ;, or &, or a newline, and then run your cutadapt:

$ for var in 1 2 3 4; do newVar="new:$var"; echo "$var: $newVar"; done
1: new:1
2: new:2
3: new:3
4: new:4
 

Or, with your data:

for infile in /data/HTS_seq/*_read1.fastq; do 
    outfile="$infile"_trim.fastq 
    cutadapt -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC \ 
      -e 0.1 -O 3 -m 30 --max-n 2 -o "$outfile" "$infile"
done


for infile in /data/HTS_seq/*_read2.fastq; do 
    outfile="$infile"_trim.fastq
    cutadapt -a AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA \ 
      -e 0.1 -O 3 -m 30 --max-n 2 -o $outfile $infile
done

But that would give a weird file name, like foo_S1_L001_read1.fastq_trim.fastq. You probably want names like foo_S1_L001_read1_trim.fastq instead, so you would want:

for infile in /data/HTS_seq/*_read1.fastq; do 
    outfile="${infile%.fastq}"_trim.fastq
    cutadapt -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC \ 
      -e 0.1 -O 3 -m 30 --max-n 2 -o "$outfile" "$infile"
done


for infile in /data/HTS_seq/*_read2.fastq; do 
    outfile="${infile%.fastq}"_trim.fastq
    cutadapt -a AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA \ 
      -e 0.1 -O 3 -m 30 --max-n 2 -o $outfile $infile
done

---

Now, the specific error you show makes me think that isn't the command you ran. Could you perhaps have run this one instead by mistake (note the / instead of \ before the _trim.fastq)?

for infile in /data/HTS_seq/*_read1.fastq; do
 outfile=$infile/_trim.fastq cutadapt -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC \ 
-e 0.1 -O 3 -m 30 --max-n 2 -o $outfile $infile
done

Your error message is:

  IOError: [Errno 20] Not a directory: '/data/HTS_seq/PV589_03664AAB_AGGACT_read1.fastq/_trim.fastq'

Which means that cutadapt tried to open /data/HTS_seq/PV589_03664AAB_AGGACT_read1.fastq/ as a directory and then look for a file named _trim.fastq inside it. The only way I can think that this would happen is if you ran the command I gave above and not the one you show in your question.

Answer 2

There is a bug here

/data/HTS_seq/*_read1.fastq 

This will go to your root directory / and there is definiately no folder called data in the root. To prove it type ls / and that is where the command is pointing, then compare this to ls ./.

To solve it go to the directory HTS_seq and type pwd then copy and paste the directory structure. I suspect ./data/HTS_seq will solve it

 for infile in ./data/HTS_seq; ...

This will look for the folder data in your current working directory rather than the root directory.

---

Okay you've been assigned /data as root ... no biggie.

PV589_03664AAB_AGGACT_read1.fastq/_trim.fastq

Why is / in the file name? What happens if you ls /data/HTS_seq/PV589_03664AAB_AGGACT_read1.fastq/_trim.fastq? and ls /data/HTS_seq/ Finally could you please ls /data/HTS_seq/PV589_03664AAB_AGGACT_read1.fastq\/_trim.fastq?

The \/ is important. What I think has happened is a / is in the file name and that will cause problems because it will think that _trim.fastq is a directory when its part of a file name, and would explain the error.

Then do the same thing using head instead of ls see what happens.

If the last ls and head works then rename the files by typing rename in the command line. If rename works it will be easy to remove / from all your files using the power of the rename function (its a Perl function BTW).