for loop cutadapt for files on a single directory

Question

I have a bigfolder where i have a lot of fastq.gz files and I want to remove the adapters from all of them.

I am trying then the following loop:

for file in LotsofRaws\/cannam/*_R1.fastq.gz; do cutadapt 
     -a runs/new/Adapters/IlluNexteraPE.fa 
        -o LotsofRaws/RemovedAdapter/Adapted$file   
           $file done;  done 

However I am getting the following error:

cutadapt: error: 

*It appears you want to trim paired-end data because you provided two input files, but then you also need to provide two output files (with -o and -p) or use the --interleaved option.*

But I just want to take the adapters from the files I am not looking to do it in the same way I would do with a single file is there any way to do this because if I try to do --interleaved I get:

cutadapt: error: Character '/' in adapter sequence 'RTNS/NEW/ADAPTERS/ILLTNEXTERAPE.FA' is not a valid IUPAC code. Use only characters XACGTURYSWKMBDHVN.

This is cutadapt 2.3 with Python 3.6.7

and if instead of using the file I directly use the sequences:

for file in LotsofRaws/cannam/*; 
         do cutadapt -a CTGTCTCTTATACACATCT 
               -a AGATCGGAAGAGC 
               --interleaved 
               -o LotsofRaws/RemovedAdapter/$file   $file ; done

I get:

FileNotFoundError: [Errno 2] No such file or directory: 'LotsofRaws/RemovedAdapter/LotsofRaws/cannam/NAM7_Glenlea_R1.fastq.gz'
During handling of the above exception, another exception occurred:

many times

What should I then do for general variant calling it is appropriate to do the --interleaved option?

Should I just the do the trimming manually or can I loop it in any way that still resembles doing it one by one but without me having to go and write line and line again.

So the files on the folder look like:

NAM11_BW415_R1.fastq.gz         NAM13_BW968_R2.fastq.gz     NAM17_AC_Barrie_R1.fastq.gz  NAM3_PI648600_R2.fastq.gz  NAM7_Glenlea_R1.fastq.gz   PI648598_R2.fastq.gz
NAM11_BW415_R2.fastq.gz         NAM13_Frontana_R1.fastq.gz  NAM17_AC_Barrie_R2.fastq.gz  NAM3_PI648842_R1.fastq.gz  NAM7_Glenlea_R2.fastq.gz   PI648611_R1.fastq.gz
NAM11_Reeder_R1.fastq.gz        NAM13_Frontana_R2.fastq.gz  NAM17_BW388_R1.fastq.gz      NAM3_PI648842_R2.fastq.gz  NAM7_Laura_R1.fastq.gz     PI648611_R2.fastq.gz
NAM11_Reeder_R2.fastq.gz        NAM14_BW874_R1.fastq.gz     NAM17_BW388_R2.fastq.gz      NAM4_PI613279_R1.fastq.gz  NAM7_Laura_R2.fastq.gz     PI648672_R1.fastq.gz
NAM11_Webill_R1.fastq.gz        NAM14_BW874_R2.fastq.gz     NAM17_RedFife_R1.fastq.gz    NAM4_PI613279_R2.fastq.gz  NAM7_Marquis_R1.fastq.gz   PI648672_R2.fastq.gz
NAM11_Webill_R2.fastq.gz        NAM14_Glenn_R1.fastq.gz     NAM17_RedFife_R2.fastq.gz    NAM4_PI613280_R1.fastq.gz  NAM7_Marquis_R2.fastq.gz   PI648724_R1.fastq.gz
NAM12_BW431_L002_R1.fastq.gz    NAM14_Glenn_R2.fastq.gz     NAM1_PI648517_R1.fastq.gz    NAM4_PI613280_R2.fastq.gz  NAM8_FL62R1_R1.fastq.gz    PI648724_R2.fastq.gz
NAM12_BW431_L002_R2.fastq.gz    NAM14_Unity_R1.fastq.gz     NAM1_PI648517_R2.fastq.gz    NAM4_PI648524_R1.fastq.gz  NAM8_FL62R1_R2.fastq.gz    PI648768_R1.fastq.gz
NAM12_BW431_L008_R1.fastq.gz    NAM14_Unity_R2.fastq.gz     NAM1_PI648519_R1.fastq.gz    NAM4_PI648524_R2.fastq.gz  NAM8_Muchmore_R1.fastq.gz  PI648768_R2.fastq.gz
NAM12_BW431_L008_R2.fastq.gz    NAM15_BW881_R1.fastq.gz     NAM1_PI648519_R2.fastq.gz    NAM5_PI648523_R1.fastq.gz  NAM8_Muchmore_R2.fastq.gz  PI648860_R1.fastq.gz
NAM12_Penhold_L002_R1.fastq.gz  NAM15_BW881_R2.fastq.gz     NAM1_PI648541_R1.fastq.gz    NAM5_PI648523_R2.fastq.gz  NAM8_Stettler_R1.fastq.gz  PI648860_R2.fastq.gz
NAM12_Penhold_L002_R2.fastq.gz  NAM15_Lillian_R1.fastq.gz   NAM1_PI648541_R2.fastq.gz    NAM5_PI648585_R1.fastq.gz  NAM8_Stettler_R2.fastq.gz  PI648862_R1.fastq.gz
NAM12_Penhold_L008_R1.fastq.gz  NAM15_Lillian_R2.fastq.gz   NAM2_PI648493_R1.fastq.gz    NAM5_PI648585_R2.fastq.gz  PI277012_R1.fastq.gz       PI648862_R2.fastq.gz
NAM12_Penhold_L008_R2.fastq.gz  NAM15_PT245_R1.fastq.gz     NAM2_PI648493_R2.fastq.gz    NAM5_PI648622_R1.fastq.gz  PI277012_R2.fastq.gz       PI648864_R1.fastq.gz
NAM12_Sumai3_L002_R1.fastq.gz   NAM15_PT245_R2.fastq.gz     NAM2_PI648602_R1.fastq.gz    NAM5_PI648622_R2.fastq.gz  PI648478_R1.fastq.gz       PI648864_R2.fastq.gz
NAM12_Sumai3_L002_R2.fastq.gz   NAM16_BA51B92_R1.fastq.gz   NAM2_PI648602_R2.fastq.gz    NAM6_Neepawa_R1.fastq.gz   PI648478_R2.fastq.gz       run_701_150904_7001410_0194_BC79VNANXX_NAM.list
NAM12_Sumai3_L008_R1.fastq.gz   NAM16_BA51B92_R2.fastq.gz   NAM2_PI648609_R1.fastq.gz    NAM6_Neepawa_R2.fastq.gz   PI648487_R1.fastq.gz       run_760_160912_7001343F_0065_AC9EV0ANXX_NAM.list
NAM12_Sumai3_L008_R2.fastq.gz   NAM16_BW928_R1.fastq.gz     NAM2_PI648609_R2.fastq.gz    NAM6_PI648532_R1.fastq.gz  PI648487_R2.fastq.gz
NAM13_BW462_R1.fastq.gz         NAM16_BW928_R2.fastq.gz     NAM3_PI648511_R1.fastq.gz    NAM6_PI648532_R2.fastq.gz  PI648566_R1.fastq.gz
NAM13_BW462_R2.fastq.gz         NAM16_BW963_R1.fastq.gz     NAM3_PI648511_R2.fastq.gz    NAM6_Thatcher_R1.fastq.gz  PI648566_R2.fastq.gz
NAM13_BW968_R1.fastq.gz         NAM16_BW963_R2.fastq.gz     NAM3_PI648600_R1.fastq.gz    NAM6_Thatcher_R2.fastq.gz  PI648598_R1.fastq.gz

Answer 1

You have various different issues here. First, the first version of the command you show would give a syntax error, not the error you show, so I am guessing what you actually ran was this:

for file in LotsofRaws\/cannam/*_R1.fastq.gz; do 
    cutadapt -a runs/new/Adapters/IlluNexteraPE.fa \ 
             -o LotsofRaws/RemovedAdapter/Adapted$file $file done;  
done 

The error there is that you have the word done after $file. So what cutadapt sees is:

cutadapt [OPTIONS] $file done

So it reads that as being given two input files instead of one. What you wanted to do was this:

for file in LotsofRaws\/cannam/*_R1.fastq.gz; do 
    cutadapt -a runs/new/Adapters/IlluNexteraPE.fa \ 
             -o LotsofRaws/RemovedAdapter/Adapted$file $file  
done 

Or, on one line, this:

for file in LotsofRaws\/cannam/*_R1.fastq.gz; do cutadapt -a runs/new/Adapters/IlluNexteraPE.fa -o LotsofRaws/RemovedAdapter/Adapted$file $file; done

One done, not two and separated from the command by a newline or a ;.

Your next attempt, with --interleaved failed because the -a option expects an actual sequence, not a file unless you use it with file:. See cutadapt --help (emphasis mine):

Finding adapters:
Parameters -a, -g, -b specify adapters to be removed from each read (or from the first read in a pair if data is paired). If specified multiple times, only the best matching adapter is trimmed (but see the --times option). When the special notation 'file:FILE' is used, adapter sequences are read from the given FASTA file.

So, the -a expects to be given a sequence unless you use it as -a file:/path/to/file. Because you used it as -a runs/new/Adapters/IlluNexteraPE.fa instead of -a file:withruns/new/Adapters/IlluNexteraPE.fa, it tried to interpret runs/new/Adapters/IlluNexteraPE.fa as a sequence and complained because the character / isn't valid in a DNA sequence.

Finally, your last lop failed because you're giving the wrong name. The first rule of debugging is "print all the things". When you see a command that doesn't work as you expect, your first step should always be to print out the command and see what is actually being run. When you do a for file in /long/path/to/file loop, the entire path is included in your $file variable. Try running this:

for file in LotsofRaws/cannam/*; do 
    echo "cutadapt -a CTGTCTCTTATACACATCT -a AGATCGGAAGAGC --interleaved -o LotsofRaws/RemovedAdapter/$file $file"; 
done

You will see that that will print commands like this:

cutadapt -a CTGTCTCTTATACACATCT -a AGATCGGAAGAGC --interleaved \
         -o LotsofRaws/RemovedAdapter/LotsofRaws/cannam/NAM11_BW415_R1.fastq.gz \
          LotsofRaws/cannam/NAM11_BW415_R1.fastq.gz

What you need is to take the name of the file, without the path. And you don't want to escape the slash after LotsofRaws. So putting all this together and going back to your first attempt, what you want is something like this:

for f in LotsofRaws\/cannam/*_R1.fastq.gz; do 
    fname=$(basename file)
    cutadapt -a "file:runs/new/Adapters/IlluNexteraPE.fa" 
             -o "LotsofRaws/RemovedAdapter/Adapted${fname}" $file
done