Run kallisto iteratively across many samples

Question

I am on a Mac using UNIX. I am trying to use the kallisto quant command on all files in a directory (instead of manually entering them). Because I'm running the analysis against the same index file, I first enter the following:

./kallisto index -i --index --make-unique index.fa 

This successfully creates an index file. Then, I tried this:

for file in *.fasta; do kallisto quant --single -l 200 -s 0.1 -o $file-aligned 
"$file"; done

While the $file-aligned folders are created, they're empty. I get an error that states that the index file is missing. So I assumed I would need to specify the index file in the command line. I then tried this:

for file in *.fasta; do kallisto quant -i index.fa --single -l 200 -s 0.1 -o
$file-aligned  "$file"; done

I get an error that says "Error: incompatible indices. Found version 0, expected version 10.

I have now also tried the following:

for file in *.fasta; do ./kallisto quant -i index --single - 200 -s 0.01 -o 
"${file}-aligned" "${file}"; 

but I get an error saying the index file is not found.

---

I am now running

or file in *.fasta; do ./kallisto quant -i index --single -l 200 -s 0.1 -o $file-aligned 

"$file"; done

This sort of works. The index file error messages are gone, but it runs quant on the same file 10 times instead of each file once. If I run echo $file it spits back the file that my command is running repeatedly.

If I close terminal, start a new session and run the same command, I get

Error: file not found ""

Answer 1

Running the jobs in parallel

Iteratively is one solution, but since you're not using bootstrapping, which can be run multithread (and is not needed if you're taking the popular tximport route) you can do better using GNU parallel, this will enable you to run as many jobs as you have execution threads on your system simultaneously. Note that Kalliso only runs multithreaded with -t if you're using bootstrapping.

1. You'll need to make the index first:

   kallisto index --make-unique -I index.fa.idx index.fa
   

2. Next lets define the input FASTQ files and the sample IDs you want to use using a tab-delimited input file like so:

   sample_1    sample_1_R1.fastq.gz
   sample_2    sample_2_R1.fastq.gz
   sample_3    sample_3_R1.fastq.gz
   

   Here the first column is the sample ID which will be used for output and the second separated by a tab character is the input file.

   You can produce this using a text editor but as a short-cut you can feed the list of fastq.gz you have into a list via:

   ls -1 *.fastq.gz > editme
   

3. Next we create a list of jobs to run on our system using a utility script Make_job_list_kallisto.sh which consumes our tab delimited input file list above. I've used your -l and -s parameters below, note that with paired end data normally these are estimated for you.

   #!/bin/bash -eu
   
   [ $# -ne 2 ] && { echo -en "\n*** This script generates jobs for GNU parallel. *** \n\n Error Nothing to do, usage: < input tab delimited list > < output run list file >\n\n" ; exit 1; }
   set -o pipefail
   
   # Get command-line args
   INPUT_LIST=$1
   OUTPUT=$2
   
   # Set counter
   COUNT=1
   END=$(wc -l $INPUT_LIST | awk '{print $1}')
   
   echo " "
   echo " * Input file is: $INPUT_LIST"
   echo " * Number of runs: $END"
   echo " * Output job list for GNU parallel saved to: $OUTPUT"
   echo " "
   
   # Main bit of command-line for job
   CMD="kallisto quant -i index.fa.idx --single -l 200 -s 0.1"
   
   # Main Loop
   [ -e $OUTPUT ] && rm $OUTPUT
   while [ $COUNT -le $END ];
   do
       LINE=( $(awk "NR==$COUNT" $INPUT_LIST) )
       # Make file list
       echo "Working on $COUNT of $END Sample ID: ${LINE[0]}, Files ${LINE[@]:1}"
       echo "$CMD -o ${LINE[0]} ${LINE[@]:1}" >> $OUTPUT
       ((COUNT++))
   done
   

   We use the script like so:

    Make_job_list_kallisto.sh input_file_list.txt job_list.txt
   

4. Now we have a job list for GNU parallel we can run our jobs in parallel like so:

   parallel --progress --jobs 4 --joblog kallisto_joblog.txt < job_list.txt
   

   This will run 4 of the kallisto jobs simultaneously (Increase this if you have more threads - don't mind consuming all the IO/CPU threads on your system) producing a nice progress report of those completed, additionally a job log will be written to the file kallisto_joblog.txt.

For paired end data I already have these scripts + similar instructions on GitHub which people might find useful.

Answer 2

For generating the index, you should be using one of -i or --index, but not both:

./kallisto index --make-unique -i index.fa.idx index.fa 

You need to point kallisto to the generated index file (usually has a .idx extension), rather than the fasta file:

for file in *.fasta;
  do kallisto quant -i index.fa.idx --single -l 200 -s 0.1 -o "${file}-aligned" "${file}";
done