10
$\begingroup$

I am on a Mac using UNIX. I am trying to use the kallisto quant command on all files in a directory (instead of manually entering them). Because I'm running the analysis against the same index file, I first enter the following:

./kallisto index -i --index --make-unique index.fa 

This successfully creates an index file. Then, I tried this:

for file in *.fasta; do kallisto quant --single -l 200 -s 0.1 -o $file-aligned 
"$file"; done

While the $file-aligned folders are created, they're empty. I get an error that states that the index file is missing. So I assumed I would need to specify the index file in the command line. I then tried this:

for file in *.fasta; do kallisto quant -i index.fa --single -l 200 -s 0.1 -o
$file-aligned  "$file"; done

I get an error that says "Error: incompatible indices. Found version 0, expected version 10.

I have now also tried the following:

for file in *.fasta; do ./kallisto quant -i index --single - 200 -s 0.01 -o 
"${file}-aligned" "${file}"; 

but I get an error saying the index file is not found.


I am now running

or file in *.fasta; do ./kallisto quant -i index --single -l 200 -s 0.1 -o $file-aligned 

"$file"; done

This sort of works. The index file error messages are gone, but it runs quant on the same file 10 times instead of each file once. If I run echo $file it spits back the file that my command is running repeatedly.

If I close terminal, start a new session and run the same command, I get

Error: file not found ""
$\endgroup$
2
  • 1
    $\begingroup$ Why are you using fasta files for reads? Kallisto expects FASTQ files as input. $\endgroup$ – gringer Jun 29 '17 at 23:44
  • $\begingroup$ Does kallisto use quality scores of reads? I had an impression that it does not, but I don't have a reference to backup it.... $\endgroup$ – Kamil S Jaron Jun 30 '17 at 7:55
7
$\begingroup$

Running the jobs in parallel

Iteratively is one solution, but since you're not using bootstrapping, which can be run multithread (and is not needed if you're taking the popular tximport route) you can do better using GNU parallel, this will enable you to run as many jobs as you have execution threads on your system simultaneously. Note that Kalliso only runs multithreaded with -t if you're using bootstrapping.

  1. You'll need to make the index first:

    kallisto index --make-unique -I index.fa.idx index.fa
    
  2. Next lets define the input FASTQ files and the sample IDs you want to use using a tab-delimited input file like so:

    sample_1    sample_1_R1.fastq.gz
    sample_2    sample_2_R1.fastq.gz
    sample_3    sample_3_R1.fastq.gz
    

    Here the first column is the sample ID which will be used for output and the second separated by a tab character is the input file.

    You can produce this using a text editor but as a short-cut you can feed the list of fastq.gz you have into a list via:

    ls -1 *.fastq.gz > editme
    
  3. Next we create a list of jobs to run on our system using a utility script Make_job_list_kallisto.sh which consumes our tab delimited input file list above. I've used your -l and -s parameters below, note that with paired end data normally these are estimated for you.

    #!/bin/bash -eu
    
    [ $# -ne 2 ] && { echo -en "\n*** This script generates jobs for GNU parallel. *** \n\n Error Nothing to do, usage: < input tab delimited list > < output run list file >\n\n" ; exit 1; }
    set -o pipefail
    
    # Get command-line args
    INPUT_LIST=$1
    OUTPUT=$2
    
    # Set counter
    COUNT=1
    END=$(wc -l $INPUT_LIST | awk '{print $1}')
    
    echo " "
    echo " * Input file is: $INPUT_LIST"
    echo " * Number of runs: $END"
    echo " * Output job list for GNU parallel saved to: $OUTPUT"
    echo " "
    
    # Main bit of command-line for job
    CMD="kallisto quant -i index.fa.idx --single -l 200 -s 0.1"
    
    # Main Loop
    [ -e $OUTPUT ] && rm $OUTPUT
    while [ $COUNT -le $END ];
    do
        LINE=( $(awk "NR==$COUNT" $INPUT_LIST) )
        # Make file list
        echo "Working on $COUNT of $END Sample ID: ${LINE[0]}, Files ${LINE[@]:1}"
        echo "$CMD -o ${LINE[0]} ${LINE[@]:1}" >> $OUTPUT
        ((COUNT++))
    done
    

    We use the script like so:

     Make_job_list_kallisto.sh input_file_list.txt job_list.txt
    
  4. Now we have a job list for GNU parallel we can run our jobs in parallel like so:

    parallel --progress --jobs 4 --joblog kallisto_joblog.txt < job_list.txt
    

    This will run 4 of the kallisto jobs simultaneously (Increase this if you have more threads - don't mind consuming all the IO/CPU threads on your system) producing a nice progress report of those completed, additionally a job log will be written to the file kallisto_joblog.txt.

For paired end data I already have these scripts + similar instructions on GitHub which people might find useful.

$\endgroup$
7
  • $\begingroup$ @ZincFingers, so -I upper case for specifying the index in my run generation scrip should in fact be lower case -i. I've tested this and it works for me. This might be your issue or you might have some other error. If it's still not working try pasting in a few lines of your jobs list here. $\endgroup$ – Matt Bashton Jul 3 '17 at 20:49
  • $\begingroup$ The error I was getting was my fault, I fixed it. Now, when I run the parallel command, I get an error from kallisto saying Error: missing read files Error: file srr3884878.gz exists and is not a directory It says this for all 17 of my files $\endgroup$ – ZincFingers Jul 3 '17 at 20:56
  • $\begingroup$ I suspect you still have an issue with you input file to Make_job_list_kallisto.sh or the jobs it's generating, without seeing the input sample ID file list or the job list it's very difficult to debug. $\endgroup$ – Matt Bashton Jul 3 '17 at 21:35
  • $\begingroup$ Check the diagnostic output of Make_job_list_kallisto.sh is sane, I get: * Input file is: test_for_BS.txt * Number of runs: 4 * Output for GNU parallel saved to: test_job_list.txt Working on 1 of 4 Sample ID: test1, Files ATCACG_L004_R1_001.fastq.gz Working on 2 of 4 Sample ID: test2, Files GTCCGC_L004_R1_001.trimmed.fastq.gz etc. $\endgroup$ – Matt Bashton Jul 3 '17 at 21:39
  • $\begingroup$ I think you might have forgotten to give an output dir or sample ID in the first column of your tab delimited input list you're giving to Make_job_list_kallisto.sh because it looks like kallisto is missing the variable which would be supplied to -o and treating the input fastq.gz file as the output dir, it's then not finding the gz.fastq file as it's trying to use that as the output dir. $\endgroup$ – Matt Bashton Jul 3 '17 at 21:54
5
$\begingroup$

For generating the index, you should be using one of -i or --index, but not both:

./kallisto index --make-unique -i index.fa.idx index.fa 

You need to point kallisto to the generated index file (usually has a .idx extension), rather than the fasta file:

for file in *.fasta;
  do kallisto quant -i index.fa.idx --single -l 200 -s 0.1 -o "${file}-aligned" "${file}";
done
$\endgroup$
9
  • 1
    $\begingroup$ Why run it sequentially? Shouldn't this be parallelized, i.e. multiprocessed $\endgroup$ – Chris_Rands Jun 30 '17 at 9:30
  • 1
    $\begingroup$ You're asking the wrong person; try adding that as a comment to the question. Kallisto quant allows multi-threaded operation with -t <threads>. $\endgroup$ – gringer Jun 30 '17 at 11:38
  • $\begingroup$ I get the same error when I point to the index file Error: incompatible indices. Found version 0, expected version 10 $\endgroup$ – ZincFingers Jun 30 '17 at 15:18
  • $\begingroup$ maybe it's the index command that needs a bit of attention... I'll update the commands to include index generation $\endgroup$ – gringer Jun 30 '17 at 16:05
  • $\begingroup$ I managed to get this to work after fiddling with the index generation: ' for f in *.fasta' do kallisto quant -i index --single -l 200 -s 0.1 -o "${file}-aligned" "${file)"; ' But it appears that it's repeatedly analyzing one of the 10 .fasta files I placed in the directory (also I'm not sure how to get that line to appear as a code) $\endgroup$ – ZincFingers Jun 30 '17 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.