1
$\begingroup$

I'm running a bioinformatics pipeline to analyze large nanopore sequencing files using FastQC. However, I keep running into an OutOfMemoryError. Here's what happens:

Unknown option: memory
Started analysis of SQK-NBD114-24_barcode16.fastq
Approx 5% complete for SQK-NBD114-24_barcode16.fastq
...
Approx 40% complete for SQK-NBD114-24_barcode16.fastq
Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap space
        at uk.ac.babraham.FastQC.Utilities.QualityCount.<init>(QualityCount.java:33)
        at uk.ac.babraham.FastQC.Modules.PerBaseQualityScores.processSequence(PerBaseQualityScores.java:141)
        at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:89)
        at java.base/java.lang.Thread.run(Thread.java:829)
^CError: FastQC failed for /home/jupyter/bioinfo_pipeline/demuxed_2/SQK-NBD114-24_barcode16.fastq. Exiting pipeline.

I attempted to allocate more memory by setting FASTQC_MEMORY="4g" and running FastQC with the following script:

# Allocate more memory to FastQC (e.g., 4GB)
FASTQC_MEMORY="4g"

# Loop over all .fastq files in the demuxed directory
for filename in "$DEMUX_DIR"/*.fastq; do
    echo -e "\nProcessing $filename..."
    
    #-= STEP 3 - SECOND quality control (FASTQC and NANOQC) =- 
    echo -e "\n~~~~~~-= Step 3, QC =-~~~~~~\n"
    OUTPUT_PC="$filename"  # Assuming OUTPUT_PC refers to the current .fastq file
    
    # Run FASTQC with increased memory allocation
    fastqc --outdir="$OUTPUT_QC" --memory="$FASTQC_MEMORY" "$OUTPUT_PC"
    
    # Check if FastQC failed due to memory issues or other errors
    if [ $? -ne 0 ]; then
    echo "Error: FastQC failed for $filename. Exiting pipeline."
        exit 1
    fi
done

But it seems like 4GB isn't enough, as the process still crashes. The file I'm working with is quite large (Nanopore sequencing data):

-rw-r--r-- 1 jupyter jupyter 442198080 Aug 27 14:49 SQK-NBD114-24_barcode16.fastq

According to free -h, I have 22 GiB of available memory. How much memory should I allocate to FastQC to prevent these errors? Is there a rule of thumb or specific calculation I should use when working with large .fastq files?

Improve this question
$\endgroup$
3
  • $\begingroup$ For now I would say give it like 15-20 GB and see if you run into the same error. Which version of FastQC are you using? $\endgroup$
    – Karthik Nair
    Commented Aug 27 at 15:45
  • 1
    $\begingroup$ Where did you get this --memory option? That doesn't seem to exist. Note the first line of your error output: Unknown option: memory. On my fastqc version 0.12.1, the --memory option is known, but only takes values in megabytes. Check that your version i) has a --memory option and ii) it can accept the string 4g as a valid value. $\endgroup$
    – terdon ♦
    Commented Aug 27 at 16:13
  • $\begingroup$ @Karthik Nair FastQC v0.11.9 $\endgroup$
    – dim
    Commented Aug 28 at 10:02

1 Answer 1

Reset to default
0
$\begingroup$

Using more threads solves the problem.

Run FASTQC with increased memory allocation

fastqc --outdir="$OUTPUT_QC" -t 10 "$OUTPUT_PC"
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.