3
$\begingroup$

I am trying to run a batch script to assemble sequences using the spades.py command on HPC. I tested the script with one sequence (1 forward and 1 reverse read) and this worked. I have 100 sequences in total (200 files) and am trying to assemble all of them without having to resubmit the script each time. This is the original script that worked:

#!/bin/bash  
 
#SBATCH --job-name=pangenome_spades  
#SBATCH --account=lilianasalvador  
#SBATCH --partition=standard  
#SBATCH --nodes=1  
#SBATCH --ntasks=1  
#SBATCH --mem-per-cpu=5gb  
#SBATCH --time=01:00:00  
 
spades.py -1 SRR8599992_1.fastq.gz -2 SRR8599992_2.fastq.gz -o /home/u32/lucykelly/spades_test -t 16 --careful 

This is the script that I've been trying to run and the error it's giving me:

#!/bin/bash  
 
#SBATCH --job-name=pangenome_spades  
#SBATCH --account=lilianasalvador  
#SBATCH --partition=standard  
#SBATCH --nodes=1  
#SBATCH --ntasks=1  
#SBATCH --mem-per-cpu=5gb  
#SBATCH --time=02:00:00  
 
module load python/3.9  
module load spades/3.15.0-hpc 

spades.py -1 *_1.fastq.gz -2 *_2.fastq.gz -o /groups/lilianasalvador/noah_100/SPAdes_output -t 16 --careful 

SPAdes genome assembler v3.15.0-hpc 
 
Usage: spades.py [options] -o <output_dir> 
spades.py: error: Please specify option (e.g. -1, -2, -s, etc)) for the following paths: SRR8599995_1.fastq.gz, SRR8599997_1.fastq.gz, SRR8599998_1.fastq.gz, SRR8600006_1.fastq.gz, SRR8600007_1.fastq.gz, SRR8600008_1.fastq.gz, SRR8600014_1.fastq.gz, SRR8600015_1.fastq.gz, SRR8600020_1.fastq.gz, SRR8600022_1.fastq.gz, SRR8600026_1.fastq.gz, SRR8600031_1.fastq.gz, SRR8600044_1.fastq.gz, SRR8600045_1.fastq.gz, SRR8600048_1.fastq.gz, SRR8600049_1.fastq.gz, SRR8600052_1.fastq.gz, SRR8600055_1.fastq.gz, SRR8600058_1.fastq.gz, SRR8600059_1.fastq.gz, SRR8600064_1.fastq.gz, SRR8600070_1.fastq.gz, SRR8600072_1.fastq.gz, SRR8600075_1.fastq.gz, SRR8600079_1.fastq.gz, SRR8600092_1.fastq.gz, SRR8600093_1.fastq.gz, SRR8600098_1.fastq.gz, SRR8600100_1.fastq.gz, SRR8600101_1.fastq.gz, SRR8600102_1.fastq.gz, SRR8600103_1.fastq.gz, SRR8600105_1.fastq.gz, SRR8600110_1.fastq.gz, SRR8600113_1.fastq.gz, SRR8600117_1.fastq.gz, SRR8600121_1.fastq.gz, SRR8600122_1.fastq.gz, SRR8600123_1.fastq.gz, SRR8600128_1.fastq.gz, SRR8600133_1.fastq.gz, SRR8600134_1.fastq.gz, SRR8600136_1.fastq.gz, SRR8600145_1.fastq.gz, SRR8600147_1.fastq.gz, SRR8600160_1.fastq.gz, SRR8600164_1.fastq.gz, SRR8600166_1.fastq.gz, SRR8600167_1.fastq.gz, SRR8600174_1.fastq.gz, SRR8600182_1.fastq.gz, SRR8600184_1.fastq.gz, SRR8600185_1.fastq.gz, SRR8600188_1.fastq.gz, SRR8600189_1.fastq.gz, SRR8600190_1.fastq.gz, SRR8600193_1.fastq.gz, SRR8600194_1.fastq.gz, SRR8600216_1.fastq.gz, SRR8600222_1.fastq.gz, SRR8600223_1.fastq.gz, SRR8600226_1.fastq.gz, SRR8600228_1.fastq.gz, SRR8600235_1.fastq.gz, SRR8600236_1.fastq.gz, SRR8600243_1.fastq.gz, SRR8600244_1.fastq.gz, SRR8600251_1.fastq.gz, SRR8600252_1.fastq.gz, SRR8600258_1.fastq.gz, SRR8600259_1.fastq.gz, SRR8600264_1.fastq.gz, SRR8600267_1.fastq.gz, SRR8600268_1.fastq.gz, SRR8600270_1.fastq.gz, SRR8600272_1.fastq.gz, SRR8600274_1.fastq.gz, SRR8600277_1.fastq.gz, SRR8600278_1.fastq.gz, SRR8600281_1.fastq.gz, SRR8600282_1.fastq.gz, SRR8600283_1.fastq.gz, SRR8600285_1.fastq.gz, SRR8600287_1.fastq.gz, SRR8600288_1.fastq.gz, SRR8600291_1.fastq.gz, SRR8600294_1.fastq.gz, SRR8600296_1.fastq.gz, SRR8600302_1.fastq.gz, SRR8600308_1.fastq.gz, SRR8600309_1.fastq.gz, SRR8600311_1.fastq.gz, SRR8600319_1.fastq.gz, SRR8600327_1.fastq.gz, SRR8600328_1.fastq.gz, SRR8600329_1.fastq.gz, SRR8600331_1.fastq.gz, SRR8600339_1.fastq.gz, SRR8599995_2.fastq.gz, SRR8599997_2.fastq.gz, SRR8599998_2.fastq.gz, SRR8600006_2.fastq.gz, SRR8600007_2.fastq.gz, SRR8600008_2.fastq.gz, SRR8600014_2.fastq.gz, SRR8600015_2.fastq.gz, SRR8600020_2.fastq.gz, SRR8600022_2.fastq.gz, SRR8600026_2.fastq.gz, SRR8600031_2.fastq.gz, SRR8600044_2.fastq.gz, SRR8600045_2.fastq.gz, SRR8600048_2.fastq.gz, SRR8600049_2.fastq.gz, SRR8600052_2.fastq.gz, SRR8600055_2.fastq.gz, SRR8600058_2.fastq.gz, SRR8600059_2.fastq.gz, SRR8600064_2.fastq.gz, SRR8600070_2.fastq.gz, SRR8600072_2.fastq.gz, SRR8600075_2.fastq.gz, SRR8600079_2.fastq.gz, SRR8600092_2.fastq.gz, SRR8600093_2.fastq.gz, SRR8600098_2.fastq.gz, SRR8600100_2.fastq.gz, SRR8600101_2.fastq.gz, SRR8600102_2.fastq.gz, SRR8600103_2.fastq.gz, SRR8600105_2.fastq.gz, SRR8600110_2.fastq.gz, SRR8600113_2.fastq.gz, SRR8600117_2.fastq.gz, SRR8600121_2.fastq.gz, SRR8600122_2.fastq.gz, SRR8600123_2.fastq.gz, SRR8600128_2.fastq.gz, SRR8600133_2.fastq.gz, SRR8600134_2.fastq.gz, SRR8600136_2.fastq.gz, SRR8600145_2.fastq.gz, SRR8600147_2.fastq.gz, SRR8600160_2.fastq.gz, SRR8600164_2.fastq.gz, SRR8600166_2.fastq.gz, SRR8600167_2.fastq.gz, SRR8600174_2.fastq.gz, SRR8600182_2.fastq.gz, SRR8600184_2.fastq.gz, SRR8600185_2.fastq.gz, SRR8600188_2.fastq.gz, SRR8600189_2.fastq.gz, SRR8600190_2.fastq.gz, SRR8600193_2.fastq.gz, SRR8600194_2.fastq.gz, SRR8600216_2.fastq.gz, SRR8600222_2.fastq.gz, SRR8600223_2.fastq.gz, SRR8600226_2.fastq.gz, SRR8600228_2.fastq.gz, SRR8600235_2.fastq.gz, SRR8600236_2.fastq.gz, SRR8600243_2.fastq.gz, SRR8600244_2.fastq.gz, SRR8600251_2.fastq.gz, SRR8600252_2.fastq.gz, SRR8600258_2.fastq.gz, SRR8600259_2.fastq.gz, SRR8600264_2.fastq.gz, SRR8600267_2.fastq.gz, SRR8600268_2.fastq.gz, SRR8600270_2.fastq.gz, SRR8600272_2.fastq.gz, SRR8600274_2.fastq.gz, SRR8600277_2.fastq.gz, SRR8600278_2.fastq.gz, SRR8600281_2.fastq.gz, SRR8600282_2.fastq.gz, SRR8600283_2.fastq.gz, SRR8600285_2.fastq.gz, SRR8600287_2.fastq.gz, SRR8600288_2.fastq.gz, SRR8600291_2.fastq.gz, SRR8600294_2.fastq.gz, SRR8600296_2.fastq.gz, SRR8600302_2.fastq.gz, SRR8600308_2.fastq.gz, SRR8600309_2.fastq.gz, SRR8600311_2.fastq.gz, SRR8600319_2.fastq.gz, SRR8600327_2.fastq.gz, SRR8600328_2.fastq.gz, SRR8600329_2.fastq.gz, SRR8600331_2.fastq.gz, SRR8600339_2.fastq.gz 
```
$\endgroup$
2
  • $\begingroup$ This is easy. I'll respond over the weekend. $\endgroup$
    – M__
    Commented May 29 at 21:07
  • $\begingroup$ There is brand new spades release with a bunch of improvements. You may check if it is compatible with your HPC setup. github.com/ablab/spades/releases/tag/v4.0.0 $\endgroup$
    – darked89
    Commented Jun 8 at 8:25

2 Answers 2

5
$\begingroup$

I have some time now, so I'm going to take a shot at this, @M__

You need to run spades per read pair. When you specify *_1.fastq.gz as the -1 param, you're giving it every _1 file at once, not one _1 file per spades command.

I'd recommend this:

for smpl in $(ls *_1.fastq.gz | xargs -I infile basename infile _1.fastq.gz)
do
    cat > "./${smpl}_spades.slurm" <<EOF
#!/bin/bash  
 
#SBATCH --job-name=pangenome_spades.${smpl}  
#SBATCH --account=lilianasalvador
#SBATCH --partition=standard
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --mem-per-cpu=5gb
#SBATCH --time=02:00:00

spades.py -1 ${smpl}_1.fastq.gz -2 ${smpl}_2.fastq.gz -o groups/lilianasalvador/noah_100/SPAdes_output_${smpl} -t 16 --careful
EOF
done

Run a sanity check on a few of the .slurm files, then

ls *.slurm | xargs -I job_file sbatch job_file
$\endgroup$
4
  • $\begingroup$ Cool, thanks. I was going to solve this via a formal Slurm array job, but this will work. $\endgroup$
    – M__
    Commented May 30 at 3:16
  • 2
    $\begingroup$ Please do add an array-based answer, I'm not familiar with arrays but would love to learn $\endgroup$
    – Ram RS
    Commented May 30 at 13:26
  • $\begingroup$ Thank you! This worked but we're only getting one contigs.fasta file when we run. Should we be getting a contigs.fasta per sequence pair or one for all the sequences we ran? $\endgroup$
    – Lucy Kelly
    Commented Jun 6 at 19:00
  • $\begingroup$ I am not familiar with spades but going by M__'s answer, you need something at the -o param as well. I'll edit my answer but you should test running 2 samples at the same time before running all of the scripts concurrently. $\endgroup$
    – Ram RS
    Commented Jun 7 at 20:46
3
$\begingroup$

The job array is a formal approach to solving this via SLURM

What I am not clear about is the number of files SRR8599992_1.fastq.gz SRR8599992_2.fastq.gaz I originally thought the final number of files was around 100, I've guessed at 120

#SBATCH --job-name=pangenome_spade
#SBATCH --account=lilianasalvador
#SBATCH --ntasks=120
#SBATCH --cpus-per-task=16 
#SBATCH --mem-per-cpu=5gb  
#SBATCH --time=02:00:00 
#SBATCH --array=599992-5600100

module load python/3.9  
module load spades/3.15.0-hpc 

spades.py -1 'SRR'${SLURM_ARRAY_TASK_ID}'_1.fastq.gz' -2 'SRR'${$SLURM_ARRAY_TASK_ID}'_2.fastq.gz' -o /groups/lilianasalvador/noah_100/'SPAdes_output_'${SLURM_ARRAY_TASK_ID}'.fastq' -t 16 --careful 

I would test this via an echo test script writing to a specific file to make sure the spades.py command produces the correct command line for each iteration.

Things to note ...

  • Array jobs is specified by the $SLURM_ARRAY_TASK_ID variable and #SBATCH --array=599992-5600100
  • ntasks is the total number of spades assemblies you are performing here I guessed at 120
  • You are parallelising via -t 16 therefore this must be reflected in your submission job via --cpus-per-task
  • The ouput file MUST use $SLURM_ARRAY_TASK_ID otherwise you will continually override your out file.

This is not an optimal SLURM job because ...

  • -t 16 you need to check with your HPC engineer, this might take a long time to get through the queue and an alternative architecture to permit 16 CPUs can get onto the cluster. HTC users will jump ahead of you, so on a hypervisor setup you can specify a separate cluster for your work. This is not well explained but hope it makes sense.
  • Shifting to high throughput computing (HTC) could be considered here to get this through a lot quicker, e.g. this is 120 sequential assemblies, 10 sequential assemblies for 12 separate submissions would be a lot quicker (you just alter the "array job" inputs range.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.