I am running a script file on a cluster with slurm through sbatch. Locally it runs perfectly well on 2 different linux versions (linux mint, ubuntu 12.04), however, it returns -9 error code while running on a cluster which is CentOS 7.3. When we are using Pypiper and run the whole .py script file, it fails when it starts running trimmomatic program


However, when running just trimmomatic program step by itself it does not fail and finishes successfully. We have a suspicion that something may be wrong with how Pypiper runs on the cluster, however, feeling rather stuck. Any suggestions would be greatly appreciated.

Here is the error that is printed out:

ILLUMINACLIP: Using 2 prefix pairs, 22 forward/reverse sequences, 0 
forward only sequences, 0 reverse only sequences
Process 109391 returned: (-9). Elapsed: 0:00:51. Peak memory: (Process: 
5.211GB; Pipeline: 0.0GB)

### Pipeline failed at:  (12-14 15:50:47) elapsed:0:23:09 _TIME_

Total time: 0:23:09

Change status from running to failed
Traceback (most recent call last):
  File "...../salmon_rna_seq.py", line 344, in <module>
File "...../salmon_rna_seq.py", line 304, in main
File "...../salmon_rna_seq.py", line 277, in process
File "...../salmon_rna_seq.py", line 195, in trim_reads
self.pipe_manager.run(cmd, self.sample_files['r1']['fastq']['trimmed_paired'])
File "...../pypiper.py", line 508, in run
process_return_code, local_maxmem = self.callprint(cmd, shell, nofail, container)  # Run command
File "...../pypiper.py", line 691, in callprint
File "...../pypiper.py", line 1111, in fail_pipeline
raise e
OSError: Subprocess returned nonzero result.

Pypiper terminating spawned child process 108475...
child process terminated
slurmstepd: error: Exceeded step memory limit at some point.

The last line may be informative. Although, we are still not sure what we could possibly do.


Slurm is killing the process because it's using more memory per core than the default allocation (that's what slurmstepd: error: Exceeded step memory limit at some point means). I've never used pyPiper (like seemingly most people, I use snakemake), but presuming that you have it running either srun or sbatch at some point then specify something like --mem-per-cpu 10G.

  • $\begingroup$ Actually I tried that and it still does not work. Will try to write to the cluster support $\endgroup$ Dec 15 '17 at 15:12
  • $\begingroup$ You might need a larger value than 10 gigs. You can use scontrol to see the exact threshold getting crossed that's causing slurm to kill the job. $\endgroup$
    – Devon Ryan
    Dec 15 '17 at 15:22
  • $\begingroup$ Awesome. We increased it to 20Gb and it run successfully. $\endgroup$ Dec 15 '17 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.