# Autodetect max number of cores and pass as an argument in Nextflow

I am creating a pipeline in Nextflow. One step is creating a pangenome with Roary. Roary takes threads as an argument and if no number of threads is supplied as an argument it defaults to one.

Is there a way in Nextflow to pass the maximum number of available threads as an argument? At the moment the best I can come up with is defining an arbitrarily large number of threads as an argument which will cap out and run with the max number of threads. I would like to know, is there a 'cleaner' or more canonical way of doing this?

At the moment my process looks like this:


process roary {

publishDir "\${params.outdir}/roary", mode: 'copy'

cpus 200

input:
file gff from gff.collect()

output:
file("*") into roary
file("pan_genome_reference.fa") into pan_genome
file("pan_genome_sequences/*") into alignment_files
file("gene_presence_absence.Rtab") into gene_presence
set file("*accessory*"),
file("*.Rtab"),
file("_*"),
file("*.txt"),
file("*csv") into roary_out

script:
"""
roary -p $${task.cpus} -e -n -v -z$$gff
"""
}


Thanks in advance. Also, apologies as I can see why this might not strictly be bioinformatics, but it's where Nextflow has the most traction.

• Workflow management has a firm foothold in bioinformatics and is in my opinion directly on topic on this site. May 29 '19 at 13:01

I have not read the full Nextflow manual in depth, but I think there is no Nextflow command to detect the maximum number of threads you can use.

However, you can try to automatize this task by using the nproc command.

nproc gives you the number of processing units available. An idea would be to include something such nproc - 1 as the max number of threads for Roary.

• Thanks, as far I could see there are ways to dynamically allocate resources (nextflow.io/docs/latest/…) but not detect threads, so this is a nice workaround.
– TW93
May 29 '19 at 14:46

Here's an example I wrote recently:

#!/usr/bin/env nextflow

params.requestedCPU = 80
params.requestedMem = 50

maxcpus = Runtime.runtime.availableProcessors()
requestedCpus = Math.round((params.requestedCPU * maxcpus) / 100)

maxmem = Runtime.runtime.totalMemory() / 10241024
requestedMem = (Math.round((params.requestedMem * maxmem) / 100))

process foo {
echo true
cpus = requestedCpus
memory = requestedMem.GB

script:
"""
echo $$requestedCpus echo$$requestedMem
"""
}


You can pass the number of CPUs or Memory directly on the command line in percentage, e.g. nextflow run main.nf --requestedCPU 100 to have all CPU usages. Hope this helps.