Within a Nextflow workflow, some process runs a shell script that creates multiple variables, e.g.
var_a=$(echo "1")
var_b=$(echo "2")
var_c=$(echo "3")
var_d=$(echo "4")
These variables contain statistics (e.g. number of input reads, quality scores, ...) that I want to keep until the end of the pipeline, to then save them and perform QC. So I'd like to export them out of this process and carry them to the end of the workflow. To be clear, the number of variables is not unknown, it's just too many for convenience if I need to list them all every time.
I understand that I can use the env
qualifier or an intermediate file to export the variables individually:
process get_vars {
output:
tuple env(var_a), env(var_b), env(var_c), env(var_d)
shell:
'''
var_a=$(echo "1")
var_b=$(echo "2")
var_c=$(echo "3")
var_d=$(echo "4")
'''
}
workflow{
get_vars | view
}
with output:
[ea/a145b4] process > get_vars [100%] 1 of 1 ✔
[1, 2, 3, 4]
However that becomes very impractical if I have a lot of such variables (that I do want to continue passing along to later processes, along with a much longer tuple of other relevant variables).
My question: is there a convenient way to "encapsulate" all these variables in the output channel, for example with a Groovy map similar to the meta
map in nf-core? So I could directly write something like:
process get_vars {
output:
val(my_vars)
script:
'''
var_a=$(echo "1")
var_b=$(echo "2")
var_c=$(echo "3")
var_d=$(echo "4")
'''
my_vars = [:]
my_vars.a = var_a
my_vars.b = var_b
my_vars.c = var_c
my_vars.d = var_d
}
workflow{
get_vars | view
}
which obviously fails because var_a
is not defined outside of the '''
block.
path
. But that means parsing again a file of variables I have already parsed, which doesn't feel like the most direct approach. If you post it as an answer I would accept it. $\endgroup$