1
$\begingroup$

Within a Nextflow workflow, some process runs a shell script that creates multiple variables, e.g.

var_a=$(echo "1")
var_b=$(echo "2")
var_c=$(echo "3")
var_d=$(echo "4")

These variables contain statistics (e.g. number of input reads, quality scores, ...) that I want to keep until the end of the pipeline, to then save them and perform QC. So I'd like to export them out of this process and carry them to the end of the workflow. To be clear, the number of variables is not unknown, it's just too many for convenience if I need to list them all every time.

I understand that I can use the env qualifier or an intermediate file to export the variables individually:

process get_vars {
  output:
    tuple env(var_a), env(var_b), env(var_c), env(var_d)
  
  shell:
  '''
  var_a=$(echo "1")
  var_b=$(echo "2")
  var_c=$(echo "3")
  var_d=$(echo "4")
  '''
}

workflow{
  get_vars | view
}

with output:

[ea/a145b4] process > get_vars [100%] 1 of 1 ✔
[1, 2, 3, 4]

However that becomes very impractical if I have a lot of such variables (that I do want to continue passing along to later processes, along with a much longer tuple of other relevant variables).

My question: is there a convenient way to "encapsulate" all these variables in the output channel, for example with a Groovy map similar to the meta map in nf-core? So I could directly write something like:

process get_vars {
  output:
    val(my_vars)
  
  script:
  '''
  var_a=$(echo "1")
  var_b=$(echo "2")
  var_c=$(echo "3")
  var_d=$(echo "4")
  '''
  
  my_vars = [:]
  my_vars.a = var_a
  my_vars.b = var_b
  my_vars.c = var_c
  my_vars.d = var_d
}

workflow{
  get_vars | view
}

which obviously fails because var_a is not defined outside of the ''' block.

$\endgroup$
3
  • 1
    $\begingroup$ Can't you simple write all vars to a plain text file and then have some groovy/nf code to loop over the content and pass the it to any data structure that is appropriate, be it a map, list, anything? $\endgroup$
    – ATpoint
    Sep 20 at 8:16
  • $\begingroup$ That does sound like a solution, at least I would only have to pass along a path. But that means parsing again a file of variables I have already parsed, which doesn't feel like the most direct approach. If you post it as an answer I would accept it. $\endgroup$
    – Alexlok
    Sep 20 at 14:23
  • $\begingroup$ I think you can use output for this, ouput: env varname, as is mentioned here nextflow.io/docs/latest/process.html#outputs $\endgroup$
    – Niklas
    Sep 20 at 19:27

1 Answer 1

2
$\begingroup$

I am not sure if i get what you mean by encapsulate, you can give your tuple a name using emit?

process get_vars {
  output:
    tuple env(var_a), env(var_b), env(var_c), env(var_d), emit: vars

  script:
  '''
  var_a=$(echo "1")
  var_b=$(echo "2")
  var_c=$(echo "3")
  var_d=$(echo "4")
  '''
}

workflow {
  get_vars()
  get_vars.out.vars.view()
}

That gives:

 [1, 2, 3, 4]
$\endgroup$
5
  • $\begingroup$ If I'm not mistaken, using emit (described in the DSL2 docs) is to create a separate (named) channel, so I still have to ensure that it keeps the order in sync with other channels, right? $\endgroup$
    – Alexlok
    Sep 21 at 16:11
  • $\begingroup$ Here is an illustration keeping in sync with a main channel: process get_vars { input: val(main_chan) output: tuple env(var_a), env(var_b), env(var_c), env(var_d), emit: vars val(main_chan), emit: main_chan shell: ''' var_a=!{main_chan} var_b=$((!{main_chan} + 1)) var_c=$((!{main_chan} + 2)) var_d=$((!{main_chan} + 3)) ''' } process next_process { input: val(vars) val(main_chan) output: tuple(val(vars), val(main_chan)) script: ''' ''' } $\endgroup$
    – Alexlok
    Sep 21 at 16:11
  • $\begingroup$ workflow { main_chan = Channel.of( 1, 2, 3 ) get_vars(main_chan) next_process(get_vars.out.vars, get_vars.out.main_chan) next_process.out.view() } $\endgroup$
    – Alexlok
    Sep 21 at 16:11
  • $\begingroup$ The key here is that next_process(get_vars.out.vars, get_vars.out.main_chan) needs to receive both channels from the previous process at once. Feel free to include this in your answer if you think it would be relevant $\endgroup$
    – Alexlok
    Sep 21 at 16:13
  • $\begingroup$ I see, you can use a id (like sample-name or meta.id in nfcore) that you include in both channels and then use to join them together into a single channel to make sure that information for one sample moves though the pipeline together. $\endgroup$
    – Niklas
    Sep 21 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.