Passing around complex metadata in Nextflow

Question

I have a number of samples (from the 1000 genomes) that I want to analyse using Nextflow. However, I need to keep track of their metadata, fields like sample ID, analysis type etc. So I have a basic metadata object like this for each sample that I then pipe into a channel:

            metadata = [
                sample: split[0],
                mapped: split[1],
                platform : split[2],
                alignment: split[3],
                tsi: split[4],
                type: split[5],
                date: split[6],
                filetype: split[7..-1],
                url: file(bam)
            ]

Then, in the first process, I attempt to use this metadata object:

process bamToFastq {
    input:
        val bam

    container 'quay.io/biocontainers/samtools:1.8--4'
    cpus 4
    disk '50 GB'

    output:
        file('r*.fastq') into fastq

    """
    samtools fastq -0 r0.fastq -1 r1.fastq -2 r2.fastq -@ ${task.cpus} $bam.url
    """
}

However, I already run into some issues:

• Nextflow doesn't realise that metadata.url is a file, since I declared the whole object using val, so I'm guessing it won't be correctly localized (based on the documentation).
• I want pass down a new metadata object for each of my generated FASTQs, that's the same as the input one, but I want to add in the FASTQ as a URL, and the read number of each FASTQ. Where/how can I do this? Nextflow doesn't provide any section of the process for structuring the output data. If this were Python I would do the following:

    metadata = bam.copy()
    metadata['fastq'] = file
    metadata['read_number'] = file.name[1]

Is there a good way to pass around structured metadata like this in nextflow? Will this solve either of my problems?

Answer 1

It's a good idea to pass around structured metadata, but together with data/files, as tuples: tutorial here

So, like this:

process bamToFastq {
    input:
        tuple val(meta), path(bam)

    container 'quay.io/biocontainers/samtools:1.8--4'
    cpus 4
    disk '50 GB'

    output:
        tuple val(meta), file('r0.fastq'), emit: read_other
        tuple val(meta), file('r{1,2}.fastq'), emit: read_paired
    """
    samtools fastq -0 r0.fastq -1 r1.fastq -2 r2.fastq -@ ${task.cpus} $bam
    """
}

If you need to update/carry over URLs/other metadata, I recommend doing this outside of processes - using map constructs on channel outputs.