0
$\begingroup$

I have a process like the following:

process process_dicoms {

    def project_dir = projectDir

    input:
    tuple val(accession_number), val(dicom_file) from AllDicomsChannel

    output:
    stdout ProcessedDicomsChannel

    """
    python $project_dir/python/process_one_dicom_file.py \
    --accession_number="$accession_number" \
        --dicom_file="$dicom_file"
    """
}

When the execution command string references the local variable $project_dir, as above, then all is well and the "-resume" option behaves as required and skips the task on subsequent runs.

However when I don't re-declare the variable locally i.e. change the execution command string to refer to the global $projectDir variable directly:

    """
    python $projectDir/python/process_one_dicom_file.py \
        --accession_number="$accession_number" \
        --dicom_file="$dicom_file"
    """

the the "-resume" option does not work for this command. On inspection of the hash codes using "-dump-hashes" the offending line is a broken hash code, for a HashMap<> object which contains "projectDir":

This is the hashes generated in the initial run

  7c75127853ea5d112d8c731e4c68d64f [java.lang.String] accession_number 
  1f5a83a6919640bc7bfa9ec7302b27b4 [java.lang.String] 1234
  a06e84a588350beddebd145ec6d87d32 [java.lang.String] dicom_file 
  7321a200798ce34efdf6bda145258f51 [sun.nio.fs.UnixPath] /mnt/c/...
  4f9d4b0d22865056c37fb6d9c2a04a67 [java.lang.String] $ 
  16fe7483905cce7a85670e43e4678877 [java.lang.Boolean] true 
  194bf1fdde8ad8c4f2bdd2801831af36 [java.util.HashMap$EntrySet] [projectDir=/mnt/c/temp/admani2-transform]

while this is for the second run with -resume enabled (NOTE final hash code for HashMap$EntrySet is different!):

  7c75127853ea5d112d8c731e4c68d64f [java.lang.String] accession_number 
  1f5a83a6919640bc7bfa9ec7302b27b4 [java.lang.String] 1234
  a06e84a588350beddebd145ec6d87d32 [java.lang.String] dicom_file 
  7321a200798ce34efdf6bda145258f51 [sun.nio.fs.UnixPath] /mnt/c/...
  4f9d4b0d22865056c37fb6d9c2a04a67 [java.lang.String] $ 
  16fe7483905cce7a85670e43e4678877 [java.lang.Boolean] true 
  2ee24c33e8559182a0acbdda69ad86be [java.util.HashMap$EntrySet] [projectDir=/mnt/c/temp/admani2-transform]   <----- This hash code is different even though HashMap identical!!

My workaround is to redeclare variables locally as needed to ensure the cache works. The issue seems similar to this one which was resolved by a workaround: https://github.com/nextflow-io/nextflow/issues/1576

Is this a bug in Nextflow? A result of hashcode instability of Java HashMap<>? Or am I doing something wrong it? It seems odd that "$=True" is in the hash dump is this part of the problem?

(NOTE: I can redeclare "def project_dir = projectDir" at the top of the file once and for all so the workaround isn't too onerous, but still... <-- NOTE workaround doesn't work in DSL2!!)

(NOTE: The same behavior occurs when referencing "${params.anything}" in an execution string, and the same workaround works.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.