3
$\begingroup$

Bug fixed

This issue appears to have been fixed in latest Nextflow: https://github.com/nextflow-io/nextflow/issues/2693

I've been using Nextflow for various workflows on a cluster using the SLURM scheduler. I've noticed that the 'parent' job does not shut down when the Nextflow workflow finishes successfully. I can cancel the job, but then Nextflow doesn't get a chance to clean things up and also doesn't write out the trace, report, etc. The parent job does end properly if the workflow ends with an error.

Question: Why doesn't the parent Nextflow job end upon successful completion?

I saw a post on the Nextflow GitHub where the original poster describes various acrobatics to get the parent job to exit cleanly, but this is something that Nextflow should handle.

Here's my SLURM submission script that I submit using sbatch run_workflow.slurm

run_workflow.slurm

#!/bin/bash
#SBATCH --time=60:15:00                             # Time limit for the job (REQUIRED).
#SBATCH --job-name=NF_Parent_Identify_D&C_regions   # Job name
#SBATCH --ntasks=1                                  # Number of cores for the job. Same as SBATCH
#SBATCH --mem=5G                                    # Total memory requested
#SBATCH --partition=<partition>                     # Partition/queue to run the job in. (REQUIRED
#SBATCH -e slurm-%j.err                             # Error file for this job.
#SBATCH -o slurm-%j.out                             # Output file for this job.
#SBATCH -A <account>                                # Project allocation account name (REQUIRED)

module load ccs/java/jdk1.8.0_202
nextflow run identify_dark_and_camouflaged_regions.nf \
    -with-report \
    -with-trace \
    -with-dag idcr-workflow.png \
    -with-timeline \
    -resume

Update 1

I'm adding example Nextflow output from a different workflow (I don't have the SLURM output from the one I mention above) where the workflow has completed, but the parent job is still running. In this case, there were errors, but I added errorStrategy 'ignore' to all of the processes listed:

executor >  slurm (876)
[06/c5672b] process > samtools_csort_proc (A-CUHS... [100%] 304 of 304, faile...
[45/26a046] process > Lsamtools_csort_proc (A-CUH... [100%] 304 of 304, faile...
[b5/0f2d7c] process > sambamba_csort_proc (A-CUHS... [100%] 304 of 304, faile...

So, regardless of whether the workflow completed successfully, Nextflow has nothing left to do, but it's still running.

Update 2 (adding partial .nextflow.log file)

Per @Steve's point, I checked the .nextflow.log file and am including part of the file, here. In this case, I don't think Nextflow has any tasks related to publishDir because I'm not using that in this workflow. The job that's still running is from the Nextflow workflow in my GitHub repository for my answer posted in my related question Samtools sort: most efficient memory and thread settings for many samples on a cluster. Whether it's reproducible, I haven't checked. Not sure how common this issue is.

In either case, my interpretation of the .nextflow.log file below is that Nextflow thinks it's hung, too. I just don't know why, or how to exit cleanly.

Partial view of .nextflow.log

...

Feb-28 05:29:04.607 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: RUNNING; exit: -; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: -; ]
Feb-28 05:30:54.613 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: COMPLETED; exit: 0; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: 2022-02-28T10:30:53.516015Z; ]
Feb-28 05:34:04.641 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
Feb-28 05:39:04.647 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
Feb-28 05:44:04.653 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
Feb-28 05:49:04.658 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
Feb-28 05:54:04.662 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled

< above line repeats hundreds of times >

...
Improve this question
$\endgroup$
1
  • $\begingroup$ Can you please create an answer with your accepted solution, so that the StackExchange algorithms don't confuse this with an unanswered question? $\endgroup$
    – gringer ♦
    Commented May 17, 2022 at 5:21

1 Answer 1

Reset to default
2
$\begingroup$

How do you know your workflow actually finishes successfully? One way to check the workflow termination status is to use the workflow.onComplete and workflow.onError event handlers. Consider resuming your workflow with something like the following appended to your script:

workflow.onComplete {

  log.info "Workflow completed at: ${workflow.complete}"
  log.info "Time taken: ${workflow.duration}"
  log.info "Execution status: ${workflow.success ? 'success' : 'failed'}"
}

workflow.onError {

  log.info "Execution halted: ${workflow.errorMessage}"
}

Note that the workflow event handlers may also be defined inside your nextflow.config if preferred.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks, @Steve. I added an example to my post. Seems that one way or another, the example I show above should have completed because Nextflow has nothing left to do. $\endgroup$
    – Mark Ebbert
    Commented Feb 28, 2022 at 14:45
  • $\begingroup$ @MarkEbbert It's hard to say what the problem is here exactly without the nextflow.log or a reproducible example. But just because all of the tasks have completed (regardless of the exit status) doesn't necessarily mean Nextflow has nothing left to do. For example, the publishDir directive works asynchronously and can be used (as you know) to 'copy' task outputs. If these are large and there are many of these, Nextflow will hang around copying files from the queue until well after the last task has completed. $\endgroup$
    – Steve
    Commented Mar 1, 2022 at 0:14
  • $\begingroup$ Thanks, @Steve. I just added part of the .nextflow.log file. Looks like Nextflow thinks it's hung, too? I haven't tested whether this is reproducible, by this is the code for my related question. I included a link to it above. This one may just remain a mystery. $\endgroup$
    – Mark Ebbert
    Commented Mar 1, 2022 at 19:07
  • $\begingroup$ I can't edit my above comment, but should read "..., but this is from the code..." $\endgroup$
    – Mark Ebbert
    Commented Mar 1, 2022 at 19:25
  • 1
    $\begingroup$ Thanks, @Steve. Submitted: github.com/nextflow-io/nextflow/issues/2693. I'll report back if I learn something. $\endgroup$
    – Mark Ebbert
    Commented Mar 2, 2022 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.