Questions tagged [workflow-management]
The workflow-management tag has no usage guidance.
18
questions
1
vote
1
answer
139
views
Remove files after process is terminated
In my pipeline nextflow, I create a channel with channel.fromSRA and the channel contains lots of heavy files.fastq.gz. Then I have a first process to unzip files and transform them in files.fasta and ...
- 31
2
votes
1
answer
197
views
Nextflow tuple from CSV cols
I am trying to create a channel with a CSV file and use it as tuple input. Yet the output I obtain does not contain the entire file but only the first element of the file (1st row).
the csv
...
- 21
3
votes
1
answer
855
views
How to solve Nextflow error: "Trace file already exists"?
When trying to run epi2me-labs/wf-artic, I get the following error:
...
- 389
1
vote
0
answers
49
views
Implement a WDL workflow for a docker container I built
I start saying my background is Biology and I'm totally new to WDL, so I'm aware my code will be messy but I'm trying to learn.
My problem is, essentially, to run a Docker container I built and tested ...
- 21
1
vote
1
answer
258
views
Handling failed blocks within scattered WDL workflow
I'm trying to adapt an existing workflow to make it more robust to failures. Below is a simplified example of the workflow.
...
- 111
2
votes
1
answer
333
views
Mounting a folder from the host to a Docker, while running it as Cromwell runtime
I have a pipeline, which is written in WDL & runs with the help of Cromwell. I use Docker...
- 23
0
votes
0
answers
325
views
BWA-MEM2 alignment-Snakemake
I have started using snakemake 6.5.2 to align fastq files with reference file. I have pasted the error below in this question. How to allocate memory in the snakefile and read the header from samfile, ...
- 11
2
votes
1
answer
526
views
Nextflow: How to initialize channel from CSV file but only after CSV file created?
Process A runs a python script which generates a CSV file in some absolute file path say "/mnt/x/blah.csv".
Process B needs to consume the CSV file line by line, after process A has finished....
- 203
2
votes
1
answer
671
views
How to execute several processes sequentially, without channels?
Using DSL2.
I have processes A, B, C which I want to run in sequence - B waits for A to finish, C waits for B to finish.
They are not communicating via channels so I can't do A.out.collect().
I have ...
- 203
1
vote
1
answer
2k
views
How to trigger next process on completion of previous process?
Simple question but I am stumped. (I am still learning Nextflow)
I have a process which takes as input a channel of files and processes each file separately.
There is a second process which needs to ...
- 203
2
votes
1
answer
151
views
How does cromwell and AWS batch decide which compute environment (on-demand or spot) to use?
For processing genomics workflows I make use of widdle and cromwell with an AWS back-end. I have followed all examples on the AWS cromwell tutorial. In my AWS batch console I can see there are 2 job ...
- 21
1
vote
1
answer
400
views
Snakemake on cluster produces empty output files
Problem:
When I run my Snakefile on our cluster, all the processes complete successfully. I can see from the logs that my shell scripts are being run correctly and are claiming to complete ...
- 21
3
votes
1
answer
494
views
Access workdir defined on command line from within Snakefile
Snakemake provides access to a workflow object within a Snakefile. This allows one to, for example, have dynamic programmatic access to the directory containing the ...
- 5,060
3
votes
2
answers
1k
views
Autodetect max number of cores and pass as an argument in Nextflow
I am creating a pipeline in Nextflow. One step is creating a pangenome with Roary. Roary takes threads as an argument and if no number of threads is supplied as an argument it defaults to one.
Is ...
- 449
14
votes
1
answer
3k
views
What is the difference between a Bioinformatics pipeline and workflow?
I want to understand the difference between pipeline systems and workflow engines.
After reading A Review of Scalable Bioinformatics Pipelines I had a good overview of current bioinformatics ...
- 487
5
votes
1
answer
185
views
Dealing with absence of coverage
In my snakemake workflow, I use deeptools bamCoverage (in a make_bigwig rule), then ...
- 3,100
9
votes
4
answers
1k
views
Ultimate reproducibility in R?
I'm looking for a convenient and reliable way to make an R analysis reproducible, either at different times or across collaborators.
Listing the package versions or a ...
- 2,624
8
votes
3
answers
507
views
What are the ways to keep track of branches in the analysis?
I'm going through an RNA-seq pipeline in R/Bioconductor and want to try multiple parameters at subsequent steps, for example, running clustering with different ...
- 2,624