Timeline for Nextflow: run sbatch (slurm job script) inside a process
Current License: CC BY-SA 4.0
23 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 5, 2021 at 10:25 | comment | added | zillur rahman |
Thanks @Steve I posted a new question with code for each channel here; bioinformatics.stackexchange.com/questions/15680/…
|
|
| Apr 5, 2021 at 10:14 | comment | added | Steve |
I'm not sure exactly @zillurrahman, sorry. Your input declaration doesn't look right, but I'd still expect some output or log on stderr if your process is executed. If you can ask a new question, I'll see if I can add an answer. Please also show the contents of shape_in, shape_ref, shape_legend and shape_sample. Cheers.
|
|
| Apr 4, 2021 at 17:48 | comment | added | zillur rahman |
Thanks @Steve. It seems ok now, getting no error. Another issue. I merged other channels with it (each channel for 22 files, 1 for each chromosome) shape_para=shape_in.merge(shape_map).merge(shape_ref).merge(shape_legend).merge(shape_sample) process phasing { input: each file(penta) from shape_para output: path("*step10*") into phased script: def (in, map, ref, legend, sample) = penta """ shapeit2 -T 32 --input-bed "${in}" --input-map "${map}" --input-ref "${ref}" "${legend}" "${sample}" --duohmm -O "${in}.phased" --seed 54321 """ } I am getting no output or error!
|
|
| Apr 4, 2021 at 10:58 | comment | added | Steve |
Sorry @zillurrahman I didn't see your qc1_chr declaration above. I think you want: qc1_chr.filter { name, bfiles -> bfiles.collect { it.getBaseName() }.every { it.endsWith("step10") } } HTH.
|
|
| Apr 3, 2021 at 16:24 | comment | added | zillur rahman |
Thanks @Steve. It gives me No such variable: Exception evaluating property 'name' for java.util.ArrayList, Reason: groovy.lang.MissingPropertyException . If I remove name from the above snippet, I got Unknown method invocation 'endsWith' on ArrayList type. Any idea?
|
|
| Apr 3, 2021 at 11:16 | comment | added | Steve |
@zillurrahman you could use separate output channels for each step, but you might not need the other step outputs. Another way would be to select just the files you need from the qc1_chr output channel using the filter operator. There's lots of ways to do this - for example: qc1_chr.filter { bfile -> ["bed", "bim", "fam"].any { bfile.name.endsWith("step10.${it}") } }.view(). HTH.
|
|
| Apr 2, 2021 at 23:07 | comment | added | zillur rahman |
Hi, This process creates multiple output files (chr1-22.step1-10.bed|bim|fam) which are saved into qc1_chr output channel. Now, say I need only those files which has string step10. How could I get that subset into a new channel? I need it into a new channel because I need to merge them with elements of other channels. Did my explanation make sense?
|
|
| Mar 31, 2021 at 8:42 | comment | added | zillur rahman |
Thanks @Steve. I have managed to run it using sbatch. It is ok now.
|
|
| Mar 31, 2021 at 3:33 | comment | added | Steve | Well done getting it working @zillurrahman. Is the issue that the file copying is taking too long, or is the issue just that while you're waiting you cannot use the current terminal? If it's the latter, it's not really a problem since the job can be run non-interactively (e.g. submitted as a job to your Slurm cluster), in the background (e.g. using screen or nohup) or can be backgrounded after i'ts started (e.g. using <ctrl-z>, then typing 'bg') to background the process. | |
| Mar 30, 2021 at 21:49 | vote | accept | zillur rahman | ||
| Mar 30, 2021 at 21:49 | comment | added | zillur rahman |
Got it! I did as following: tuple val("${chrom}"), path("chr${chrom}*.{bed,bim,fam}") into qc1_chr and it works. Only thing is, file transfer taking too long. After finishing the process Waiting files transfer to complete (2 files) it is taking too much time. If I check output folder, I can see all my desired outputs but the waiting files transfer to complete message is still on the screen and I can not do anything in this tab. Any suggestion? However, Thanks for helping all the way!
|
|
| Mar 30, 2021 at 20:40 | comment | added | zillur rahman |
Ohh! There was some typo in the PRE_PHASING_QC.sh script. I ran without the slurm, identfied those typos, edited them and now everything is Ok. Now, How could I declare the output files so that I get all the outputs in a single directory? So far, I tried tuple val("${chrom}"), val(steps), path("${chrom}.${steps}.{bed,bim,fam}") into qc1_bfile and tuple val(prefix), val("${chrom}"), path("${prefix}-${chrom}.step4-HRC.txt") into qc1_hrc But got error: Missing value decalred as output parameter: steps How could I decalre steps so that all outputs from each step gathered in a folder?
|
|
| Mar 30, 2021 at 14:19 | comment | added | Steve |
No worries @zillurrahman. I just updated the NF code to process all chromosomes like what we have done previously. I think you'd rather avoid running each chromosome one at a time. In your first comment above, you get the Missing 'bind' declaration in input parameter error since you're missing parameters for the "path/to/hce-ec-eur-fin-pass-qc" and "1" options.
|
|
| Mar 30, 2021 at 14:15 | history | edited | Steve | CC BY-SA 4.0 |
Made the prephasing example process more robust
|
| Mar 30, 2021 at 12:50 | comment | added | zillur rahman |
Thanks @Steve So far I declared input_prefix and chrom as hrc_sites_file was declared in your script and specify them in command line. I got output for chromosome 1, chr1.step1a, chr1.step1b but in the next step it shows that no such file as chr1.step1b but it exists in my work/ directory. Any thought? When I use chrom 2, I got chr2.step1a, chr2.step1b and same error in the next step.
|
|
| Mar 30, 2021 at 12:31 | comment | added | zillur rahman |
Thanks @Steve . I tried nextflow run prephasing1.nf --hrc_sites_file "path/to/hrc_sites.tab.gz" "path/to/hce-ec-eur-fin-pass-qc" "1" It give me Missing 'bind' declaration in input parameter . Should we use tuple(val),val(input_prefix),path(bfiles) from some_channel ? How could I declare input channel in this particular case? It seems I could run chromosome by chromosome? That if I put "1" in nf run parameter, it will only run for chromosome 1? Should we declare it in in_prefix=file(params.input_prefix) and params.chr=1 then we could change/call the arguments in command line?
|
|
| Mar 30, 2021 at 10:32 | history | edited | Steve | CC BY-SA 4.0 |
typo
|
| Mar 30, 2021 at 2:32 | history | edited | Steve | CC BY-SA 4.0 |
added 3965 characters in body
|
| Mar 29, 2021 at 10:02 | comment | added | Steve |
From our previous discussions, I thought plink took a prefix for it's '--bfile' option? Note this isn't the same as '--bfiles'. If I'm correct, I think your first task will be to work out how to supply a prefix on the command line so that it gets passed down to your plink command. In your NF process, I think you'll want: PRE_PHASING_QC.job "${prefix}". And in your script I think you'll want: plink --bfile "$1"
|
|
| Mar 29, 2021 at 7:45 | comment | added | zillur rahman |
My job script takes bfiles as input (e.g plink --bfiles input_bfiles --other argument --output). I just tried to modify this as : plink --bfiles $1 and tried to use input channel from nextflow. """ PRE_PHASING_QC.job "${bfiles}" """" In this case nothing happens, no error or success.
|
|
| Mar 28, 2021 at 23:48 | comment | added | Steve |
@zillurrahman: You get the error Missing value declared as output parameter: chr because you've defined 'chr' in your output block, but this value wasn't defined in your input block (or globally in your script block). You didn't show what your input block looks like. I suspect it should look something like: tuple val(chr), val(num), val(steps), path(bfiles). I don't know what you job script does, but if your job script references absolute paths to files, you'll need to refactor it so that they can be passed in as variables on the command line.
|
|
| Mar 28, 2021 at 22:59 | comment | added | zillur rahman |
Thanks @Steve . I think you understand my explanation and your answer makes sense. I have updated the question with new code, error and expected output. Hope it helps to debug
|
|
| Mar 28, 2021 at 12:28 | history | answered | Steve | CC BY-SA 4.0 |