1
$\begingroup$

enter image description hereenter image description hereI was trying to run a slurm job script inside nextflow process. Is it possible? For instance, I have a job script 'PRE_PHASING_QC.JOB' and outside nextflow it runs perfectly, I want to integrate it in a nextflow script. May be my example code could be an explanation of what I want.

prephase='PRE_PHASING_QC.JOB'
process prephasing {
input:
file(prephasing) from prephase
"""
sbatch "${prephasing}
"""
}

The error:

sbatch: error: This does not look like a batch script
...

I tried to make my job script executable, put it in bin and then include it in the nextflow process according to @Steve. So, the script runs without any error but got no output. The original job script takes 3 files (bed,bim,fam) as input and produce multiple outputs. I guess that I have to declare outputs inside nextflow process. So far I tried as follows:

process prephasing {
publishDir params.out, mode: 'copy'
output:
tuple val(chr),val(num),val(steps),path("${chr}${num}.${steps}.{bed,bim,fam,log}") into out1
script:
"""
PRE_PHASING_QC.job
"""

It gives:

Error executing process > 'prephasing'
Caused by:
Missing value declared as output parameter: chr

My expected output would be chr{1-22}.step{1-10}.{bed,bim,fam,log}

#!/bin/bash 
#SBATCH --job-name=prephase 
#SBATCH --account=p697_norment 
#SBATCH --time=1:00:00 
#SBATCH --ntasks=1 
#SBATCH --mem-per-cpu=7600M 
#SBATCH --cpus-per-task=7 
#SBATCH --array=1-22 

source /cluster/bin/jobsetup 

set -o errexit 

module load plink 

export prefix=chr${SLURM_ARRAY_TASK_ID} 
export hrc_sites_file=/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/HRC.r1-1.GRCh37.wgs.mac5.sites.tab.gz 
export HRC_1000G_check_bim=/cluster/projects/p697/github/norment/moba_qc_imputation/software/HRC-1000G-check-bim.pl

# Step 1a (exclude duplicated), 1b (exclude indels and non-ACTG alleles), 1c (exclude ambiguous SNPs) 

plink --bfile hce-ec-eur-fin-pass-qc --chr ${SLURM_ARRAY_TASK_ID} --out $prefix --make-bed || true
plink --bfile $prefix --list-duplicate-vars ids-only suppress first --out $prefix.stepla || true 
plink --bfile $prefix --exclude $prefix.stepla.dupvar --make-bed --out $prefix.stepla || true 
plink --bfile $prefix.stepla --snps-only just-acgt --make-bed --out $prefix.steplb || true 

awk '{ if (($5=="T" && $6=="A")||($5=="A" && $6=="T")ii($5=="C" && $6=="G")||($5=="G" && $6=="C")) print $2, "ambig" ; else print $2 ;}' $prefix. steplb.bim | grep ambig > $prefix.steplb.snplist.txt || true 
plink --bfile $prefix.steplb --exclude $prefix.steplb.snplist.txt --make-bed --out $prefix.steplc || true 

# Step 2. Zero out Mendelian errors
plink --bfile $prefix.steplc --me 1 1 --set-me-missing --make-bed --out $prefix.step2 || true 

# Step 3. Missingness and HWE + Step 4. Compute allele frequencies 
# NB! --mind 0.02 filter is removed from the next command, to avoid sample QC per chunk, which may leave inconsistent number of individuals avail able per chunk. 
plink --bfile $prefix.step2 --hwe le-6 --geno 0.02 --make-bed --out $prefix.step3 || true 
plink --bfile $prefix.step3 --freq --make-bed --out $prefix.step4 || true 

# Step 4-10. Run Oxford perl script and apply first 5 lines of its Run-plink.sh command, followed by '--update-name' command (update marker name s) 
perl ${HRC-IRC_1000G_check_bim} -b $prefix.step4.bim -f $prefix.step4.frq -r ${hrc_sites_file} -h || true 
plink --bfile $prefix.step4 --exclude Exclude-$prefix.step4-HRC.txt --make-bed --out $prefix.step5 || true 
plink --bfile $prefix.step5 --update-chr Chromosome-$prefix.step4-HRC.txt --make-bed --out $prefix.step6 || true 
plink --bfile $prefix.step6 --update-map Position-$prefix.step4-HRC.txt --make-bed --out $prefix.step7 || true 
plink --bfile $prefix.step7 --flip Strand-Flip-$prefix.step4-HRC.txt --make-bed --out $prefix.step8 || true 
plink --bfile $prefix.step8 --a2-allele Force-Allele1-$prefix.step4-HRC.txt --make-bed --out $prefix.step9 || true 
plink --bfile $prefix.step9 --update-name ID-$prefix.step4-HRC.txt --real-ref-alleles --make-bed --out $prefix.step10 || true
$\endgroup$
9
  • 1
    $\begingroup$ When you say your script ran without error but you got no output files, are you looking in the workDir or publishDir? When I'm developing, I usually only write the output block declaration after I've tested the tool/script a couple of times. This is because your task process can still run successfully without an output block declaration. I can then look in the workdir to see what files have been created, etc, then finally declare them in the output block. Then I start on the next (downstream) process(es) and repeat. $\endgroup$
    – Steve
    Mar 28 at 23:59
  • $\begingroup$ The publishDir is empty and in the workDir there are only log files (chr.log, chr.step1.log, chr.step2.log etc.). It is strange! $\endgroup$ Mar 29 at 7:37
  • 1
    $\begingroup$ So only files declared in your output block will get published into your publishDir. Ignore output in the publishDir while testing and developing. Or better, comment out the publishDir directive until you get your script block working in some capacity. In any case, your job script will show you what's going on. Happy to take a look if you need help. $\endgroup$
    – Steve
    Mar 29 at 10:08
  • $\begingroup$ Thank you @Steve. I added the job file as a png (not sure how to add text file). In line 21 hce-ec-eur-fin-pass-qc is the only input which create chr(1-22).(bed,bim,fam) files (for 22 chromosomes) which are used in the subsequent commands. I tried to replace hce-ec-eur-fin-pass-qc with $1 but got no output. Maybe you could give a better idea. $\endgroup$ Mar 29 at 13:15
  • 1
    $\begingroup$ Please correct me if I'm wrong, but it looks like this script kicks off a job array. This isn't a good fit inclusion "as-is" into a Nextflow workflow, since it won't be portable at all. I think breaking it up into two scripts is the best option here. I'll see if I can give an example of this. Also, you can paste text in a code-block. I've updated your question with the text. Click edit to see the changes I've made. $\endgroup$
    – Steve
    Mar 29 at 22:45
1
$\begingroup$

Using Nextflow to submit jobs this way makes little sense, since:

sbatch exits immediately after the script is successfully transferred to the Slurm controller and assigned a Slurm job ID. The batch script is not necessarily granted resources immediately, it may sit in the queue of pending jobs for some time before its required resources become available.

Assuming your job script is basically a shell script, make it executable (e.g. chmod +x PRE_PHASING_QC.JOB) and move it into a folder called bin in the root directory of your project repository. Nextflow automatically adds this folder to your $PATH, making scripts in this folder accessible in your pipeline without needing localize them and without needing to specify an absolute path to invoke them. Your code then becomes:

process prephasing {

    """
    PRE_PHASING_QC.JOB [options] <args>
    """
}

Finally, add process.executor = 'slurm' to your nextflow.config file. You may need to configure some other options, like resource requests, but that should get your script integrated into your pipeline without much refactoring, if any.



Contents of PRE_PHASING_QC.sh:

#!/bin/bash

set -euo pipefail

hrc_sites_file="${1}"
input_prefix="${2}"
chrom="chr${3}"

# Step 1a (exclude duplicated), 1b (exclude indels and non-ACTG alleles), 1c (exclude ambiguous SNPs)

plink --bfile "${input_prefix}" --chr "${3}" --out "${chrom}" --make-bed
plink --bfile "${chrom}" --list-duplicate-vars ids-only suppress first --out "${chrom}.step1a"
plink --bfile "${chrom}" --exclude "${chrom}.step1a.dupvar" --make-bed --out "${chrom}.step1a"
plink --bfile "${chrom}.step1a" --snps-only just-acgt --make-bed --out "${chrom}.step1b"

awkcmd='
($5 == "T" && $6 == "A") || ($5 == "A" && $6 == "T") ||
($5 == "C" && $6 == "G") || ($5 == "G" && $6 == "C")
{
    print $2, "ambig"
    next
}
{
    print $2
}'

awk "${awkcmd}" "${chrom}.step1b" | grep "ambig" > "${chrom}.step1b.snplist.txt"
plink --bfile "${chrom}.step1b" --exclude "${chrom}.step1b.snplist.txt" --make-bed --out "${chrom}.step1c"

# Step 2. Zero out Mendelian errors
plink --bfile "${chrom}.step1c" --me 1 1 --set-me-missing --make-bed --out "${chrom}.step2"

# Step 3. Missingness and HWE + Step 4. Compute allele frequencies
# NB! --mind 0.02 filter is removed from the next command, to avoid sample QC per chunk, which may leave inconsistent number of individuals avail able per chunk.
plink --bfile "${chrom}.step2" --hwe 1e-6 --geno 0.02 --make-bed --out "${chrom}.step3"
plink --bfile "${chrom}.step3" --freq --make-bed --out "${chrom}.step4"

# Step 4-10. Run Oxford perl script and apply first 5 lines of its Run-plink.sh command, followed by '--update-name' command (update marker name s)
HRC-1000G-check-bim.pl -b "${chrom}.step4.bim" -f "${chrom}.step4.frq" -r "${hrc_sites_file}" -h
plink --bfile "${chrom}.step4" --exclude "Exclude-${chrom}.step4-HRC.txt" --make-bed --out "${chrom}.step5"
plink --bfile "${chrom}.step5" --update-chr "Chromosome-${chrom}.step4-HRC.txt" --make-bed --out "${chrom}.step6"
plink --bfile "${chrom}.step6" --update-map "Position-${chrom}.step4-HRC.txt" --make-bed --out "${chrom}.step7"
plink --bfile "${chrom}.step7" --flip "Strand-Flip-${chrom}.step4-HRC.txt" --make-bed --out "${chrom}.step8"
plink --bfile "${chrom}.step8" --a2-allele "Force-Allele1-${chrom}.step4-HRC.txt" --make-bed --out "${chrom}.step9"
plink --bfile "${chrom}.step9" --update-name "ID-${chrom}.step4-HRC.txt" --real-ref-alleles --make-bed --out "${chrom}.step10"

Make PRE_PHASING_QC.sh executable (chmod +x PRE_PHASING_QC.sh), and replace your job script with the following. This file won't be needed by your Nextflow process, but shows you how the above shell script can be invoked:

#!/bin/bash
#SBATCH --job-name=prephase
#SBATCH --account=p697_norment
#SBATCH --time=1:00:00
#SBATCH --ntasks=1
#SBATCH --mem-per-cpu=7600M
#SBATCH --cpus-per-task=7
#SBATCH --array=1-22

set -e

. /cluster/bin/jobsetup

export PATH="/cluster/projects/p697/github/norment/moba_qc_imputation/software:${PATH}"

module load plink

cd /path/to/hce-ec-eur-fin-pass-qc

/path/to/PRE_PHASING_QC.sh \
    /cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/HRC.r1-1.GRCh37.wgs.mac5.sites.tab.gz \
    hce-ec-eur-fin-pass-qc \
    "${SLURM_ARRAY_TASK_ID}"

Copy PRE_PHASING_QC.sh and the Oxford Perl script, HRC-1000G-check-bim.pl, into your project's bin/ directory. You may also need to make the Perl script executable. Then your Nextflow process becomes:

hrc_sites_file = file(params.hrc_sites_file)

Channel
    .fromFilePairs( "${params.input_dir}/*.{bed,bim,fam}", size:3 )
    .set { plink_data }

Channel
    .of(1..22)
    .set { chroms }


process prephasing {

    input:
    tuple val(prefix), path(bfiles) from plink_data
    each chrom from chroms
    path hrc_sites_file

    """
    PRE_PHASING_QC.sh "${hrc_sites_file}" "${prefix}" "${chrom}"
    """
}

Once the above runs successfully, you can declare your process outputs. Please note that none of the above was tested; hopefully not too many typos, but YMMV.

$\endgroup$
18
  • 1
    $\begingroup$ @zillurrahman: You get the error Missing value declared as output parameter: chr because you've defined 'chr' in your output block, but this value wasn't defined in your input block (or globally in your script block). You didn't show what your input block looks like. I suspect it should look something like: tuple val(chr), val(num), val(steps), path(bfiles). I don't know what you job script does, but if your job script references absolute paths to files, you'll need to refactor it so that they can be passed in as variables on the command line. $\endgroup$
    – Steve
    Mar 28 at 23:48
  • 1
    $\begingroup$ Got it! I did as following: tuple val("${chrom}"), path("chr${chrom}*.{bed,bim,fam}") into qc1_chr and it works. Only thing is, file transfer taking too long. After finishing the process Waiting files transfer to complete (2 files) it is taking too much time. If I check output folder, I can see all my desired outputs but the waiting files transfer to complete message is still on the screen and I can not do anything in this tab. Any suggestion? However, Thanks for helping all the way! $\endgroup$ Mar 30 at 21:49
  • 1
    $\begingroup$ Well done getting it working @zillurrahman. Is the issue that the file copying is taking too long, or is the issue just that while you're waiting you cannot use the current terminal? If it's the latter, it's not really a problem since the job can be run non-interactively (e.g. submitted as a job to your Slurm cluster), in the background (e.g. using screen or nohup) or can be backgrounded after i'ts started (e.g. using <ctrl-z>, then typing 'bg') to background the process. $\endgroup$
    – Steve
    Mar 31 at 3:33
  • 1
    $\begingroup$ Thanks @Steve. I have managed to run it using sbatch. It is ok now. $\endgroup$ Mar 31 at 8:42
  • 1
    $\begingroup$ @zillurrahman you could use separate output channels for each step, but you might not need the other step outputs. Another way would be to select just the files you need from the qc1_chr output channel using the filter operator. There's lots of ways to do this - for example: qc1_chr.filter { bfile -> ["bed", "bim", "fam"].any { bfile.name.endsWith("step10.${it}") } }.view(). HTH. $\endgroup$
    – Steve
    Apr 3 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.