2
$\begingroup$

I have sequenced a bacterial genome with a GridIon from ONT. Basically what I want to check is whether or not trimming 50 bps at the beginning of the reads will improve alignment against the reference genome and ultimately the call of a consensus sequence.

For that what I have done so far is:

  1. concatenated all the fastqs generated from the Gridion into a single (very large) fastq file: ref494_cat_all.fastq
  2. aligned this concatenated fastq against the reference genome ref_genbank.fasta with minimap2: ~/Downloads/minimap2-2.17_x64-linux/minimap2 -ax map-ont ref_genbank/ref_genbank.fasta ref494_cat_all.fastq > ref494_aligned_trim50.sam
  3. Performed the general samtools steps to covert sam to bam, index and sort

Then, in order to get the consensus sequence I first try to generate a .vcf.gz file, according to this tutorial

bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam | bcftools call -mv -Oz -o calls.vcf.gz

Immediatelly after calling the command above I got:

Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid
[mpileup] 1 samples in 1 input files

Then, after a minute or two, I got:

Failed to open -: unknown file type

I'm pretty sure that there's no issue with .bam file, as I could visualise the alignment with IGV or tablet.

Does anyone know what the issue can be?

$\endgroup$
  • 2
    $\begingroup$ Please capture the output of the first command and show us the first few lines (via bcftools view if it's not text): bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam > output.tmp $\endgroup$ – John Marshall Jul 9 at 7:28
  • 1
    $\begingroup$ I got Killed output to the terminal after a minute or two. Also this command generates an empty output.tmp file. $\endgroup$ – BCArg Jul 9 at 7:49
  • $\begingroup$ As noted below, killed due to memory exhaustion. However: you deserve an error message. Mpileup typically doesn't use that much memory and I have been unable to reproduce this on (non-ONT) data here. If you are able to, it would be interesting to try current develop bcftools to see if this still happens and/or to run it under the debugger and see where it crashes. $\endgroup$ – John Marshall Jul 9 at 9:12
  • $\begingroup$ The latter is quite easy: run gdb --args bcftools mpileup -f ref_genbank/ref_genbank.fasta -o /dev/null ref494_aligned.sorted.bam, then type run and wait for it to crash. Then type backtrace; the output from this will be instructive. $\endgroup$ – John Marshall Jul 9 at 9:17
  • $\begingroup$ Just before the program got killed I got Program terminated with signal SIGKILL, Killed. The program no longer exists. The output of backtrace was No stack That you cannot reproduce this error on non-ONT data, you think it can be related to the size of my input files? My bam is 14GB (coverage is about 2.000, we ran for about 48h, which we won't do routinely) $\endgroup$ – BCArg Jul 9 at 9:58
0
$\begingroup$

The - is usually used to mean standard input when reading data. So the message would suggest that the second bcftools command fails to read from stdin, so it fails to read the output of the first bcftools command:

$ echo "foo bar" | /ref/software/bcftools-1.8/bcftools call -mv -Oz -o calls.vcf.gz
Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid
Failed to open -: unknown file type

So there is something wrong with either your input file or with the output produced when running the first bcftools command on that input file. We can't really help more unless you share the file.

$\endgroup$
  • $\begingroup$ I doubt there's any problem with the input files, as I can visualise the alignment on table or IGV, so I guess the bcftoools mpileup is not generating any output (please see my comment to @John Marshall). I guess I will report the problem on their github page. $\endgroup$ – BCArg Jul 9 at 7:53
  • 1
    $\begingroup$ @BCArg yes, precisely. If you got Killed in the terminal, then the process is being killed by the operating system, most likely because it is taking up too much memory. Sounds like you need to run this on a more powerful machine. $\endgroup$ – terdon Jul 9 at 7:56
  • 1
    $\begingroup$ yes mystery solved, I monitored the the performance of the command bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam > output.tmp with htop and the program got killed right after I reached the maximum of my RAM memory (16GB), so in fact running the command on the cluster appears to be the solution. $\endgroup$ – BCArg Jul 9 at 8:12
0
$\begingroup$

It's possible you're looking at instructions for a recent version of samtools / bcftools, whereas your computer has an older version. Can you try samtools mpileup -g -f ... to output the pileup as BCF format instead of bcftools mpileup -f ...?

Update: it's good to know that the samtools / bcftools pipelines are at least producing consistent results, which suggests that the error is not associated with the output format on the mpileup side of the pipeline, and also good to see that you've identified that this was a memory issue.

The most frequent cause of memory issues I've had with sorted BAM files has been trying to figure out variants in extremely high coverage data (e.g. over 1000X), which would make sense for a bacterial isolate sequenced on a GridION. I use my own script to carry out a reservoir sampling on mapped BAM format files, which reduces coverage down to a more manageable level. Here's an example command to normalise to 100X coverage where possible:

samtools view reads.bam | ./samNormalise.pl -coverage 100 | \
   samtools view -b > reads_normalised.bam
$\endgroup$
  • $\begingroup$ did not work unfortunately, got the following messages after calling: samtools mpileup -g -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam | bcftools call -mv -Oz -o calls.vcf.gz [warning] samtools mpileup option g` is functional, but deprecated. Please switch to using bcftools mpileup in future.` Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid [mpileup] 1 samples in 1 input files Failed to open -: unknown file type $\endgroup$ – BCArg Jul 9 at 7:22
  • 1
    $\begingroup$ The VCF/BCF-outputting mpileup command was transferred from samtools to bcftools relatively recently. So in fact it is @gringer's suggestion that is instructions for an earlier version of samtools / bcftools. $\endgroup$ – John Marshall Jul 9 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.