Understanding fast5 format

Question

I would like to reproduce the results on figure 1 from THIS publication. I downloaded the data and explore it with h5ls and h5dump based on HDF5. I would like to understand:

1. h5ls -r <_fast5file_> | grep Dataset:

/Analyses/Basecall_2D_000/BaseCalled_2D/Alignment Dataset {6925}
/Analyses/Basecall_2D_000/BaseCalled_2D/Fastq Dataset {SCALAR}
/Analyses/Basecall_2D_000/BaseCalled_complement/Events Dataset {5576}
/Analyses/Basecall_2D_000/BaseCalled_complement/Fastq Dataset {SCALAR}
/Analyses/Basecall_2D_000/BaseCalled_complement/Model Dataset {4096}
/Analyses/Basecall_2D_000/BaseCalled_template/Events Dataset {5793}
/Analyses/Basecall_2D_000/BaseCalled_template/Fastq Dataset {SCALAR}
/Analyses/Basecall_2D_000/BaseCalled_template/Model Dataset {4096}
/Analyses/Basecall_2D_000/HairpinAlign/Alignment Dataset {5107}
/Analyses/Basecall_2D_000/Log Dataset {SCALAR}
/Analyses/Calibration_Strand_000/Log Dataset {SCALAR}
/Analyses/EventDetection_000/Reads/Read_0/Events Dataset {11465}

Alignment? Then complement/Fastq is not the complement of template/Fastq (analyzed with h5dump). What is it? Also, events have sometimes 4

2. Some other datasets that were acquired somewhere else have the raw data and fastq but there is no associated information about kmer and mean(current)/std/length/etc... How do I generate events from raw data?

I went through all documentations and tutorials that I could found in the web but couldn't find anything simple and complete. It feels as if it was a secret.

Example fast5file would be ecoli_er2925.MSssI.timp.100215.fast5/pass/kelvin_100215_ecolisssI_2515_1_ch195_file0_strand.fast5 from PRJEB13021

Answer 1

FAST5 is a proprietary format developed by Oxford Nanopore Technologies, and there's not much great documentation online. All FAST5 files will have the Raw/ field, which contains the original measured Signal. Additional Analyses/ fields can be added by tools such as basecallers which convert signal->ACGT (Guppy) or signal-level aligners (Tombo).

In particular, /Analyses/Basecall_2D_000/ contains the results of 2D basecalling, added by Guppy when run with the --post-out flag. I think Tombo adds the event-level annotations you mention (k-mer mean/std/length). In 2D basecalling, both the physical forward and reverse strands for the same stretch of DNA are measured and basecalled. The basecalls will contain errors, so these will not be exact reverse complements of one another.

Hope this answers your questions, I have a slightly more in-depth explanation on my blog: https://timd.one/blog/genomics/file_formats.php

Answer 2

I'll give a different perspective. Oxford Nanopore FAST5 is NGS within a HDF5 database. A clear description is here.

HDF5 is a format used for rapid I/O of large data sets.

Oxford Nano appear to have written a bit boilerplate around HDF5 and called it something that sounds like 'fasta'.

A cool part of HDF5 (not FAST5) is HDF5 View, which can be downloaded here. Its cool because you can visualise the file structure of HDF5 db via a GUI. I very much suspect if you use HDF5 View on your FAST5 database you will be able to navigate (see not change) the entire file structure.

To answer your question directly h5ls is listing the HDF5 database. To construct a HDF5 db you assign the data under a series of 'keys' which forms the directory structure. However, you perhaps might not need Oxford tools, because Python pandas have direct series of methods to connect, construct and manipulate HDF5.

I raised the HDF5 in a question then answered here on Bioinfo SE here.

Again the reason for HDF5's importance is unlike an SQL connection, data transfer and disconnect, HDF5 shifts data very quickly.

I do question how Oxford Nanopore can lay claim to FAST5 because HDF5 is open source and always has been. If they want to lay claim to some boilerplate, be my guest - I write the stuff all the time ;-)

HDF5 however is a serious project with major development time and appears to still be under active development.