Annotation format design

Question

Bashing file formats is a favorite pastime in bioinformatics, and annotation file formats such as GFF and BED seem to get special attention. A lot of this frustration stems from community's shockingly inconsistent adherence to specifications and conventions, but there are also some (dare I say objectively) problematic design choices in each of these formats.

• GFF (and its more common derivatives GTF and GFF3) use 1-based closed interval notation, which optimizes for human comprehension but is far inferior to 0-based half-open interval notation (such as used by BED) for computations involving interval arithmetic.

• Although BED and GTF were designed for very specific use cases (visualization and gene prediction, respectively), they have been used and abused in a much wider set of contexts. For example, the BED fields related to the thick part are irrelevant if you're not plotting them in a genome browser.

• BED supports a single level of feature decomposition (a feature can be broken up into blocks). GTF supports two levels (exons grouped by transcript_id, transcripts grouped by gene_id). In contrast, GFF3 supports an arbitrary number of levels, and uses parent/child relationships defined by ID and Parent attributes to declare a directed acyclic graph of features.

• Data that does not fit into mandatory pre-defined fields must be relegated to optional fields or free-form attribute key/value pairs. While this flexibility is powerful, a common complaint is that "all the action" happens in these optional/free-form fields.

• There is a dearth of validation tools, and those that do exist focus primarily on validating syntax and not semantics. To use an aging analogy, it's one thing to say an XML file is valid, but it's completely different to validate it against a schema. There are essentially no widely used tools that do the latter for annotation files.

If we were tasked with creating a new annotation format, and if we were guaranteed the resources needed to develop it, and interest and wide adoption from the wider community (one can dream!), what design criteria should be considered in the development of this new format? What, if anything, makes an objectively good annotation data format?

Answer 1

Presuming we consider "human readable", "easily parsable", and "quickly queryable" to be objectively good qualities (and if not, I worry for the future):

1. Text-based: It's absurdly common to want to use grep or awk on annotations. Sure, one could make variants of these that are binary-format aware, but why reinvent the wheel. Of course text files don't innately allow region-based queries of their contents, so on to point 2...
   • This should further explicitly be line-based. Fields would need to be tab-separated (ideally it'd use the ascii record separator character instead, but I fear that ship has long ago sailed).
2. A strictly defined line order: This is one of my personal pet-peeves about BED/GFF/GTF files. If you're going to make a text-based annotation format, everyone would end up ahead if said format made explicit that it should be sorted. This then allows things like tabix to be used so the "query a region" problem is then solved. But I would go further than that. The issue with things like GFF is that there are multiple inter-dependent lines and there's no strict rule about whether a parent line absolutely must come before a child line. This just makes implementing things a nightmare and more often than not a randomly ordered file will just break tools. Since GTF or GFF-style interdependent lines likely be how any annotation format works, this ordering between lines with the same start position should be made explicit in the format.

Answer 2

The problem is that GFF, fundamentally, is a relational format: it provides tags that relate individual rows via one-to-many relationships (e.g. gene–exon). This indirectly highlights the second complication: individual rows have different types, and therefore store different attributes in the 9th column.

Over the last few decades (!), we have accumulated a wealth of theory and tools to work with this kind of data. And the usual solution is to create a database schema for a relational database, and to use database drivers and database query languages (e.g. SQL, but increasingly also data-relational mappers such as LINQ and dplyr).

Using a text-based format is attractive for many reasons that Devon mentions, but it is fundamentally at odds with a lot of the theory and tools for relational data. This creates an impedance mismatch.

I’m convinced that, in the long run, the solution is going to be to revert to using relational databases for complex annotations. I say “revert” even though these databases already exist, they are simply often ignored in bioinformaticians’ day-to-day work (I never use them). Because this is objectively the best technical solution, and we have the research to back this up.

Answer 3

I quite like BED and GFF3 (I don't like GTF/GFF2, though). As text-based formats, I don't think they leave us much room for improvement. Anyway, if you want a new format, here is one. The following is a hybrid between GFF3 and BED. It is a TAB-delimited text-based format with the following fields:

1. chr, required
2. start (0-based), required
3. end, required
4. strand
5. ID
6. type (see below)

Like BED, only the first 3 columns are required, the rest are optional. What is different starts with the "type" field as in GFF. This "type", when present, defines the columns following it. For example, if type==coding, we could have cdsStart, cdsEnd, blockCount, blockSizes and blockStarts like BED; if type==exon, column 7 could be a "phase", indicating the phase of the first base. This way, "type" makes this format highly extensible while still relatively easy to parse in comparison to using optional tags all the way. In addition, we may have semi-colon-separated "key"="value" pairs at the end of each line as in GFF3. Example:

chr1   10000  50000  -  x1 coding   10100   40800  2  1000,2000   10000,48000
chr1   10000  11000  -  *  exon     *       foo1=bar1;foo2=bar2
chr1   48000  50000  -  *  exon     2
chr2   10000  50000

The above only gives a bare bone of the format. There are subtle questions such as: 1) the uniqueness/scope of ID; 2) whether to have parent ID as a fixed field or a type-specific field; 3) where to put display name; 4) sorting order. These are also important in practice.

PS: I like SQL a lot and think it should get used more often in bioinformatics, but I don't think it replaces text formats completely. Formats are useful for serialization and data transfer. They require less skill to work with and less software/hardware resource to deploy. Carefully engineered binary representations of formats can be much more efficient in a lot of use cases.

Answer 4

Just throwing this out there, as all the best stuff has already been said, but why does one need to specify a strict specification for a bioinformatic data format? As you say, what typically happens is all the action ends up in the optional fields.

There's a lot to be said for the libertarian values of just letting people figure it out on their own. Take optional fields in the BAM tag spec. for example. Here you can have your own tags, but they must start with an "X", a "Y" or a "Z" and be followed by "[A-Za-z0-9]". Why? Why can't people use their own tag names like "read is unique" or "edit distance to genome" or whatever they want. Are we not to be trusted with the power of naming things and recalling things arbitrarily? The result is one must look up the tag name's actual meaning in the publication of the tool that produced it, or ask on certain forums, etc. And this is assuming one knows the tool that produced the reads - if not then who knows wtf 'XT' stands for.

Essentially, downstream readers of optional fields already trust that 'XT' is what it thinks it is. So if we're happy to use optional fields in any capacity, why stop at fields?

Extend this logic to the columns of the data format. Let users determine column names. You may once in a blue moon come across a file with the chromosome column as "chr" rather than "chromosome", but most of the time you'll be looking for "read is quantum-radioactive", and detecting and fixing naming-errors is not going to be as much of a problem as using a dataformat that does not store what it is you want to store. Or worse, it can store it, but in a really illogical way that results in everyone who uses the dataformat to have to ask at least 3 questions here or on other forums before they understand whats really going on.

So you end up with a general solution to the problem of storing tabular information in a compatible way across many different types of software, also known as SQL (or any other kind of general-purpose database such as REDIS, Neo4J, Numpy, etc). In fact, does it even matter what the datastore is, so long as it's tabular and has the "chromosome" and "position" values for each item?

TL;DR - We will not think of tomorrow's best data format today, because the nature of tomorrow's data is not yet known. Less policing in this area would most likely result in more robust software, where nothing is taken for granted and no assumptions about the data can be made until the schema has been parsed.

In truth, the only reason we think we need to specify our dataformats is because we know if we didn't, other people will do something we didn't think of - and we, i think falsely, assume that will be bad.