Why do SAM and BAM use different coordinate systems?

Question

BAM files are, at least as far as I know, simply binary compressed versions of SAM files. They have the exact same information and are used in the same way. Why then does the SAM format use a 1-based coordinate system while BAM uses a 0-based one?

This is the relevant section of the SAM format specification (emphasis mine):

1-based coordinate system A coordinate system where the first base of a sequence is one. In this coordinate system, a region is specified by a closed interval. For example, the region between the 3rd and the 7th bases inclusive is [3, 7]. The SAM, VCF, GFF and Wiggle formats are using the 1-based coordinate system.

0-based coordinate system A coordinate system where the first base of a sequence is zero. In this coordinate system, a region is specified by a half-closed-half-open interval. For example, the region between the 3rd and the 7th bases inclusive is [2, 7). The BAM, BCFv2, BED, and PSL formats are using the 0-based coordinate system.

The two coordinate systems are a very common source of error in bioinformatics, so there must be some reason why these two largely equivalent formats use different systems. For that matter, why do the VCF and BCF formats, whose relationship is equivalent to that between SAM and BAM, also do this? I am guessing there is a good reason and it is equally applicable to both pairs of sister formats.

Note that the samtools manual states that "all position coordinates are 1-based" when giving regions, so it must handle the conversion internally. Indeed, I can confirm that this command returns identical md5sums for both inputs:

md5sum <(samtools view -b sample.bam chr1:36931633-36931871 | samtools view) \ 
       <(samtools view sample.bam chr1:36931633-36931871) 

Answer 1

Arithmetic on a zero-based coordinate system is less complicated than that on a one-based system, so it appears zero-based is often (not exclusively) used for binary data formats, like BAM or bigBed, or text formats like BED, where computers are used to more efficiently calculate lengths of or set operations on intervals.

A more complete answer on the benefits of zero-based accounting is available here, which references the original note (EWD831) on this subject from Dijkstra.

When reading other textual data formats like SAM or VCF, a one-based index is perhaps more "natural" for humans used to counting digits on their hands. If you visually inspect an annotation on the closed interval [3,7] on a line that denotes some abstract chromosome and starts at position 1, it is clear to you where those data points are positioned. Colloquially, you understand that the interval starts on the third position and ends on the seventh. Or, perhaps, people would typically label the initial item in this set the first item, and not the zeroth item.

A one-based index therefore seems to get used often with formats used for visual inspection or presentation of genomic intervals or sequences, such as in genome browsers — or via CLI tools like samtools, when exporting reads to human-readable SAM.

The language used for working with and organizing data can determine choice of indexing, as well. For programs written in C and offshoot languages, including an API like htslib, binary data is typically worked with as chunks of memory accessed via pointers or arrays, and it is convention to iterate over that data in increments that use a zero-based index. Contrariwise, R's roots in FORTRAN and focus on linear algebraic operations (and matrix indexing) may perhaps explain why that language uses one-based indices for its lists, matrices, and vectors.

The designer of the BAM and SAM formats, Heng Li, comments below(*):

I think 1-based is easier for biologists and mathematicians while 0-based is easier for C/C++/Java programmers. SAM/VCF is for biologists to read. BAM/BCF is for a small number of programmers. I wrote the initial specs of sam, bam and bcf2. PS: when sam and vcf are stored in memory by htslib, everything is 0-based in accordance with C conventions. bam and bcf are largely direct dumps of their in-memory representations.

Note: Original comments were unfortunately deleted from answer by someone, but the quotes were taken verbatim at the time they were written.