Theoretical limit of human genome compression

Question

How small can a compressed file containing the human genome be? I'm aware that this question cannot actually be answered, since it is asking for the Kolmogorov complexity of the human genome, which is not computable. So reformulating: What is the largest known lower bound that we can put on the size of a file containing the human genome?

An example answer could be: on average we know that the human genome contain at least this amount X of randomness, therefore it is not possible to do better than Y bytes.

I'm agnostic on which human genome, but I would say without polymorphisms to make it simpler.

The compressed sequence would need to be stand alone, without using a reference (I asked a question on that topic here: Compressing the human genome to few megabytes).

There should be a program able to compress and decompress the genome in a lossless way. Apart from that, any kind of modelling and coding is allowed.

Answer 1

For the most modern CHM13 assembly (v2.0; Complete T2T reconstruction of a human genome; https://github.com/marbl/CHM13), the best known compression without any use of a reference is 567 MB (1.45 bits/bp; 1.80 bits/distinct canonical 31-mer).

Upper estimates of genome compressibility can be made using standard general or specialized compressors (see, e.g., this comparison). In my experience, xz can achieve great compression ratios on genomic sequences, especially when used with the maximum compression level (-9) and a single thread (-T1). The compression performance can be further boosted by using a large dictionary (-e --lzma2=preset=9,dict=512MiB). Importantly, the original FASTA file needs to be cleaned – linebreaks hamper compression as they act as noise; the reformatting can be done by seqtk seq -U (which also converts all nucleotides to the upper case). It the case of the CHM13v2 assembly, xz compresses the human genome to 634 MB.

However, even better performance can be achieved with specialized compressors, especially if they have particular support for reverse complements and mutational models. Among the best ones is GeCo3, which has been shown to compress CHM13v2, with a particular combination of parameters, to 567 MB 3.

Lower estimates are impossible in general (we can't reject the existence of a program that would create the whole sequence from small amount of starting conditions), but can be done for particular compression techniques, e.g., particular variants of dictionary compressors. However, this leads to intractable optimization problems (see e.g. the problem of minimal string attractors).

A useful auxiliary measure for estimating the compressibility of single genomes is the number of distinct k-mers for a "reasonably" selected k (e.g., k=31). The lower bound on the compressibility of k-mer sets can then be roughly approximated by 2 bits per distinct k-mer, however, this is limited by the embedded structural assumptions, e.g., that single genomes are in principle random strings with occasional (possibly distant) repeats. For instance, the CHM13v2 human genome assembly contains 2,512,390,070 distinct canonical 31-mers, which would then correspond to 628 MB assuming 2bits/kmers, while the best available specialized compressors can compress it down to 567 MB, i.e., by approx 10% better.

Updates

• May 27, 2022 – updated the upper bound based on the latest results with CHM13v2 (xz & GeCo3), changed the lower bound wording to an auxiliary estimate as the assumptions have shown not to hold

Answer 2

There's a benchmark that compared dozens of compressors on DNA data, including human genome: http://kirr.dyndns.org/sequence-compression-benchmark/

In particular, this table shows file size of compressed human genome, produced by the strongest setting of each compressor: http://kirr.dyndns.org/sequence-compression-benchmark/?d=Homo+sapiens+GCA_000001405.28+%283.31+GB%29&cs=1&cg=1&com=yes&src=all&nt=4&only-best=1&bn=1&bm=size&sm=same&tm0=name&tm1=size&button=Show+table

The best result is currently 657 MB on that particular genome build (GCA_000001405.28), when compressed with fastqz.

By the way, xz (mentioned in another answer) is currently number 20 in that table. (Though the genome is un-modified straight from the database, not the "cleaned" form.)

Disclosure: I worked on that benchmark.

Note: What is being compressed here is a FASTA file containing a particular assembly of a human genome. Compared to an actual genome it has this extra content: Sequence names and brakes between contigs, end of line characters, N characters representing unknown bases, and soft-masked repeats (represented as lowercase). On the other hand, it also misses some information from the actual genome: 1. It's an unphased haploid sequence, while the real genome is diploid (two copies of each chromosome). 2. It includes no epigenetic information (DNA methylation, histone modifications), nucleosome positioning, 3D structure of packed DNA, etc. So, it's just a digital model of a genome with a certain degree of accuracy, that is useful, simple, and relatively affordable to obtain. It's a format commonly used for representing genomes in databases.