6
$\begingroup$

How small can a compressed file containing the human genome be? I'm aware that this question cannot actually be answered, since it is asking for the Kolmogorov complexity of the human genome, which is not computable. So reformulating: What is the largest known lower bound that we can put on the size of a file containing the human genome?

An example answer could be: on average we know that the human genome contain at least this amount X of randomness, therefore it is not possible to do better than Y bytes.

I'm agnostic on which human genome, but I would say without polymorphisms to make it simpler.

The compressed sequence would need to be stand alone, without using a reference (I asked a question on that topic here: Compressing the human genome to few megabytes).

There should be a program able to compress and decompress the genome in a lossless way. Apart from that, any kind of modelling and coding is allowed.

$\endgroup$
0

2 Answers 2

9
+250
$\begingroup$

For the most modern CHM13 assembly (v2.0; Complete T2T reconstruction of a human genome; https://github.com/marbl/CHM13), the best known compression without any use of a reference is 567 MB (1.45 bits/bp; 1.80 bits/distinct canonical 31-mer).

Upper estimates of genome compressibility can be made using standard general or specialized compressors (see, e.g., this comparison). In my experience, xz can achieve great compression ratios on genomic sequences, especially when used with the maximum compression level (-9) and a single thread (-T1). The compression performance can be further boosted by using a large dictionary (-e --lzma2=preset=9,dict=512MiB). Importantly, the original FASTA file needs to be cleaned – linebreaks hamper compression as they act as noise; the reformatting can be done by seqtk seq -U (which also converts all nucleotides to the upper case). It the case of the CHM13v2 assembly, xz compresses the human genome to 634 MB.

However, even better performance can be achieved with specialized compressors, especially if they have particular support for reverse complements and mutational models. Among the best ones is GeCo3, which has been shown to compress CHM13v2, with a particular combination of parameters, to 567 MB 3.

Lower estimates are impossible in general (we can't reject the existence of a program that would create the whole sequence from small amount of starting conditions), but can be done for particular compression techniques, e.g., particular variants of dictionary compressors. However, this leads to intractable optimization problems (see e.g. the problem of minimal string attractors).

A useful auxiliary measure for estimating the compressibility of single genomes is the number of distinct k-mers for a "reasonably" selected k (e.g., k=31). The lower bound on the compressibility of k-mer sets can then be roughly approximated by 2 bits per distinct k-mer, however, this is limited by the embedded structural assumptions, e.g., that single genomes are in principle random strings with occasional (possibly distant) repeats. For instance, the CHM13v2 human genome assembly contains 2,512,390,070 distinct canonical 31-mers, which would then correspond to 628 MB assuming 2bits/kmers, while the best available specialized compressors can compress it down to 567 MB, i.e., by approx 10% better.

Updates

  • May 27, 2022 – updated the upper bound based on the latest results with CHM13v2 (xz & GeCo3), changed the lower bound wording to an auxiliary estimate as the assumptions have shown not to hold
$\endgroup$
5
$\begingroup$

There's a benchmark that compared dozens of compressors on DNA data, including human genome: http://kirr.dyndns.org/sequence-compression-benchmark/

In particular, this table shows file size of compressed human genome, produced by the strongest setting of each compressor: http://kirr.dyndns.org/sequence-compression-benchmark/?d=Homo+sapiens+GCA_000001405.28+%283.31+GB%29&cs=1&cg=1&com=yes&src=all&nt=4&only-best=1&bn=1&bm=size&sm=same&tm0=name&tm1=size&button=Show+table

The best result is currently 657 MB on that particular genome build (GCA_000001405.28), when compressed with fastqz.

By the way, xz (mentioned in another answer) is currently number 20 in that table. (Though the genome is un-modified straight from the database, not the "cleaned" form.)

Disclosure: I worked on that benchmark.

Note: What is being compressed here is a FASTA file containing a particular assembly of a human genome. Compared to an actual genome it has this extra content: Sequence names and brakes between contigs, end of line characters, N characters representing unknown bases, and soft-masked repeats (represented as lowercase). On the other hand, it also misses some information from the actual genome: 1. It's an unphased haploid sequence, while the real genome is diploid (two copies of each chromosome). 2. It includes no epigenetic information (DNA methylation, histone modifications), nucleosome positioning, 3D structure of packed DNA, etc. So, it's just a digital model of a genome with a certain degree of accuracy, that is useful, simple, and relatively affordable to obtain. It's a format commonly used for representing genomes in databases.

$\endgroup$
7
  • 1
    $\begingroup$ I quickly tested xz on this build (GCA_000001405.28) in the same way as in my reply (seqtk seq -U; xz -9 -T1) and got 667 MB, which is very close to fastqz-slow in the referenced table (657 MB). With general compressors, it's important to remove eols in the sequences, as well as to convert all bases to the upper case. Note that none of these steps modify the genome in any way – fastas can always be converted to a fixed width, and the upper/lower case info are usually just masked repeats. $\endgroup$ Jun 12, 2021 at 17:19
  • 1
    $\begingroup$ As the table above says 770 MB (i.e., +103 MB) for xz with similar parameters (-e9), I assume the file probably wasn't in a form that would facilitate compression using general compressors, and the observed gain of specialized compressors came mainly from removing the eols/upper case conversion. $\endgroup$ Jun 12, 2021 at 17:21
  • 1
    $\begingroup$ An additional experiment: I verified if xz could outcompete even fastqz-slow, and it indeed can. GCA_000001405.28 can be compressed down to 652 MB by 'xz -T1 -e --lzma2=preset=9,dict=512MiB' (again, after 'seqtk seq -U'). $\endgroup$ Jun 12, 2021 at 18:25
  • 1
    $\begingroup$ @Kamil S Jaron, cleaning is one reason, another is that it's just different genome build. Though as you see Karel obtain similar number with his method. $\endgroup$
    – Kirr
    Jun 15, 2021 at 20:03
  • 1
    $\begingroup$ @Karel Břinda, by the way, it's good idea to tweak xz's dict (dict=512MiB), I should benchmark this setting too. Thanks. $\endgroup$
    – Kirr
    Jun 15, 2021 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.