How to compute the Shannon entropy for a strand of DNA?

Question

I'm confused by the computation of sequence logo. Wikipedia gives a process about this without a concrete example.

Let's just consider DNA, so there are 4 bases (nucleic acids).

The following data comes from the book "Machine Learning - A Probabilistic Perspective (Figure 1)"

The Shannon entropy of position $i$ is:

${\displaystyle H_{i}=-\sum _{b=a}^{t}f_{b,i}\times \log _{2}f_{b,i}}$

Where ${\displaystyle f_{b,i}}$ is the relative frequency of base

This post is computing position 3, where it seems that the relative frequency of a is 100%, and the relative frequency of 3 other bases are 0%.

Now, how do you calculate the Shannon entropy of position 3? It seems to be 0, which is apparently incorrect, where am I wrong?

Answer 1

Why do you think the entropy of 0 is incorrect? It intuitively makes sense, as there is no uncertainty about the base at position 3, and thus there is no entropy.

However, what is plotted in a sequence logo isn't the entropy, but rather a measure of the "decrease in uncertainty" as the sequence is aligned. This is calculated by taking the entropy at this position if we randomly aligned sequences ($H_g(L)$), and subtracting from it the entropy of the alignment ($H_g(s)$): $$ R_{sequence}=H_g(L) - H_s(L) $$

The entropy at position 3 based on a random alignment is calculated by assuming there are 4 equally likely events (one for each base), and thus: $$ \begin{align*} H_g(L) & = -((1/4 \times -2) + (1/4 \times -2) + (1/4 \times -2) + (1/4 \times -2)) \\ H_g(L) & = 2 \end{align*} $$ Notably, $H_g(L)$ will always be 2 when dealing with nucleotide sequences.

So in your example, we have: $$ \begin{align*} R_{sequence}&=H_g(L) - H_s(L) \\ R_{sequence}&=2 - 0 \\ R_{sequence}&=2 \text{ bits of entropy} \end{align*} $$

Finally, to work out the height of each base at each position in the logo, we multiply the frequency of that base by the overall information gain $R_{sequence}$. In this case the frequency of the base A at position 3 is of course 1: $$ \begin {align*} H(b,l) &=f(b, l) \times R_{sequence} \\ H(A, 3) &= f(a, 3) \times 2 \\ &= 1 \times 2 \\ &= 2 \end{align*} $$

Source: Sequence logos: a new way to display consensus sequences. (Thomas D. Schneider, R. Michael Stephens, 1990)

Note: I've excluded the correction factor $e(n)$ from the overall information-gain calculation for the sake of simplicity. Refer to the paper above for an explanation of this.