How to compute the Shannon entropy for a strand of DNA?

I'm confused by the computation of sequence logo. Wikipedia gives a process about this without a concrete example.

Let's just consider DNA, so there are 4 bases (nucleic acids).

The following data comes from the book "Machine Learning - A Probabilistic Perspective (Figure 1)" The Shannon entropy of position $$i$$ is:

$$H_{i}=-\sum _{b=a}^{t}f_{b,i}\times \log _{2}f_{b,i}}$$

Where $$f_{b,i}}$$ is the relative frequency of base

This post is computing position 3, where it seems that the relative frequency of a is 100%, and the relative frequency of 3 other bases are 0%.

Now, how do you calculate the Shannon entropy of position 3? It seems to be 0, which is apparently incorrect, where am I wrong?

• Note that the frequency is always in the interval [0,1] instead of [0,100]. H is 0 for columns with only one symbol. – Peter Menzel Jul 30 at 6:48

Why do you think the entropy of 0 is incorrect? It intuitively makes sense, as there is no uncertainty about the base at position 3, and thus there is no entropy.

However, what is plotted in a sequence logo isn't the entropy, but rather a measure of the "decrease in uncertainty" as the sequence is aligned. This is calculated by taking the entropy at this position if we randomly aligned sequences ($$H_g(L)$$), and subtracting from it the entropy of the alignment ($$H_g(s)$$): $$R_{sequence}=H_g(L) - H_s(L)$$

The entropy at position 3 based on a random alignment is calculated by assuming there are 4 equally likely events (one for each base), and thus: \begin{align*} H_g(L) & = -((1/4 \times -2) + (1/4 \times -2) + (1/4 \times -2) + (1/4 \times -2)) \\ H_g(L) & = 2 \end{align*} Notably, $$H_g(L)$$ will always be 2 when dealing with nucleotide sequences.

So in your example, we have: \begin{align*} R_{sequence}&=H_g(L) - H_s(L) \\ R_{sequence}&=2 - 0 \\ R_{sequence}&=2 \text{ bits of entropy} \end{align*}

Finally, to work out the height of each base at each position in the logo, we multiply the frequency of that base by the overall information gain $$R_{sequence}$$. In this case the frequency of the base A at position 3 is of course 1: \begin {align*} H(b,l) &=f(b, l) \times R_{sequence} \\ H(A, 3) &= f(a, 3) \times 2 \\ &= 1 \times 2 \\ &= 2 \end{align*}

Note: I've excluded the correction factor $$e(n)$$ from the overall information-gain calculation for the sake of simplicity. Refer to the paper above for an explanation of this.

• Thanks for your answer. What does "g" mean/represent/come from in $H_g(L)$ – czlsws Jul 30 at 22:33
• I'm not sure. My guess is g=genome and s=sequence(ing) but that's pure conjecture since the paper doesn't spell it out – Migwell Jul 31 at 3:00
• Thanks for your reply. And does L mean/represent/come from location since the paper says "at various positions (L)". – czlsws Jul 31 at 3:47
• Yes, that would be my assumption. This is important to include in the equation because it's only calculating the information at a single position. The authors propose another equation for an entire sequence (which is just summing these per-position scores) – Migwell Jul 31 at 3:57
• Thanks a lot. And last question, does R come from/represent/mean acids? – czlsws Jul 31 at 13:35