BERT Language Model and Gene Sequences - How Do I Relate Clusters of Sequences?

Question

I hope you'll indulge a question from a computer scientist with limited bioinformatics knowledge. I've been working with the Google tool for language modeling called BERT. It's generally regarded as state of the art when encoding language. It allows you to say that the sentences "I went to the bank to cash my check." and "My wife is going to the Credit Union to pay the mortgage" are more similar than if compared to "he was fishing and fell off the bank." In short, BERT takes sentences and encodes them so that related concepts are near each other and unrelated concepts are farther apart.

BERT is an amazing tool and I wondered if it could successfully model the language of the genome if it were encoded into "sentences" (I chose 7-mer words and sentences of length 64, arbitrarily). Below is the result of encoding the full genomes of human, bonobo, gorilla, chimp, and orangutan. There are about 270K "sentences" here. You can see there are four major blobs, with indications of small tight clusters. BERT has decided that the points indicated in red are somehow semantically related. I think is is significant, or at the very least interesting.

My first thought was that clustering was trivial degenerate cases. Maybe all the "AAAAAAA..." were put near each other, and all the "TTTTTTTTT...." etc. I printed out samples from the clusters and visually that is not the case. Also I've made histograms of the average Levenshtein distance in groups. Within a tight group, the LD is statistically much lower (but not trivially low) as compared to unclustered regions (the big blob is all unclustered).

If there is a biological basis for the clustering, then new sequences could be similarly transformed, and classified according to cluster membership (and related biological function). This exhausts my biological knowledge. I've taken sequences from clusters and used BLAST to try and get some insight but I'm way out of my comfort zone with that.

My primary question is, given a set of sequences (in my case len=448), how do you figure out what they all have in common (given my hypothesis is that BERT says they are related based on proximity)? And my backup question is, do you think this is interesting or a waste of time?

Answer 1

The UMAP results look very pretty :)

If I were to try to make sense of this blob, I'd try to do the following.

1. Pull out Gene Ontology (GO) terms for a list of candidate genes (you can try to subset Uniprot / Swissprot for some examples). Then I feed through some genes through your fragmented BERT kmer approach to see where they land. I'd be particularly interested in seeing where transmembrane proteins and enzymes land here.
2. I'd try to color this by introns vs exons (you should be able to extract this sort of information from your genome files). There is a good chance that the difference between introns / exons could explain some of these differences.
3. It maybe worth looking into genes that only encode RNAs vs genes that encode proteins.

Answer 2

In truth deep neural networks quite mainstream now, albeit they are a pain because of their reliance on very expensive graphics cards to process the data (GPU) rather than traditional CPU.

There are two issues,

• The algorhithms need an application to a biological phenomena and this is dependent on the model. DNA applications (usually) use a different model to language applications (below)
• The model is only as good what it is trained against. In languages this would be people saying stuff, like in your example. In biology it is a phenotype. Thus you need to feed it the right answer and all variants of the right answer. It will then spot the pattern and recognise this pattern in de novo inputs.

What we are talking is deep learning using a neural network yes we use them, but again the models are (usually) different. Convolution neural networks (CNN) are common in DNA/genomics because it assess the physical linkage of DNA. CNN is also used in image recognition, so like when you type 'dog' into google and see dog images, chances are the search is a consequence of CNN classification.

In languages recurrent neural networks particularly long short term memory (LSTM) models are common because they need to learn the patterns and idioms of a language. Google of course will only use their development called TensorFlow to build their models, which is very geeky, for example they have a programming construct called a "placeholder". In my view a placeholder is something found in Adobe PhotoShop rather than a programming language etc... TensorFlow exponents outside Google tend to use the Keras interface. In Python there is Natural Language Toolkit — NLTK 3.5 to do this.

Summary So, to answer your question, by dumping DNA sequence into a language RNN its fun, but it wouldn't mean very much because the evolution of language, or construction of a given language (i.e. the training set) is a very different data to mechanism(s) involved in the evolution of DNA.

Fundamentally, we don't speak DNA.

I think it would be fun dumping whale songs and dolphin squeaks into these programs to see whether there is any convergence with our own speech.

---

My advice is to use an unsupervised training approach to identify your clusters. This would be model-free and training-free and would help further analysis in "supervised-learning" OR just stick with the clusters it generates.