[I'm not an expert in this field, this is only my "best try". Maybe you could get better mathematically motivated answers on ai.stackexchange]
First note that, as pointed in the DNA BERT paper, k-mers are a very common approach to working with sequences. This is true both for older approaches as well as newer Transformer-type models. For example the recent Nucleotide Transformer, DNAGPT or MuLan-Methyl also use k-mers.
While I'm not aware of a published systematic comparison between k-mers and single bases, I'm sure I've seen papers that vary k
as a hyperparameter (so would notice if k=1
gives best results), and I have to believe someone somewhere at some point has thought about not using k-mers.
There are some Deep Learning papers that use individual one hot-encoded nucleotides, for example SpliceAI, but they typically only look at relatively short, fixed-length, sequences (relatively: this paper does go up to 10k).
As to the "why", my (limited) understanding is that there are two factors. If you think of NLP, a token is typically a word from the language. Taking each nucleotide as a token means you consider a language with only 4 words, that is a very limited amount of information. It does make sense to use more meaningful tokens: to respond to your comment, even ViT uses patches (and does not attempt to work on individual pixels). Based on our knowledge of biology, we can expect that most sequences have meaningful "motifs" that are made of a few consecutive nucleotides. If you feed the nucleotides to the Transformer, it has to use attention to detect these motifs as well as their long-range interaction. If instead you use a "patch" (or k-mer) that captures entire motifs, this can be learned by an initial linear projection and you only use the attention for long-range interactions.
Finally, the other reason is computing power, as hinted to in the DNAGPT paper:
In the non-overlapped k-mers strategy, the shift is equal to K, resulting in N/k tokens for an N-length sequence and improving the efficiency by k times.
Maybe taking 1-mers could work just as well as 6-mers, but the number of relationships to consider increases even more, a non-overlapping k-mer strategy does decrease the of interactions to consider, and an overlapping k-mer strategy where each patch is preprocessed (e.g. with a linear model) can also increase efficiency (so to speak the model does not need to consider short-range interactions which are already accounted for).