# Using average of sequence similarities when delimiting genera - how to deal with outliers?

Sometimes I am trying to see where a (bacterial) genus ends and where another one begins based on 16S rRNA phylogenies. Of course, the length and support of the branch matter a lot, but this question in particular is about sequence similarity.

As far as I understand it, in general, sequences within the same genus are expected to have roughly >94-95% sequence similarity at the 16S rRNA level. But that leaves out a lot of variation. For example, in doodle A below, there may be <90% similarity between sequence 'a' and sequence 'e' ; however, there is always >95% similarity between, say, a and b, b and c, c and d, d and e...So all would belong to the same genus. Previously I had considered just comparing every sequence with every other sequence in my group of interest, get an average and a standard deviation, and see if the average > 94-95%. However, it occured to me that if I have a tree shaped like that in doodle B (a couple of closely-related sequences from one genus and one very distantly-related sequence from another genus), I could get the very same average (and only slight changes in the standard deviation).

It feels like there should be a simple mathematical property to distinguish between A and B based on similarity values, and/or point out outliers like sequence 'a' in B, but I do not know what that would be. Help would be appreciated!

Mock dataset A:

sequence1 sequence2 sequence similarity (%)
a b 99
a c 93
a d 92
a e 95
b c 99
b d 98
b e 97
c d 95
c e 94
d e 88

Mock dataset B:

sequence1 sequence2 sequence similarity (%)
a b 90
a c 90
a d 90
a e 90
b c 99
b d 99
b e 99
c d 99
c e 97
d e 97
• Wouldn't the average for all comparisons to "a" in dataset B be below 95? That would indicate that a is not the same genus as the others, following your definition of >94-95 similarity. Sep 21, 2023 at 17:24
• @Niklas Thank you. My issue is that this is simple enough for one sequence, but would get more difficult to see from the table if I add more (one hundred? five hundred? a thousand?). Of course, I can still spot the outlier easily enough in the tree...most times...but I feel that adds a degree of subjectivity that I am not quite content with. I would prefer something more rigorous. Sep 22, 2023 at 8:43
• @M__ I am not quite sure how to answer. I guess either? Basically, for me A and B correspond to a group of sequences (a-e), that could be seen as taxa, and can be easily represented as trees. So more or less interchangeable. I can visually spot outliers in the trees, but I hoped for something more objective, and working with the taxa at the level of sequence similarity felt like an option. Maybe it's not the only option? Sep 22, 2023 at 8:56
• @Laura I am not really sure if that is so complicated, basically you would just need to compute the average % identity across all comparisons that contain one of your taxa and see if that is below 95%? You could also plot it to make it more visual. It depends a lot on what you expect (is this a frequent or rare thing, will most species always be from one single genus?).. I think to "look at the tree" using some metric, total branch lengths (sum of all branches) could be informative for your problem, have you tried using that? Sep 22, 2023 at 11:04

## 1 Answer

The basic question is where do genera start and stop within phylogenetics and the 16S 5% rule? This is not a trivial question within molecular taxonomy, especially for bacteria.

1. The key is to define the tree structure for 16S
2. Define the tree structure for other genes

Background and tree stats The difference between Tree A and Tree B is referred to as 'balance'. Tree A is unbalanced, whilst Tree B is balanced and that is a formal description. I would personally use the term Tree A was "heavily paraphyletic", or "extensive paraphyly" - at a technical level that is accepted, I have used frequently. Equally I'd describe Tree B as a "singular monophyly". The statistical basis for tree balance is called the Colless value.

"Colless statistic". It's fairly easy to calculate. The reason it is not often used is because the tree needs to be phylogenetically robust (below). Not much importance is given to it because if a branch isn't robust thats a headache for this statistic (it can't tell the difference between robust and non-robust).

Difference between branches The statistical difference between taxa A and taxa B is measured by bootstrapping (I'm sure you know this), where >= 80% is bifurcation is robust.

Colless statistic might be used, but its fairly obvious, bootstrapping is needed and iqtree is best package for this operation.

The second more difficult part of the question is the delineation of a genus. The 5% threshold of 16S has always been seen as a guide. I have never seen it rigidly enforced and it creates a conflict with phylogenetic theory as your example points out. Telling what is and ain't a genus ... you are right that a key is the tree structure.

1. Tree structure and bootstrapping ...
• A declared monophyly must be supported by a bootstrap > 80%
• A declared paraphyly ... each paraphyletic branch must be supported by a bootstrap > 80%
1. 16S versus other genes and tree structure The tree structure needs confirming particularly as the bootstrap values 16S will often be low. The starting point is to consider the traditional MLST loci (I forget what they are but stuff like atpD, etc ...), build trees from them and :
• Don't combine 16S with protein genes (they evolve at different rates), although it could be done in a partitioned model
• MLST protein genes and core-house keeping genes can be combined without partitioning the data (there's a test, but it'll basically work).

Take the MLST genes repeat the tree and assess the bootstraps. Basically it will produce the same tree, but: a) confirms the tree structure; b) can provide stronger bootstrap values (which confirm the tree).

5% stuff All the above needs doing before any 5% stuff is considered. It is my personal view that using 5% 16S as a mean value is sort of bending the numbers, particularly in a paraphyly where the divergence is e.g. between 2 to 10%.

Monophyly vs paraphyly This where the tree structure is needed. Basically, a robust monophyly would be regarded as a taxonomic singular unit. In this case if its close to the 5% threshold genus rule:

• even if the pairwise distance between taxa within the monophyly is say 8% via 16S;
• and this is supported by other protein loci;
• and there's no suspicion this could represent a family.

In the above criteria I'd say its evidence of a genus. 10% is a bit big distance however.

If the tree structure is paraphyletic and the maximum different is say 10% between two taxa - one at the top of the paraphyly and the other as a most recent child ... then thats tricky. It could easily be two genera within the paraphyletic tree structure. For example, the vertebrates as paraphyletic, viz. amphibian, reptile, bird, edentates, marsupial, mammals and that represents 6 monophyletic Classes. Thus if that was my result I would leave open the possible interpretations: maybe this is a single genus; alternatively it could be two (or more) genera. The key to this is defining the nodes (internal branches) via bootstraps to be confident of the tree structure.

Generally a single robust paraphyletic tree structure, i.e. heavily imbalanced, where each paraphyletic branch is robustly supported would not be regarded as a single genus especially when 16S genetic differences is >> 5%. Taxonomically, this scenario would need further phylogenetic analysis (separate question) and classical bacterial taxonomists would want phenotypes before splitting a genus.

Reviewing the question against the answer - what I would do for the monophyly (diversity 10%) is split off the basal groups (at 10%) view robust bootstrap support. Then forward the in-group as the genus and the full monophyly as a putative genus. So that 10% is not the divergence unequivocally defining a single genus. So thats quite easy on the picture drawn. Thus the in-group is <8% because 10% is a bit high.

It is harder to do this with a paraphyletic tree structure and thats a separate question on the advanced techniques needed to investigate this.

• I would like to note out that the terms paraphyletic and monophyletic do not refer to trees, but to clades. Sep 25, 2023 at 13:56
• Hello, all terms of the "-phyletic" group inherently describe clades, a clade can either be mono-, para-, or polyphyletic. Since a tree has no inherent clades, these terms only make sense in combination when adding clades (e.g. taxonomy) to the tree. Clades are not necessarily monophyletic (which is why they can also be para- or polyphyletic). Sep 25, 2023 at 14:34
• Clades versus monophylys. I'm not getting into that discussion because of its acrimonious history. I have used "paraphyletic clade" and singularly used "clade" as monophyly as a published definition (a paper reviewed by Science journal). The semantics are not rigid and phylogenetics is more concerned with statistical robustness than "one person says this, another says that".
– M__
Sep 25, 2023 at 14:38
• I am only raising this because you refer to the trees as "heavily paraphyletic", or "extensive paraphyly"  Sep 25, 2023 at 15:49
• Thanks, personal semantics as mentioned ("imbalance" better)
– M__
Sep 25, 2023 at 16:00