I an confused with the term 'outlier' and 'outgroup' in phylogeny. Could someone please explain with the example phylogenetic tree above which is the correct term? Thank you very much !!!
$\begingroup$ Welcome to the group. Can you explain your question more. What do you need explained about the tree? $\endgroup$– BioathleteNov 30, 2018 at 14:43
$\begingroup$ An "outlier" is a term used in general statistics to infer that a minority of the data set deviates substantially from the majority. An example would be the number of radical substitutions present in a house-keeping gene. An outgroup is described below and singularly refers to phylogeny. $\endgroup$– M__ ♦Dec 28, 2018 at 21:01
Distance- and ML-based algorithms using reversible models can't find the root of trees. A classical method to root a tree is to use an outgroup (not outlier), which is a species/sequence known to directly descend from the root of the tree. In your case, it is relatively easy: add a chicken or fish ortholog to your dataset and put the root on the chicken/fish branch.
Outgroup works well for single-copy genes used in species tree construction (e.g. ADH). However, it doesn't always work. The culprit is the "known" part. Say a gene has two copies A and B in vertebrates. Copy A was lost in rodent and copy B lost in primate. If you choose a chicken A gene as the outgroup, the correct tree should be ((primary-A,chicken-A),rodent-B). Without knowing the true history, you may forcefully put the root at the chicken-A branch and build a wrong tree ((primate-A,rodent-B),chicken-A).
There are a few other tree rooting methods. An easy approach is to put the root at the longest branch in the tree, assuming the presence of molecular clock. When the species tree is known, you can root a gene tree by minimizing the number gene duplication/loss events in the history. I generally prefer the latter approach when the relevant information is available.
$\begingroup$ How to differentiate between outgroup and outlier? Thank you very much :) @user172818 $\endgroup$ Nov 29, 2018 at 1:51
1$\begingroup$ @ZhengKeongNg In phylogenetics, "outlier" is not clearly defined. Avoid that word. $\endgroup$ Nov 29, 2018 at 4:04
you may use the OD-seq algorithm to filter outliers in a multiple alignment. https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-015-0702-1