The answer really is that a root is determined using external evidence, rather than from the internal branches.
An algorithm such as root digger can be used to estimate the root placement based on a branch that for example might minimise the anomalous branch lengths in all other taxa. Excessive comparative branch-lengths could indicate an error in rooting the tree and this can be the basis of the estimate. However, it is an estimate.
R libraries such as
ape are more likely to simply mid-point root the tree, which isn't really cool, but its easy to calculate.
The three approaches for rooting are either:
- Using a known outgroup to the taxa of interest, which unequivocally predates their divergence.
- External information such the evolution of key morphological features of the organism.
- Published studies.
Point 1 This can be used singly to find the root - a "root-finding tree" - and the tree of interest presented by constructing a second tree without this root in the alignment. For example the Archaea predate bacteria, one tree determines where the Archaea attach to taxa of interest. If the outgroup is very distinct this could cause artefacts.
Point 2 Finding an outgroup much close to the taxa of interest using external information, for example gram-negative versus gram-positive bacteria, e.g. if all the bacteria in your analysis were one or the other and again do the two tree strategy (point 1).
Point 3 Using large kingdom wide phylogenies from published studies to determine where the most ancestral branch in your taxa.
This is not an exhaustive list about how to use external information to determine the location of the root, but it is very widely recognised.
Directional versus reversible matrices
Directional matrix (e.g. most options in Beast) need to know where the root is, or they will 'guess it': ideally and this information can be feed into the algorithm when it is started. This is important if you are doing molecular dating.
In fact, you can do this even with reversible matrices (e.g. RAxML) but it makes no difference to the answer.
If you have identified the root and the phylogeny you have does not represent that root correctly you can "re-root" the tree using the outgroup identified from point 1 to 3. A program such FigTree are cool for re-rooting. Re-rooting is perfectly cool providing you present the methodology that identifies the root taxa or taxon (singular).