Phylogenetics: Theory and Overview

Speciation Genomics · PB 495/595 Plant Evolutionary Biology

George P. Tiley

5 October 2026

Phylogenetics

Summary

  • We’ve extensively discussed population structure, but now we want to capture the history of shared ancestry among populations and species.
  • A phylogenetic tree is a hypothesis of shared ancestry, estimated from characters (usually DNA) under an explicit model.
  • Today: the likelihood of a tree from first principles, models of sequence evolution, model selection and ML.
  • Thursday: the multispecies coalescent, species-tree estimation, coalescent species delimitation, and introgression.

Learning objectives

By the end you should be able to:

  • Read a phylogeny (tips, nodes, branch lengths, rooting, clades) and define homology.
  • Distinguish probability from likelihood, and explain why programs report \(\log L\).
  • Compute the likelihood of one site on a small tree.
  • Explain rate variation among sites (partitions, \(+\Gamma\)) and choose a model with AIC/BIC.

Why molecular phylogenies?

Species tree of eight baobab species and three outgroups. Tips are marked by flower type: long yellow or red hawkmoth-pollinated flowers in A. rubrostipa, A. perrieri, A. za and A. madagascariensis; short white mammal-pollinated flowers in A. grandidieri, A. suarezensis and A. digitata; white hawkmoth-pollinated flowers in A. gregorii. Three short internal branches, 0.08 and 0.14 coalescent units, are highlighted in red.

Baobabs: ASTRAL tree from 357 nuclear gene trees; data and traits from Karimi et al. (2020).

Tree thinking

Anatomy of a phylogeny

  • Tips (leaves) = sampled taxa; internal nodes = inferred common ancestors.
  • Branch lengths = expected substitutions/site (or, once dated, time).
  • Clade = an ancestor + all its descendants (monophyletic).
  • Homology: similarity by common descent — the signal; homoplasy (convergence/reversal) is the noise.

An unrooted four-taxon tree with tips A, B, C and D, internal nodes s and t, and labels pointing to an internal node (vertex), a branch (edge), and a leaf (terminal node).

Rooting gives time a direction

Left, the unrooted four-taxon tree with a root r placed on the internal branch between s and t. Right, the corresponding rooted tree with r at the top, s ancestral to A and B, and t ancestral to C and D.

Placing a root r on the internal branch gives one rooted tree: \(((A,B),(C,D))\).

Same unrooted tree, different root

Left, the same unrooted tree with the root r placed on the branch leading to A. Right, the rooted tree it implies: A is sister to everything else, then B, then C and D.

Root on the branch to A instead and the same unrooted tree reads \((A,(B,(C,D)))\). An unrooted four-taxon tree has five branches, so five possible rootings — the outgroup picks one.

The inference pipeline

From sequences to a supported tree — model choices enter at every step:

  1. Homology / alignment — orthology assessment, multiple sequence alignment.
  2. Model — a substitution model (partitions, rate variation).
  3. Search / inference — optimize or sample trees (ML, Bayesian).
  4. Support — bootstrap or posterior probabilities.

Distance & parsimony

  • Distance methods (e.g. neighbor-joining): reduce sequences to pairwise distances, build a tree by clustering. Fast, but limitations beyond populations or closely related species.

  • Parsimony: choose the tree minimizing the number of changes. Intuitive, but can fail under various scenarios such as high rates of evolution (“long-branch attraction”).

  • Parsimony built the first broad molecular trees of plants: 475 and 499 seed-plant sequences of the plastid gene rbcL (Chase et al. 1993).

Neighbor-joining by hand: where distances come from

Toy alignment: 5 taxa, 17 sites.

A  GAACCGGTTACGGGTCT
B  GAACCCAGTAGGGGGCT
C  AGCGCGGGTCCGAGGCG
D  GACGAGGGGCCAGGGGT
E  GACGCGGGGCCAGTGCT

The distance between two sequences is simply how many sites differ:

A  GAACCGGTTACGGGTCT
B  GAACCCAGTAGGGGGCT
        ***  *   *     <- 5 sites differ

So \(d(A,B)=5\). In real data we divide by the alignment length and correct for sites that changed more than once (the Jukes–Cantor correction, later in this deck). In this toy every change happened once, so a plain count is enough.

The distance matrix

Counting differences for all 10 pairs gives one table:

A B C D E Total
A – 5 9 9 8 31
B 5 – 10 10 9 34
C 9 10 – 8 7 34
D 9 10 8 – 3 30
E 8 9 7 3 – 27

Total = add up a row: how far that taxon is from all the others.

Goal: a tree whose branch lengths, added along the path between two tips, give back these distances.

Plan: find two taxa that are neighbors (they sit next to each other on the tree), merge them into one new node, shrink the table, and repeat.

Which pair are neighbors?

The closest pair is not always the right one. Two taxa can look close just because everyone is close to everyone, or look far apart just because each sits on a long branch. So NJ adjusts each distance for how far the two taxa are from the rest:

\[ \text{score}(i,j) = (n-2)\times d(i,j) - \text{total}(i) - \text{total}(j) \]

Here \(n\) is the number of taxa still in the table (5 now, so \(n-2=3\)). The lowest score wins: a small distance helps, and so does being far from everyone else.

Pair Distance Totals Score
D–E 3 30, 27 \(3\times3-30-27=-48\)
A–B 5 31, 34 \(3\times5-31-34=\mathbf{-50}\)

D–E is the closest pair, but A and B are each much farther from C, D and E than from each other, so A–B scores lower.

Step 1: score every pair

Distances

A B C D E
A – 5 9 9 8
B 5 – 10 10 9
C 9 10 – 8 7
D 9 10 8 – 3
E 8 9 7 3 –

Scores (\(3d-\text{totals}\))

A B C D E
A – −50 −38 −34 −34
B −50 – −38 −34 −34
C −38 −38 – −40 −40
D −34 −34 −40 – −48
E −34 −34 −40 −48 –

Totals: A 31, B 34, C 34, D 30, E 27. For example, A–C: \(3\times9-31-34=-38\).

The lowest score is A–B (\(-50\)): join A and B into a new node, u.

Step 1: how long are the branches to A and B?

A and B are 5 apart, so (A’s branch) + (B’s branch) = 5. How do we split it?

Every path from A or B to any other taxon goes through \(u\). So if B is farther from the others than A is, the extra distance must be on B’s own branch.

  • Totals: A = 31, B = 34. B is 3 farther in total.
  • Both totals include the A–B distance, so the 3 extra comes from the other 3 taxa: \(3 \div 3 = 1\) extra per taxon.
  • So B’s branch is 1 longer than A’s, and the two add to 5:

\[ \text{A's branch} = 2, \qquad \text{B's branch} = 3 \]

In one line: \(\text{A's branch} = \tfrac12\left[d(A,B) + \dfrac{\text{total}(A)-\text{total}(B)}{n-2}\right] = \tfrac12\left[5 + \dfrac{31-34}{3}\right] = 2\)

Step 1: distances from the new node

How far is \(u\) from C? Both A and B give an answer: take their distance to C and remove their own branch.

  • From A: \(d(A,C) - 2 = 9 - 2 = 7\)
  • From B: \(d(B,C) - 3 = 10 - 3 = 7\)

The two agree (they always do when distances come from a tree). The same for D and E:

via A via B \(u\) to it
C 9 − 2 = 7 10 − 3 = 7 7
D 9 − 2 = 7 10 − 3 = 7 7
E 8 − 2 = 6 9 − 3 = 6 6

A and B are now replaced by \(u\) (distances among C, D, E don’t change):

u C D E Total
u – 7 7 6 20
C 7 – 8 7 22
D 7 8 – 3 18
E 6 7 3 – 16

Step 2: score every pair again

Distances

u C D E
u – 7 7 6
C 7 – 8 7
D 7 8 – 3
E 6 7 3 –

Scores (\(2d-\text{totals}\))

u C D E
u – −28 −24 −24
C −28 – −24 −24
D −24 −24 – −28
E −24 −24 −28 –

Now \(n=4\), so we multiply by \(n-2=2\). Totals: u 20, C 22, D 18, E 16. For example, D–E: \(2\times3-18-16=-28\).

A tie: u–C and D–E both score \(-28\). With four nodes they describe the same unrooted tree, so either is fine. We join D and E into a new node, w.

Step 2: join D and E

Branch lengths. D and E are 3 apart. Their totals are 18 and 16, a difference of 2 spread over the other 2 nodes: \(2\div2=1\). So D’s branch is 1 longer than E’s, and they add to 3:

\[\text{D's branch} = 2,\qquad \text{E's branch} = 1\]

Distances from \(w\). Take a distance and remove the branch:

  • To u: \(d(D,u)-2 = 7-2 = 5\), and \(d(E,u)-1 = 6-1 = 5\) ✓
  • To C: \(d(D,C)-2 = 8-2 = 6\), and \(d(E,C)-1 = 7-1 = 6\) ✓

New table (3 nodes):

u C w
u – 7 5
C 7 – 6
w 5 6 –

Step 3: the last three nodes

Three nodes can only be joined one way: each hangs off a single central point.

\[ (\text{u's branch}) + (\text{C's branch}) = 7,\quad (\text{u's branch}) + (\text{w's branch}) = 5,\quad (\text{C's branch}) + (\text{w's branch}) = 6 \]

To get C’s branch: add C’s two distances (7 and 6) and subtract the u–w distance (5). That counts C’s branch twice and cancels u’s and w’s, so halve it:

\[ \text{C's branch} = \tfrac12(7+6-5) = 4,\qquad \text{u's branch} = \tfrac12(7+5-6) = 3,\qquad \text{w's branch} = \tfrac12(5+6-7) = 2 \]

Every branch now has a length.

Neighbor-joining: the result

\[(\,C{:}4,\ (A{:}2,\,B{:}3){:}3,\ (D{:}2,\,E{:}1){:}2\,)\]

Check: add up the branches on the path between two tips and compare with the table.

Pair Path Sum Table
A–B 2 + 3 5 5
A–C 2 + 3 + 4 9 9
A–D 2 + 3 + 2 + 2 9 9
A–E 2 + 3 + 2 + 1 8 8
B–C 3 + 3 + 4 10 10
Pair Path Sum Table
B–D 3 + 3 + 2 + 2 10 10
B–E 3 + 3 + 2 + 1 9 9
C–D 4 + 2 + 2 8 8
C–E 4 + 2 + 1 7 7
D–E 2 + 1 3 3

All ten match, because these distances came from a tree.

  • NJ gives an unrooted tree; add a root afterwards using an outgroup.
  • It is greedy: it takes the best-looking join at each step and never revisits it. No tree is scored, so there is no likelihood and no measure of confidence.
  • Real distances are noisy and undercount hidden changes, so the result is only as good as the distance correction.

⟹ This approach is usually appropriate for populations or closely-related species.

Probability and likelihood

Two rules of probability

Roll a fair 10-sided die twice.

AND — multiply (independent events)

A 1 on the first roll and a 2 on the second?

\[\Pr\{1 \text{ and } 2\} = \tfrac{1}{10}\times\tfrac{1}{10} = \tfrac{1}{100}\]

OR — add (mutually exclusive events)

A 1 or a 2 on one roll?

\[\Pr\{1 \text{ or } 2\} = \tfrac{1}{10}+\tfrac{1}{10} = \tfrac{1}{5}\]

Every likelihood on a tree is built from these two rules: multiply along a history, add over the histories we can’t observe.

Probability vs likelihood

I rolled 2, 1, 1, 4. Is that surprising?

\[\Pr\{2,1,1,4 \mid \text{10-sided die}\} = \left(\tfrac{1}{10}\right)^4 = \tfrac{1}{10\,000}\]

Maybe I used a 4-sided die. Is that less surprising?

\[\Pr\{2,1,1,4 \mid \text{4-sided die}\} = \left(\tfrac{1}{4}\right)^4 = \tfrac{1}{256}\]

The die is the model. Probability fixes the model and asks about data; likelihood fixes the data and compares models.

The likelihood function

The likelihood of a model is the probability of the observed data under it:

\[ \begin{aligned} L(\text{4-sided die}) &= f(X \mid \text{4-sided die}) = \prod_{h=1}^{N} f(x_h \mid \text{4-sided die}) \\[4pt] L(\theta) &= f(X \mid \theta) = \prod_{h=1}^{N} f(x_h \mid \theta) \end{aligned} \]

  • \(X = (x_1,\dots,x_N)\) are the \(N\) observations (for an alignment, its sites); \(\theta\) is the model parameters. Do not confuse with our popgen parameter \(\theta\)
  • The product is the AND rule — observations are assumed independent.
  • Maximum likelihood: pick the \(\theta\) (a combination of many parameters) that makes the observed data least surprising.

Why programs always report \(\log L\)

  • Tree likelihoods are products of thousands of site probabilities, each \(<1\).
  • A double-precision number can’t go below about \(10^{-308}\); a few thousand sites gets there.
  • The product underflows to 0. Many computer programs do not store numbers smaller than 16 decimal places.
  • So everything is done on the log scale. Throughout, \(\log\) means the natural log.

The natural log of p plotted against p from 0 to 1. The curve plunges steeply near zero; the probability 1/256 maps to minus 5.55 and 1/10,000 maps to minus 9.21.

Logs turn products into sums

\[\log(x\times y) = \log x + \log y \quad\Longrightarrow\quad \ell(\theta) = \log L(\theta) = \sum_{h=1}^{N}\log f(x_h\mid\theta)\]

Model \(\Pr\{2,1,1,4\}\) \(\log\Pr\)
10-sided die \(1/10\,000\) \(-9.21\)
4-sided die \(1/256\) \(-5.55\)

The log-likelihood is maximized by the model that surprises us the least.

Likelihood on a tree

Pretend we know the ancestors

A rooted three-tip tree. The root has state A and connects by branch v1 to tip A and by branch v2 to an internal node, also state A, which connects by branches v3 and v4 to two tips both T.

One alignment column: tips A, T, T. Suppose both ancestors were A. By the AND rule:

\[ \Pr = \pi_A \cdot P_{AA}(v_1)\cdot P_{AA}(v_2)\cdot P_{AT}(v_3)\cdot P_{AT}(v_4) \]

\(P_{ij}(v)\) = probability that state \(i\) becomes \(j\) along a branch of length \(v\).

The root: \(\pi_A\)

The same three-tip tree with the root node highlighted in red.

\[ \Pr = \bbox[#FBE3E3,3px]{\pi_A} \cdot P_{AA}(v_1)\cdot P_{AA}(v_2)\cdot P_{AT}(v_3)\cdot P_{AT}(v_4) \]

The probability that the root is in state A: the equilibrium (stationary) frequency of A.

Branch \(v_1\): root → tip A

The three-tip tree with the branch from the root to tip A highlighted in red.

\[ \Pr = \pi_A \cdot \bbox[#FBE3E3,3px]{P_{AA}(v_1)}\cdot P_{AA}(v_2)\cdot P_{AT}(v_3)\cdot P_{AT}(v_4) \]

Start as A, still A after \(v_1\).

Branch \(v_2\): root → internal node

The three-tip tree with the branch from the root to the internal node highlighted in red.

\[ \Pr = \pi_A \cdot P_{AA}(v_1)\cdot \bbox[#FBE3E3,3px]{P_{AA}(v_2)}\cdot P_{AT}(v_3)\cdot P_{AT}(v_4) \]

Branches \(v_3\) and \(v_4\): A → T, twice

The three-tip tree with the branch from the internal node to the first T tip highlighted in red.

\[ \Pr = \pi_A \cdot P_{AA}(v_1)\cdot P_{AA}(v_2)\cdot \bbox[#FBE3E3,3px]{P_{AT}(v_3)}\cdot P_{AT}(v_4) \]

Two probability components to keep track of

The three-tip tree with the branch from the internal node to the second T tip highlighted in red.

\[ \Pr = \underbrace{\pi_A}_{\text{equilibrium}} \cdot \underbrace{P_{AA}(v_1)\, P_{AA}(v_2)\, P_{AT}(v_3)\, \bbox[#FBE3E3,3px]{P_{AT}(v_4)}}_{\text{transition probabilities}} \]

Both come from the substitution model.

But we never know the ancestors

The three-tip tree with question marks at the root and the internal node.

Ancestral states are unobserved. So we add (the OR rule) over every state they could have had:

\[ L = \sum_{x}\sum_{y}\pi_x\,P_{xA}(v_1)\,P_{xy}(v_2)\,P_{yT}(v_3)\,P_{yT}(v_4) \]

\(x\) = root state, \(y\) = internal-node state, each \(\in\{A,C,G,T\}\).

One term at a time

The three-tip tree with an A C G T strip beside each internal node; A is selected at the root and A at the internal node.

\(\pi_A P_{AA}P_{AA}P_{AT}P_{AT}\)

The same tree with A selected at the root and C at the internal node.

\(\pi_A P_{AA}P_{AC}P_{CT}P_{CT}\)

The same tree with G selected at the root and T at the internal node.

\(\pi_G P_{GA}P_{GT}P_{TT}P_{TT}\)

Each choice of ancestral states is one history; its probability is a product. The site likelihood is the sum over all of them.

All 16 histories

A four-by-four grid of the three-tip tree, one copy for every combination of root state (rows A, C, G, T) and internal-node state (columns A, C, G, T), joined by plus signs.

With \(n\) tips there are \(n-1\) internal nodes, so \(4^{\,n-1}\) histories per site: 20 taxa → \(4^{19} \approx 2.7\times10^{11}\).

A lot of repetition

The same four-by-four grid with the lower subtree highlighted in red in every tree of the first column, showing that the same calculation for the internal node in state A recurs in all four rows.

The lower-subtree term \(P_{yT}(v_3)P_{yT}(v_4)\) depends only on \(y\) — yet it’s recomputed for every root state.

Pruning: do the repeated part once

Same sum, grouped so the repeated part appears only once (\(ab+ac=a(b+c)\)):

\[ L = \sum_{x}\pi_x\ \underbrace{P_{xA}(v_1)}_{\text{tip 1}}\ \sum_{y}P_{xy}(v_2)\ \underbrace{\big[P_{yT}(v_3)\,P_{yT}(v_4)\big]}_{\text{the two T tips, if the internal node is } y} \]

The bracket depends only on \(y\), not on the root state \(x\). So compute it once for each of the 4 values of \(y\), write the 4 numbers down, and reuse them for every root state.

Pruning does this at every node, from the tips toward the root. At each node keep a list of 4 numbers \(L_k\), one per base:

\(L_k(s)\) = how likely the tips below node \(k\) are, if node \(k\) has base \(s\)

(its formal name is the conditional likelihood).

Pruning, step 1: start at the tips

A tip’s base is known, so its list is simple: the observed base explains the data perfectly (1), and any other base cannot (0).

Our column, tips A, T, T:

Base Tip 1 (A) Tip 3 (T) Tip 4 (T)
A 1 0 0
C 0 0 0
G 0 0 0
T 0 1 1

Pruning, step 2: move up one node

For each possible base at a node, handle the two children separately, then combine:

  1. One child: the chance the node’s base turns into each possible child base, times that child’s list, added up over the child’s bases.
  2. Two children: multiply their answers (given the node’s base, the two sides evolve independently).

Example: node \(S\) (children: tips 3 and 4, both T). If \(S\) has base \(y\):

  • Tip 3: \(\sum_x P_{yx}(v_3)\,L_3(x) = P_{yT}(v_3)\), because tip 3’s list is 0 except at T.
  • Tip 4: likewise \(P_{yT}(v_4)\).
  • So \(L_S(y) = P_{yT}(v_3)\times P_{yT}(v_4)\): the conditional likelihood we reused above.

In symbols, for any node \(k\) with children \(i\) and \(j\):

\[ L_k(s) = \Big[\sum_{x}P_{sx}(v_i)\,L_i(x)\Big]\times\Big[\sum_{x}P_{sx}(v_j)\,L_j(x)\Big] \]

Sum = OR, multiply = AND

Pruning, step 3: the root, then done

Treat the root \(R\) like any other node (children: tip 1 and \(S\)) to get its list \(L_R\). Nothing lies above it, so weight each base by its equilibrium frequency \(\pi_s\) and add:

\[ L_{\text{site}} = \sum_{s}\pi_s\,L_R(s) \]

The whole algorithm for one site:

  1. Write the 0/1 lists for the tips.
  2. Work upward, node by node, using the step 2 recipe.
  3. At the root, weight by \(\pi\) and add.

Cost: one small calculation per internal node, so it grows with the number of taxa, versus \(4^{\,n-1}\) histories by brute force (Felsenstein 1981). Repeat for every site.

A worked number: the set-up

The three-tip tree with question marks at the root and the internal node.

One alignment column: tips A, T, T. Call the internal node S and the root R.

  • Branch lengths (expected substitutions per site): \(v_1=0.2\), \(v_2=v_3=v_4=0.1\).
  • Model JC69: all four bases equally common (\(\tfrac14\) each) and all changes equally likely.

Along a branch, a base either stays the same or changes to one particular other base:

Branch length Chance of staying Chance of changing to one specific base
0.1 0.90638 0.03121
0.2 0.82445 0.05852

Check: 0.90638 + 3 × 0.03121 = 1 (stay, or change to one of the 3 other bases).

A worked example: node \(S\)

\(S\) has two T tips on branches of length 0.1. For each possible state of \(S\), how likely are two T tips?

If \(S\) is… What must happen Probability
T both branches stay T \(0.90638^2=\mathbf{0.82152}\)
A (or C, or G) both branches change to T \(0.03121^2=0.00097\)

If \(S\) were anything other than T, two separate changes would be needed: about 850 times less likely.

We keep these as a list with one entry per state of \(S\):

\[ L_S = (A,\ C,\ G,\ T) = (0.00097,\ 0.00097,\ 0.00097,\ 0.82152) \]

A worked example: node \(R\) (the root)

For each possible root state, two things must work out: tip 1 (A, branch 0.2) and the subtree below \(S\) (branch 0.1). Multiply the two.

Root = A. Tip 1 stays A: 0.82445. For the subtree, add over the possible states of \(S\), each weighted by the chance of the root becoming that state:

\[ \underbrace{0.90638 \times 0.00097}_{S=A} + \underbrace{0.03121 \times 0.00097}_{S=C} + \underbrace{0.03121 \times 0.00097}_{S=G} + \underbrace{0.03121 \times 0.82152}_{S=T} \]

\[ = 0.00088 + 0.00003 + 0.00003 + 0.02564 = 0.02658 \]

Root = A: \(0.82445 \times 0.02658 = 0.02191\). The same recipe gives the other roots:

Root is Tip 1 (needs A) Subtree below \(S\) Product
A 0.82445 (stay) 0.02658 0.02191
C 0.05852 (change) 0.02658 0.00156
G 0.05852 (change) 0.02658 0.00156
T 0.05852 (change) 0.74470 0.04358

For C and G the subtree term equals A’s (they treat \(S\) the same way). For T it is mostly \(S=\) T staying T: \(0.90638\times0.82152=0.74461\), plus a little from \(S=\) A, C, G.

A worked example: the probability of the whole graph

Each root state is equally likely a priori (\(\tfrac14\)). Weight the products and add:

\[ L = \tfrac14\,(0.02191 + 0.00156 + 0.00156 + 0.04358) = \mathbf{0.01715}, \qquad \log L = -4.066 \]

  • Check: adding up all 16 root/internal-state histories by brute force gives the same number. Pruning is the same sum, organized.
  • Which root state is most plausible? Each one’s share of the total: T 64%, A 32%, C 2%, G 2%. The two T tips outvote the single A, because tip 1’s branch (0.2) is the longest, so it says the least about the root.
  • That was one site. An alignment has many sites, and we combine their likelihoods next.

From one site to an alignment

Sites are assumed independent — the AND rule again. \(L_h\) is the likelihood of site \(h\) (the pruning result):

\[ L = \Pr(\text{data}\mid \tau, \text{model}) = \prod_{h=1}^{N} L_h \qquad\Longrightarrow\qquad \ell = \sum_{h=1}^{N}\log L_h \]

  • Maximum likelihood (ML): find the topology \(\tau\), branch lengths and model parameters that maximize \(\ell\) (RAxML, IQ-TREE).
  • Identical columns (site patterns) are computed once and weighted by their counts.

Models of sequence evolution

Where do the \(P_{ij}(v)\) come from?

An unrooted four-taxon tree with its five branches labelled v1 to v5.

A model needs:

  • Topology and branch lengths \(v_i\), in expected substitutions per site.
  • A rate matrix \(Q\) saying how bases change into each other per unit of \(v\).

From these two, every \(P_{ij}(v)\) follows.

Branch lengths confound rate and time

The four-taxon tree with the internal branch v3 highlighted and labelled v3 equals 0.01, meaning one substitution expected per 100 sites.

For a 10,000 bp locus:

\[ \begin{aligned} \tfrac{1\ \text{sub}}{1\ \text{My}}\times 100\ \text{My} &= \tfrac{100\ \text{subs}}{10\,000\ \text{bp}} = 0.01\\[4pt] \tfrac{10\ \text{subs}}{1\ \text{My}}\times 10\ \text{My} &= \tfrac{100\ \text{subs}}{10\,000\ \text{bp}} = 0.01 \end{aligned} \]

Same branch length, a tenfold difference in time. Separating them needs a clock model.

The substitution process is a Markov chain

Model each site as a continuous-time Markov chain over states {A, C, G, T}. The instantaneous rate matrix \(Q\) gives the rate of change from state \(i\) to \(j\):

\[ Q = \{q_{ij}\},\qquad q_{ii} = -\sum_{j\neq i} q_{ij}\ \text{(rows sum to 0)}. \]

The probability of ending in state \(j\) after time \(t\), starting in \(i\):

\[ P(t) = e^{Qt}, \qquad P_{ij}(t) = \Pr(j \text{ at } t \mid i \text{ at } 0). \]

  • \(P(t)\) averages over all unobserved histories on a branch — including multiple hits.
  • Markov process: only the current state matters, not how it got there. Named after Andrey Markov (1856–1922).
  • Time-reversible: \(\pi_i q_{ij} = \pi_j q_{ji}\) — the process looks the same run backwards.

Jukes & Cantor (1969): the simplest \(Q\)

\[ Q = \begin{array}{c|cccc} & A & C & G & T\\ \hline A & -3\lambda & \lambda & \lambda & \lambda\\ C & \lambda & -3\lambda & \lambda & \lambda\\ G & \lambda & \lambda & -3\lambda & \lambda\\ T & \lambda & \lambda & \lambda & -3\lambda \end{array} \]

  • Every change happens at the same rate \(\lambda\) (Jukes and Cantor 1969).
  • Equilibrium frequencies are equal: \(\pi_A=\pi_C=\pi_G=\pi_T=0.25\).
  • A leaves A at total rate \(3\lambda\), so \(q_{AA}=-3\lambda\): the rows sum to 0 and probability is conserved.

Deriving \(P(t)\) for JC69: the set-up

A site that starts as A, a branch of length t, and a question mark at the end with dashed arrows to the four possible bases A, C, G and T.

  • Let “events” happen at rate \(\mu\). Each event draws a new base uniformly from all four — so an event can redraw A as A.
  • A specific change, say A→C, happens at rate \(\mu/4\). That’s \(\lambda\), so \(\mu = 4\lambda\).

Deriving \(P(t)\) for JC69: the Poisson step

The number of events in time \(t\) is Poisson with mean \(\mu t\):

\[ \begin{aligned} \Pr(\text{no event}) &= e^{-\mu t} \qquad\qquad \Pr(\text{at least one event}) = 1 - e^{-\mu t}\\[4pt] \Pr(\text{last event drew a T}) &= \tfrac14 \end{aligned} \]

\[ \begin{aligned} P_{AT}(t) &= \tfrac14\left(1 - e^{-\mu t}\right) = \tfrac14 - \tfrac14 e^{-4\lambda t}\\[4pt] P_{AA}(t) &= \underbrace{e^{-4\lambda t}}_{\text{nothing happened}} + \underbrace{\tfrac14\left(1-e^{-4\lambda t}\right)}_{\text{events, but ended on A}} = \tfrac14 + \tfrac34 e^{-4\lambda t} \end{aligned} \]

Check: \(P_{AA} + 3P_{AT} = 1\). These are \(e^{Qt}\) for the JC69 \(Q\).

In branch-length units

Measure a branch by the expected number of real substitutions per site, \(v\): the rate of leaving a base times time, \(v = 3\lambda t\). So \(4\lambda t = \tfrac43 v\) and

\[ P_{ii}(v) = \tfrac14 + \tfrac34 e^{-4v/3}, \qquad P_{ij}(v) = \tfrac14 - \tfrac14 e^{-4v/3}. \]

Transition probabilities against branch length: the probability of staying A falls from 1 toward one quarter, and the probability of A becoming C rises from 0 toward one quarter.

The model corrects for multiple hits

The observed proportion of differing sites p plotted against the true distance d. p rises along the one-to-one line at first, then bends and saturates at three quarters; at d of 0.8 the gap between d and p is labelled hidden multiple hits.

Two sequences separated by a total path length \(v\) differ at a proportion

\[p = \tfrac34\left(1 - e^{-4v/3}\right)\]

of sites. Invert it to estimate \(v\) from the data, the JC distance \(d\):

\[d = -\tfrac34\log\!\left(1 - \tfrac43 p\right)\]

The raw proportion \(p\) always underestimates \(v\) once sites have been hit twice.

HKY85: transitions ≠ transversions

The four bases arranged in a square, purines A and G on the left, pyrimidines C and T on the right. Thick red arrows mark the transitions A to G and C to T; thin grey arrows mark the four transversions.

\[ Q = \begin{array}{c|cccc} & A & C & G & T\\ \hline A & - & \pi_C\beta & \pi_G\beta\kappa & \pi_T\beta\\ C & \pi_A\beta & - & \pi_G\beta & \pi_T\beta\kappa\\ G & \pi_A\beta\kappa & \pi_C\beta & - & \pi_T\beta\\ T & \pi_A\beta & \pi_C\beta\kappa & \pi_G\beta & - \end{array} \]

GTR: let the data decide

\[ Q = \begin{array}{c|cccc} & A & C & G & T\\ \hline A & - & \pi_C a\mu & \pi_G b\mu & \pi_T c\mu\\ C & \pi_A a\mu & - & \pi_G d\mu & \pi_T e\mu\\ G & \pi_A b\mu & \pi_C d\mu & - & \pi_T f\mu\\ T & \pi_A c\mu & \pi_C e\mu & \pi_G f\mu & - \end{array} \]

  • Six exchangeabilities \(a\ldots f\) (five free, one fixed for scale) plus three free base frequencies: 8 parameters (Tavaré 1986).
  • Most models in use sit between JC69 and GTR. Classic RAxML uses GTR for DNA; IQ-TREE picks a model for you.
  • Amino-acid and codon models exist too, with their own complications.

A hierarchy of substitution models

Models differ in how much of \(Q\) they let vary (base frequencies \(\pi\), exchangeabilities):

Model Base freqs Rates Free \(Q\) parameters
JC69 equal all equal 0
K80 equal transitions ≠ transversions (\(\kappa\)) 1
HKY85 free transitions ≠ transversions (\(\kappa\)) 4
GTR free all six exchangeabilities free 8

Each model is a special case of the one below it: they are nested.

Rate variation among sites

Every model above can add rate variation: some sites evolve quickly, others hardly at all.

Even RuBisCO, encoded by rbcL — the workhorse gene of early plant phylogenetics — combines a mass of strongly constrained sites with a few residues under positive selection in most land-plant lineages (Kapralov and Filatov 2007).

The distribution of sites under selection in *rbcL* and their relevance in the context of a 3-D protein structure.

The distribution of sites under selection in rbcL and their relevance in the context of a 3-D protein structure. Figs. 1 and 2 from Kapralov and Filatov (2007)

Approach 1: partition the alignment

An alignment drawn as a bar of five blocks alternating blue (slow, r1 Q) and red (fast, r2 Q).

Each region gets its own rate multiplier on the same \(Q\). For JC69 in region \(k\):

\[ P_{ii}(v) = \tfrac14 + \tfrac34 e^{-4 r_k v/3}, \qquad P_{ij}(v) = \tfrac14 - \tfrac14 e^{-4 r_k v/3} \]

  • Pro: each site’s likelihood is computed once.
  • Con: the regions must be chosen a priori, and rates still vary within them.

Approach 2: a mixture at every site

Let every site have any of \(K\) rates, and average (the OR rule) over them. With \(K=4\) equally likely rates:

\[ L(\theta) = \prod_{h=1}^{N}\left[\tfrac14\Pr(x_h\mid r_1) + \tfrac14\Pr(x_h\mid r_2) + \tfrac14\Pr(x_h\mid r_3) + \tfrac14\Pr(x_h\mid r_4)\right] \]

Here \(x_h\) is alignment column \(h\). No partitions needed. But where do \(r_1,\dots,r_4\) come from?

Rates from a gamma distribution

Gamma densities with mean 1 for shape alpha 0.2, 1 and 5. Alpha 0.2 is L-shaped with most mass near zero; alpha 1 is exponential; alpha 5 is a hump centred near 1.

Assume relative rates follow a gamma distribution with mean 1 (Yang 1994):

\[r \sim \text{Gamma}(\text{shape}=\alpha,\ \text{rate}=\alpha)\]

  • Small \(\alpha\) → strong heterogeneity.
  • Large \(\alpha\) → nearly equal rates.
  • One extra parameter: “\(+\Gamma\)”.

The discrete gamma

The exponential density (gamma with alpha 1) split into four equal-probability quartiles by dashed lines; red stems at the quartile means 0.137, 0.477, 1.000 and 2.386.

Split the gamma into \(K\) equal-probability categories (often 4) and use each category’s mean as its rate (Yang 1994).

\(\alpha\) \(r_1\) \(r_2\) \(r_3\) \(r_4\)
1 0.137 0.477 1.000 2.386
0.5 0.033 0.252 0.820 2.894

The rates average to 1.

Beyond the gamma

  • Free-rate model: estimate the \(K\) rates and their weights directly, with no gamma shape assumed (Yang 1995). Usually written \(+R_K\) in IQ-TREE.
  • Needs enough data to estimate the categories; \(+\Gamma\) is still widely used.
  • \(+I\): a proportion of sites that never change.

Choosing a model

The Akaike Information Criterion

\[ \mathrm{AIC} = 2k - 2\log \hat L \]

  • \(k\) = number of free parameters; \(\hat L\) = the maximum likelihood.
  • More parameters always raise \(\hat L\); the \(2k\) term charges for them (Akaike 1974).
  • Goal: the best — or least-worst — model among those we tried.

A true model exists

A cartoon of three blue axes with a red point high above the floor labelled truth, f: the process that really generated the data.

Our models live in a smaller space

The truth point with a vertical red line dropped to its projection on the floor, labelled its projection into the space where our models live.

An approximating model

The projection connected by a horizontal red line to a second point on the floor labelled an approximating model, g (e.g. GTR).

The KL divergence contains the AIC

The right triangle completed by a dashed hypotenuse from truth to model labelled KL(f, g); the vertical side is labelled fixed, expectation of log f, and the horizontal side model-specific, minus the expectation of log g.

\[ \begin{aligned} \mathrm{KL}(f,g) &= \int f(x)\log\frac{f(x)}{g(x)}\,dx\\[4pt] &= \underbrace{E_f[\log f]}_{\text{same for every model}} - \underbrace{E_f[\log g]}_{\text{model-specific}} \end{aligned} \]

AIC estimates the model-specific part (times \(-2\)), corrected for the bias of using the same data to fit \(g\).

Several candidate models

Three candidate models JC69, HKY and GTR on the floor, each connected to the projection of the truth; HKY has the shortest connection and is highlighted in red.

AIC estimates each model’s distance from the truth, up to a constant shared by all of them.

Comparing candidate models: a baobab gene

ModelFinder on one 1,950 bp nuclear gene from the baobab data (18 sequences):

Model \(k\) \(\log \hat L\) AIC \(\Delta\)AIC \(\Delta\)BIC
JC69 33 −4221.76 8509.52 214.48 186.60
JC69+Γ 34 −4206.20 8480.40 185.35 163.05
K80 34 −4157.71 8383.43 88.38 66.08
K80+Γ 35 −4141.26 8352.52 57.47 40.75
HKY85 37 −4125.81 8325.63 30.58 25.00
HKY85+Γ 38 −4109.52 8295.05 0 0
GTR 41 −4123.40 8328.79 33.75 50.47
GTR+Γ 42 −4106.94 8297.88 2.83 25.13
  • The biggest gains: transition/transversion bias (JC69 → K80) and rate variation (\(+\Gamma\)).
  • GTR+Γ fits a little better (+2.6 log-likelihood units) but spends 4 more parameters. AIC calls it close (ΔAIC 2.8); BIC, with its heavier penalty, rejects it (ΔBIC 25).

Alternatives to AIC

\[ \mathrm{AIC_c} = 2k - 2\log\hat L + \frac{2k^2 + 2k}{N - k - 1} \qquad\qquad \mathrm{BIC} = k\log N - 2\log\hat L \]

  • AICc corrects for small sample size \(N\) (alignment length).
  • BIC penalizes parameters more heavily as \(N\) grows (Schwarz 1978).
  • There’s no consensus on which is best; programs report all three. ModelFinder uses BIC by default.

ML in practice

Many loci, one analysis

Ten small alignment blocks scattered in a circle, each four rows deep, joined by an arrow into one long concatenated supermatrix.

If you can analyze one locus well, you can analyze a hundred — usually by concatenating them into one supermatrix.

Should each gene have its own model?

The same concatenated supermatrix with red dashed lines at every locus boundary, asking whether to use one model per locus.

Partitioned analyses give each block its own \(Q\), rate variation and/or relative rate. Concatenated ML with partitioned mixture models is in almost every phylogenomics paper.

Same data, different models, different trees

What is sister to all other flowering plants? Disagreement between the slowly evolving sites and fast sites (Xi et al. 2014).

Conflicting evidence for early angiosperm evolution based on partitioning strategies, from Xi et al. 2014.

Conflicting evidence for early angiosperm evolution based on partitioning strategies and model choice. Fig. 1 from Xi et al. (2014)

The Amborella genome paper (Amborella Genome Project 2013) has Amborella alone as sister to all other angiosperms, the more accepted hypothesis.

Target capture: conserved exons, variable flanks

A U-shaped curve of relative rate across a target-capture locus, low over the targeted exon and rising through the flanking introns; below, two partitioning schemes: one partition per locus, and separate intron, exon and intron partitions.

Plant phylogenomics mostly uses target capture (Hyb-Seq): baits for low-copy exons, plus the flanking introns that come along with them (Weitemier et al. 2014; Johnson et al. 2016).

Data-driven partitioning and model selection

ModelFinder results: BIC scores ranking candidate substitution models, estimated free-rate categories, and distances between trees inferred under different models.

ModelFinder searches substitution models, \(+\Gamma\) / \(+I\) / free-rate, and partitioning schemes, and returns the best by BIC (Kalyaanamoorthy et al. 2017).

IQ-TREE 2 (Minh et al. 2020):

  • fast, with model selection built in;
  • well-annotated log files;
  • many conveniences for many loci;
  • most often returns the highest likelihood in benchmarks.

Support for ML trees: the bootstrap

  • Resample alignment columns with replacement, re-estimate the tree, repeat 100–1000×.
  • Bootstrap support for a clade = the fraction of replicate trees that contain it (Felsenstein 1985).
  • It measures how consistently the data point to a clade, given the model. It is not the probability that the clade is true.

Key takeaways

Summary

  • A phylogeny is a model-based hypothesis.
  • Neighbor-joining: a distance-based method that is fast and works well when divergence is not too high. It often gives a reasonable starting tree.
  • Likelihood: considers the uncertainty in the unobserved states.
  • Models: JC69 ⊂ HKY ⊂ GTR, plus \(+\Gamma\) / partitions. An example was for a single baobab gene, HKY85+Γ wins on AIC and BIC. This could lead to different branch lengths and potentially a different gene tree.
  • Confidence intervals can be placed on ML trees using bootstrapping.

References

Akaike, Hirotugu. 1974. “A New Look at the Statistical Model Identification.” IEEE Transactions on Automatic Control 19 (6): 716–23. https://doi.org/10.1109/TAC.1974.1100705.
Amborella Genome Project. 2013. “The Amborella Genome and the Evolution of Flowering Plants.” Science 342 (6165): 1241089. https://doi.org/10.1126/science.1241089.
Chase, Mark W., Douglas E. Soltis, Richard G. Olmstead, et al. 1993. “Phylogenetics of Seed Plants: An Analysis of Nucleotide Sequences from the Plastid Gene rbcL.” Annals of the Missouri Botanical Garden 80 (3): 528–80. https://doi.org/10.2307/2399846.
Felsenstein, Joseph. 1981. “Evolutionary Trees from DNA Sequences: A Maximum Likelihood Approach.” Journal of Molecular Evolution 17 (6): 368–76. https://doi.org/10.1007/BF01734359.
———. 1985. “Confidence Limits on Phylogenies: An Approach Using the Bootstrap.” Evolution 39 (4): 783–91. https://doi.org/10.1111/j.1558-5646.1985.tb00420.x.
Grass Phylogeny Working Group II. 2012. “New Grass Phylogeny Resolves Deep Evolutionary Relationships and Discovers C4 Origins.” New Phytologist 193 (2): 304–12. https://doi.org/10.1111/j.1469-8137.2011.03972.x.
Hasegawa, Masami, Hirohisa Kishino, and Taka-aki Yano. 1985. “Dating of the Human-Ape Splitting by a Molecular Clock of Mitochondrial DNA.” Journal of Molecular Evolution 22 (2): 160–74. https://doi.org/10.1007/BF02101694.
Johnson, Matthew G., Elliot M. Gardner, Yang Liu, et al. 2016. “HybPiper: Extracting Coding Sequence and Introns for Phylogenetics from High-Throughput Sequencing Reads Using Target Enrichment.” Applications in Plant Sciences 4 (7): 1600016. https://doi.org/10.3732/apps.1600016.
Johnson, Matthew G., Lisa Pokorny, Steven Dodsworth, et al. 2019. “A Universal Probe Set for Targeted Sequencing of 353 Nuclear Genes from Any Flowering Plant Designed Using k-Medoids Clustering.” Systematic Biology 68 (4): 594–606. https://doi.org/10.1093/sysbio/syy086.
Jukes, Thomas H., and Charles R. Cantor. 1969. “Evolution of Protein Molecules.” In Mammalian Protein Metabolism, edited by Hamish N. Munro, 21–132. New York: Academic Press. https://doi.org/10.1016/B978-1-4832-3211-9.50009-7.
Kalyaanamoorthy, Subha, Bui Quang Minh, Thomas K. F. Wong, Arndt von Haeseler, and Lars S. Jermiin. 2017. “ModelFinder: Fast Model Selection for Accurate Phylogenetic Estimates.” Nature Methods 14 (6): 587–89. https://doi.org/10.1038/nmeth.4285.
Kapralov, Maxim V., and Dmitry A. Filatov. 2007. “Widespread Positive Selection in the Photosynthetic Rubisco Enzyme.” BMC Evolutionary Biology 7 (1): 73. https://doi.org/10.1186/1471-2148-7-73.
Karimi, Nisa, Corrinne E. Grover, Joseph P. Gallagher, Jonathan F. Wendel, Cécile Ané, and David A. Baum. 2020. “Reticulate Evolution Helps Explain Apparent Homoplasy in Floral Biology and Pollination in Baobabs (Adansonia; Bombacoideae; Malvaceae).” Systematic Biology 69 (3): 462–78. https://doi.org/10.1093/sysbio/syz073.
Magallón, Susana, Sandra Gómez-Acevedo, Luna L. Sánchez-Reyes, and Tania Hernández-Hernández. 2015. “A Metacalibrated Time-Tree Documents the Early Rise of Flowering Plant Phylogenetic Diversity.” New Phytologist 207 (2): 437–53. https://doi.org/10.1111/nph.13264.
Minh, Bui Quang, Heiko A. Schmidt, Olga Chernomor, et al. 2020. “IQ-TREE 2: New Models and Efficient Methods for Phylogenetic Inference in the Genomic Era.” Molecular Biology and Evolution 37 (5): 1530–34. https://doi.org/10.1093/molbev/msaa015.
Schwarz, Gideon. 1978. “Estimating the Dimension of a Model.” The Annals of Statistics 6 (2): 461–64. https://doi.org/10.1214/aos/1176344136.
Tagliacollo, Victor A., and Robert Lanfear. 2018. “Estimating Improved Partitioning Schemes for Ultraconserved Elements.” Molecular Biology and Evolution 35 (7): 1798–1811. https://doi.org/10.1093/molbev/msy069.
Tavaré, Simon. 1986. “Some Probabilistic and Statistical Problems in the Analysis of DNA Sequences.” Lectures on Mathematics in the Life Sciences 17: 57–86.
The Angiosperm Phylogeny Group. 2016. “An Update of the Angiosperm Phylogeny Group Classification for the Orders and Families of Flowering Plants: APG IV.” Botanical Journal of the Linnean Society 181 (1): 1–20. https://doi.org/10.1111/boj.12385.
Weitemier, Kevin, Shannon C. K. Straub, Richard C. Cronn, et al. 2014. “Hyb-Seq: Combining Target Enrichment and Genome Skimming for Plant Phylogenomics.” Applications in Plant Sciences 2 (9): 1400042. https://doi.org/10.3732/apps.1400042.
Xi, Zhenxiang, Liang Liu, Joshua S. Rest, and Charles C. Davis. 2014. “Coalescent Versus Concatenation Methods and the Placement of Amborella as Sister to Water Lilies.” Systematic Biology 63 (6): 919–32. https://doi.org/10.1093/sysbio/syu055.
Yang, Ziheng. 1994. “Maximum Likelihood Phylogenetic Estimation from DNA Sequences with Variable Rates over Sites: Approximate Methods.” Journal of Molecular Evolution 39 (3): 306–14. https://doi.org/10.1007/BF00160154.
———. 1995. “A Space-Time Process Model for the Evolution of DNA Sequences.” Genetics 139 (2): 993–1005. https://doi.org/10.1093/genetics/139.2.993.