fix(hsj): make the HSJ score rooting-invariant (T-374, HSJ half)

[ms609]

Closes the HSJ half of red-team finding T-374 (P1). Stacked conceptually on
PR #280 (T-375/T-376), which had to land first for the primary term to be a
function of the data at all.

What was actually wrong

The brief's premise needs correcting, and it changed the shape of the work.
score_hierarchy_block()'s a(n)/p(n) recurrences already are the
paper's Algorithm 1 (p.15), lines 6-7, term for term. The two-state DP was
implemented, and it was already rooting-invariant: its branch costs are
symmetric and it minimises over the root's own state. No DP rework was needed.

The defect was that d(u, v) — the secondary dissimilarity — was read off
fitch_label_char()'s directional resolution. Two distinct mechanisms:

1. Semantic. "-" was admitted as an ordinary state of a secondary
   character. Where a controlling primary codes a structure absent its
   secondaries do not exist, so "-" there is not a state the character takes.
   Admitting it let the uppass propagate "-" inwards and label a node in the
   middle of the present region "inapplicable", where it is disjoint from
   every present neighbour in every secondary at once — charging that branch
   d = m, the full alpha. That over-charge is wrong under any rooting; the
   paper's d counts nonmatching secondaries among characters that apply
   (p.5).
2. Directional. The remaining resolution was a DELTRAN uppass whose
   direction, and tie-break support counts whose subtrees, were properties of
   the input rooting.

Copying Algorithm 1 literally would not have fixed this either: its line 2 sets
L(n) to the first-pass Fitch labelling — a downpass, hence root-dependent —
and it takes "Tree, T, with root r" as input. Theorem 2 claims it returns the
minimal score, which would make it rooting-invariant; the measurements below are
evidence against that claim as stated.

Key measurement that localised it

Under any most-parsimonious reconstruction of secondary j, the number of
branches on which j changes is FitchLen_j. So for an all-present block
the alpha term is (alpha/m) * sum_j FitchLen_j exactly, whichever
labelling the uppass picks — that regime was always invariant. Confirmed 50/50
on random data, and it reproduces Fig. 1 exactly (6+alpha and 3+2alpha, i.e.
7 and 5 at alpha = 1). The bug could therefore only ever surface on blocks
with mixed present/absent primaries, which is what collapsed the design space.

Root-dependent topologies / worst spread, over every edge-rooting of 30
random 9-tip topologies at alpha = 1:

regime	93c81a9a	+ fix 1	+ fix 2
all-present	0/30 0.00	0/30 0.00	0/30 0.00
mixed, m = 2	21/30 1.00	3/30 0.50	0/30 0.00
mixed, m = 4	26/30 1.00	7/30 0.25	0/30 0.00
? primary, m = 2	6/30 0.50	1/30 0.50	0/30 0.00
? primary, m = 4	18/30 0.50	3/30 0.50	0/30 0.00

Re-run crossed with 3-state secondaries after review: 0/30 and spread 0.0000 in
all twelve cells.

Headline symptom, closed. Twelve HSJ searches on 16-tip mixed matrices: 3/12
returned trees whose TreeLength() disagreed with the reported score (worst gap
0.75, one pool spanning 31.25/31.5/31.75/32) both at 93c81a9a and after fix
1 alone; now 0/12, every pool internally consistent.

Why canonical rooting, and why that is not PR #278 again

The exact minimum over labellings is the inapplicable-Fitch problem and is not
available in linear time. (The per-branch min-over-MPR relaxation is refuted: a
0,1,0,1 caterpillar gives sum-of-min 0 against min-of-sum 2.) A composite
Sankoff over {absent} plus all secondary combinations, with the symmetric HSJ
cost matrix, would be exact and is likewise rooting-invariant — the matrix
satisfies the triangle inequality, so even edge-subdivision is safe — but it is
exponential in the number of secondaries and is what the XFORM path already
does; it would delete HSJ's reason to exist alongside XFORM. So the labelling
passes are rooted canonically at tip 0 inside the kernel.

This is deliberately not PR #278's reporting-boundary canonicalisation, which
the brief rightly pre-empted. That left search optimising a root-dependent
objective and reconciled the number afterwards. This makes the objective the
search optimises a function of the unrooted topology, so MaximizeParsimony
agrees with TreeLength by construction rather than by re-scoring. It buys
invariance, not exactness. The option was put to the maintainer with the
measured residual attached and chosen explicitly.

Scope of the freeing rule (review catch)

A secondary is freed only where its primary cannot be present, not merely
where it may be absent. The lax test would wildcard the secondaries of a "?"
primary even when they were observed — discarding real evidence that the
structure is present — and would be self-erasing, since a block of all-"?"
primaries would then have an empty applicable domain and a silently zero alpha
term. ValidateHierarchy whitelists "?" primaries, so that data is reachable.
This matches recode_hierarchy.R's tipStates == -2L / tipSecKnown path,
added under T-379 for the same reason.

Tests

In test-ts-hsj.R, all phrased against what the paper requires. Verified
against a build of 93c81a9a in a throwaway worktree — note
testthat::set_max_fails(Inf), as the default cap of 10 silently hid the last
test on the first attempt:

test	pre-fix failures
rooting invariance, fixed mixed block	2 (alpha = 0 passes, as the control)
rooting invariance, random matrices, crossed over 2/3-state secondaries and "0"/"?" primaries	13
"-" vs "?" at an inapplicable secondary	5
MaximizeParsimony pool vs reported score	5

Three are regression floors (pass before and after, to catch a fix that buys
invariance by breaking what was right): Fig. 1 = 7/5; the all-present closed
form; and "an observed secondary at a ? primary still counts", which pins the
narrowness of the freeing rule — discarding information is perfectly
rooting-invariant, so no other test here would have caught the lax version.

Two tests had to be rebuilt before they tested anything. The "-" vs "?" test
first used the p.5 "at most 1 per branch" bound, which summed over a tree is far
too loose and passed against the pre-fix build. The MaximizeParsimony test
was first bounded by maxSeconds and was flaky in both directions; it is now
bounded by maxReplicates, on a construction where 14 of the first 40 seeds are
discordant pre-fix.

hsj-paper-oracle.R goes 13 pass / 4 fail at 93c81a9a to 17 / 0, with every
other check identical across the two builds. Its rooting check now sweeps all
2n-3 edges and adds the discriminating mixed-block cases; its old header
corollary about the missing two-state DP is corrected.

Other

• Four now-false root-invariance comments narrowed to XFORM rather than deleted
  (ts_tbr.cpp x2, ts_rcpp.cpp, R/MaximizeParsimony.R).
  ts_tbr.cpp:802-818 was already correct and is untouched.
• Per-score cost unchanged (0.030 s median for 40 x 60-tip HSJ TreeLength,
  both builds); the canonical order is built once per block, not per character.
• Full suite: 33 failures before and after, identical file-by-file — all
  pre-existing test_dir-outside-R CMD check artefacts.
• CI note: windows-latest (release) and EasyTrees shinytest2 fail on the
  cpp-search base branch too, in the "Set up R dependencies" step —
  pre-existing and unrelated to this PR.
• NEWS and both vignettes updated. HSJ scores on mixed data may differ.
  Measured by differencing the two builds at each tree's own rooting over 180
  tree-matrix pairs: 152 unchanged, 27 decreased (by up to 1), 1 increased by
  0.25 — a rise is possible because the score is now taken at a fixed
  canonical rooting rather than at whichever rooting the tree arrived in.

🤖 Generated with Claude Code