fix: derive XFORM secondary state spaces from contrasts, not token strings

[ms609]

Fixes #11
Fixes #12
Fixes #17

A hierarchy block took its secondary characters' states from the distinct token
strings observed in each column, minus "-" and "?". Three defects followed.

#11 (T-393) — an ambiguity token became a state of its own

"{01}" is neither "-" nor "?", so it survived the filter and was admitted as a
level: one Hamming step from both "0" and "1" rather than matching either. A
polymorphic cell therefore cost a step that no resolution of it requires, and each such
cell multiplied the combination count. Levels now come from the phyDat contrast
matrix, mirroring DataSet::token_states, which the HSJ path already reads.

An unresolved secondary is carried as a bit mask of admissible levels rather than a
single level index, so an ambiguity narrower than the full state space still constrains
the combination. Both C++ consumers (unpack_xform, ts_sankoff_test) were changed
together.

A side effect worth noting: level order was previously locale collation over token
strings, so sort() could order "{01}" before "0". It is now the contrast column
order.

#12 (T-394) — a secondary with no observed state made the whole score Inf

Validation asks that a secondary be coded inapplicable where its primary is absent,
never that any state of it remain observed. Dropping a taxon can therefore leave a
secondary all-gap in a dataset that validated. prod() over a zero-width state space
gave nStates == 1, no tip admitted any state, and TreeLength() returned Inf
silently; MaximizeParsimony() then aborted at the reporting if with
missing value where TRUE/FALSE needed, because diff(range(c(Inf, Inf))) is NaN.

Such a secondary is now carried as one unobserved level: it adds nothing to any tree's
length, but still counts towards its block's gain cost, which the hierarchy fixes rather
than the taxa sampled. The NaN is guarded independently of what produced it, in
.ReportXformScore().

#17 (T-401) — the report block scored its own collapse output

MaximizeParsimony() rescores its returned pool at the canonical rooting. It now scores
the binary pool a contracted tree came from, so it no longer depends on the HSJ/XFORM
collapse no-op staying in place, and warns when it does so — reporting the length of
trees the user is not handed is itself the discrepancy T-385 was filed for.

The comment does not claim the two pools have equal lengths. That is not derivable:
the collapse flags are decided over ds.blocks[], the Fitch term alone, so an
XFORM-blind unblocking would contract branches that are not zero-length under the
Sankoff term.

Intended score changes

X-transformation lengths change on data with polymorphic or partly ambiguous secondary
characters, and do not increase. A cold review fuzzed ~780 random matrices against a
brute-force "minimum over concrete resolutions" oracle: 0 mismatches. Across 1170
old-vs-new comparisons: 573 decreased, 597 identical, 0 increased. Separately,
lengths that were Inf become finite. Data coded only with unambiguous tokens, ? and
- is otherwise unaffected.

New hard error: a secondary with more than 31 states cannot be recoded (one bit per
level). Documented in NEWS.md, the roxygen and vignettes/inapplicable.Rmd.

Pre-fix failure output

tests/testthat/test-ts-xform-statespace.R against cpp-search @ 99b79d8f
(capped at 10):

2. Failure (:50): Polymorphic secondary costs no more than its best resolution
   `actual`: 4.0   `expected`: 3.0            # length; also n_states 4 vs 3
3. Failure (:70): Polymorphic cells do not inflate the state space
   Expected `RecodeHierarchy(ds, h)` to run silently. Actual noise: warnings.
4. Failure (:71): `actual`: 82.0  `expected`: 17.0
5. Failure (:81): A secondary beyond the mask s width -- expected an error
6. Failure (:118): Polymorphism narrows a multistate secondary
   `actual`: 5.0   `expected`: 4.0
7. Failure (:121): `recoded$tip_sec_known[2, 1]`  `actual`: 0  `expected`: 3
8. Failure (:141): Search and TreeLength agree on a multi-bit secondary mask
   `actual`: 0     `expected`: 3
9. Failure (:180): Unobserved secondary leaves a finite length
   `actual`: Inf   `expected`: 2.0
10. Error (:189): Search over a subset with an unobserved secondary
   Error in if (diff(range(canonicalScores)) > sqrt(...)) :
     missing value where TRUE/FALSE needed

One test in the file passes pre-fix and is not a regression test for these issues:
"Polymorphism narrows a multistate secondary without freeing it"'s length assertion.
It guards against the cheaper fix I did not take — treating any ambiguity as wholly
unknown, which would free the tip to state "2" and under-count. Its n_states and
mask assertions do discriminate, and are listed above.

Both encodings are numerically identical for a binary secondary (1 << (L-1) == L for
L in {1,2}), so the suite would have stayed green had the unpack_xform site been
left unpatched. That hole is now closed by a multi-bit-mask test driving both consumers
end to end.

Validation

ubuntu-arm64 green. Two CI failures are pre-existing on cpp-search, verified by
dispatching the same workflows on trunk:

• windows — fails in Set up R dependencies (pak resolution), 2/2 on cpp-search.
• gcc-ASAN tests — aborts on a std::vector<int> OOB read, identical assertion on
  cpp-search. Filed as HSJ scoring reads past the end of a std::vector<int>, caught by -D_GLIBCXX_ASSERTIONS #51 with two local reproducers; also corroborated TreeLength() on a multifurcating tree performs out-of-bounds heap writes and returns a score read from memory *before* the buffer — reachable with no error and no warning from MaximizeParsimony()'s DEFAULT output. #16 with
  evidence that its fault is an OOB write and that tryCatch() is no defence.

An earlier revision of this branch did crash ubuntu-arm64 with double free or corruption. That was this PR's own test calling TreeLength() on a polytomous tree
to demonstrate the wrong length — which is precisely the unfixed #16 corruption. The
test no longer performs that call.

Local: 561 targeted tests pass; the new file, test-recode-hierarchy.R and the T-330
file are clean under -D_GLIBCXX_ASSERTIONS. spell_check_package(vignettes = TRUE)
and check_man() clean.

🤖 Generated with Claude Code

[ms609]

Final validation, on 21de433f8 (run 30971389983):

• ubuntu-arm64: success.
• windows: fails in Set up R dependencies (error in pak subprocess), identical to cpp-search — infrastructure, no package code reached.

Two commits since the PR body was written, both from review:

• fe79c502 — the polytomy warning fired before the substituted pool was known to be non-empty, so it could promise a length recomputed from binary trees and then report the search's own score. RecodeHierarchy() also still built a full taxon-by-character character matrix that, once the state space came from the contrast matrix, was read only for each tip's primary token; the integer token matrix already carries it.

• 21de433f — the test named "Search and TreeLength agree on a multi-bit secondary mask" held by construction: since T-385 MaximizeParsimony() derives its score by calling TreeLength() on the pool, so the assertion was the same computation twice and reverting the unpack_xform half would have left it green.

  No stronger assertion is available on that fixture: brute-forcing all 105 six-taxon topologies gives optimum 3 under {01} and under each of 0, 1, 2, ?, so neither an optimum comparison nor a lower bound on replicate_scores separates the two readings. The test now asserts against that independently brute-forced optimum, is renamed to what it does, and says plainly that it does not establish agreement between the two mask consumers. What keeps those consistent is that they are edited together, recorded in .AGENTS/memory/feature-inapplicable.md.

So one claim in the PR body above is overstated and worth reading with this correction: the multi-bit-mask test closes the coverage hole (a multi-bit mask is now produced and driven through the search) but does not prove the two consumers agree.