docs(ast): document the role tables and close out #1701

Activity

1. dekobon

   added

   documentationImprovements or additions to documentationImprovements or additions to documentation

   refactorCode maintainability / tech-debt cleanup (no behaviour change)Code maintainability / tech-debt cleanup (no behaviour change)

   on Oct 23, 2026
2. dekobon

   added a parent issue on Oct 23, 2026

   • refactor(ast): one role declaration per language for Checker, Getter and metric dispatchers #1690
3. dekobon

   marked this as blocked by 10 issues on Oct 23, 2026

   • design(ast): fix the role inventory and prove the role-table macro under the gates #1691
   • feat(ast): add the role census, kind-coverage ratchet and bypass lint #1692
   • refactor(getter): declare Modifier, DeclKeyword and Primitive through role tables #1693
   • refactor(abc,cyclomatic): declare Comparison and ShortCircuit in Java, Python and C++ with every consumer — go/no-go #1694
   • refactor(abc,cyclomatic): declare the remaining paying constructs by role across languages #1695
   • refactor(checker,getter): declare FuncOpener, Closure, Container and opens_space; migrate entry-point callers #1696
   • refactor(cognitive,nexits): declare Branch, Switch, Loop, Catch, ElseIf and Exit #1697
   • refactor(checker,getter): declare Call and StringLit; fold the alterator and find/count consumers #1698
   • refactor(checker): declare DocTarget, DocComment and Docstring rows ahead of the documentation epic #1699
   • test(parity): derive cross-consumer parity from the role tables beside the hand-written suites #1700
4. dekobon commented on Oct 23, 2026

   

   dekobon

   on Oct 23, 2026

   OwnerAuthor

   More actions

   Tracking is ready

   The body now links the master issue and the work packages it references. Native parent and blocked-by relationships match the listed completion prerequisites. Implementation and acceptance checkboxes remain open; prerequisite closure and the R04: #1694 go decision govern execution order.
5. dekobon commented on Oct 31, 2026

   

   dekobon

   on Oct 31, 2026

   OwnerAuthor

   More actions

   Steps added from R07's final reviews (#1697)

   The two closing reviews of R07: #1697 — the epic-alignment audit and the architecture review — surfaced three items that are DSL-grammar or series-closing work rather than R07 defects. They are R11's by its own scope ("roles/mod.rs module doc: the row grammar"; the umbrella's completion evidence rests on role_tables.py reading every row faithfully), so they go on this package rather than into standalone issues.

   1. Name the pays [] row mode and make the Python reader see it

   pays [] — a split row with no payers — is a fourth row mode spelled as an empty list. In role_table! it selects [split] (roles/mod.rs:713-719), so OWNERS_PAY is false and only a pays_itself owner pays at the leaf. But canonical() emits owners=[…] payers=[] wraps=[…] pays_never=false, which is byte-identical to an owner-paying row with the same owners (roles/mod.rs:830-848), and utils/role_tables.py records pays_mode = "split" only from the presence of a pays [..] list. So python_reader_agrees_with_the_compiled_tables (roles/mod.rs:1554) cannot tell the two apart, and a reader that misread the mode would still pass. Today 23 rows use the shape — 22 ElseIf rows plus PHP's. The split-empty semantics are pinned by a_split_row_with_no_payers_pays_only_at_its_self_paying_owners (roles/mod.rs:1990) on the compiled side only.

   • Give the mode a name in the DSL (e.g. pays only_marked, or keep pays [] and emit the mode) and have canonical() emit it as a field beside pays_never (e.g. mode=owners|split|never), so the differential test discriminates.
   • Teach utils/role_tables.py to emit the same field from pays_mode; extend utils/role_tables-test.py (or the gate self-tests) with a row of each mode.
   • If the spelling changes, migrate the 23 rows in one commit; the census, coverage and bypass baselines must be byte-identical before and after.
   • The module-doc row grammar already names pays [] beside pays never (R07, 7d8c1eea); update it to whatever spelling lands.

   2. Decide the Switch row's body projection

   is_switch_body_kind (roles/java.rs:53-62, roles/groovy.rs) names the switch_block the Switch owner nests its arms in: a structural fact about the construct, pinned by switch_body_is_the_owners_child_and_no_rows in control_flow_tests.rs, but a free fn beside the table rather than a row attribute, and switch_block stays an unclaimed kind in .role-coverage-baseline.txt. Only cognitive reads it (it opens its nesting level at the body so a ternary or lambda in the switch's subject does not inherit a level). Two grammars want it today.

   • Decide: a DSL body attribute on the Switch row (Switch { owns [SwitchExpression(body SwitchBlock)] pays [Switch] }) that claims the kind for coverage and gives consumers a body_kind() projection, or record in the module doc that a consumer's structural projection of an owner's child lives as a named fn beside the table with a row test pinning it. Either is acceptable; the doc step must say which and why.
   • If the attribute lands, the two is_switch_body_kind fns and their // role-bypass: allow sites go, and switch_block leaves the coverage baseline.

   3. Tier-1 homes for the series' drivers

   R07 (and R06 before it) measured zero drift with a set of drivers that live only in the working caches (.claude/rules/temp-files.md tier 2), which is a citation to nothing once the host is reimaged. The evidence-record comment on #1697 carries the commands and revisions, so the claims are reproducible; the drivers are not in the tree. They will be needed again for R09 and R10, which is why this is a closing step here rather than an R07 follow-up:

   • oracle.py — runs bca and the verbatim-oracle over the integration corpora + fixture roots per language and writes one JSON per (file, metric view).

   • compare.py — diffs two oracle output trees and prints the differing/compared/key-problem summary.

   • recovery_oracle.py — the error-recovery-shape sweep (stranded C# catch, Perl when, Bash heredoc-in-compound).

   • perturb.py — the production-line perturbation sweep (reads each file once, writes mutants from the in-memory copy, restores in finally; see .claude/rules/testing.md).

   • coverage_diff.py — covered-count (not percentage) diff per file between two llvm-cov JSON reports.

   • ir_check.sh — the iRules/Tcl parity probe.

   • Move each into utils/ with a *-test.py self-test wired into make pre-commit, following the existing gate conventions (repo root from Path(__file__), tomllib, no cwd dependence), or decide per driver that it is one-off and say so on this issue. The verbatim oracle's Rust source (verbatim-oracle-src/) needs the same decision — it may belong under big-code-analysis-bench/ or xtask/.

   • The 21 R07 control-flow fixtures and 5 C5b fixtures become R10's corpus rows (noted on test(parity): derive cross-consumer parity from the role tables beside the hand-written suites #1700); nothing to do here beyond confirming they landed.

   Added by the R07 close-out; #1697's completion comment lists these as its deferred items.
6. dekobon commented on Nov 1, 2026

   

   dekobon

   on Nov 1, 2026

   OwnerAuthor

   More actions

   R07's three close-out items on this issue are now tracked as their own sub-issues so each lands with its own closing keyword and this issue stays open for the actual close-out:

   • §3 drivers and fixtures → chore(utils): tier-1 homes for the role-series drift drivers and the R07 fixtures #1744. Also records the decision that ir_check.sh and snapdiff.py are one-off and not landed, and corrects the fixture inventory (25 distinct files, none a recovery shape).
   • §1 pays [] mode → refactor(roles): emit the pays mode in canonical() and role_tables.py #1745. Count corrected: 21 rows, all ElseIf (not 23 / "22 + PHP").
   • §2 Switch-body projection — stays here, sequenced after R09. refactor/1699-doc-roles declares HoldsDocInBody { owns [StaticInitializer, Block, SwitchBlock] } for Java, which makes the current "no row claims it" wording (roles/java.rs:53-59, control_flow_tests.rs:427-430) false once R09 lands. The paragraph should say "not the Switch row's; cognitive's structural projection lives as a named fn beside the table", and the two test/doc sites need correcting at the same time. One author, after R09.

   Batch: fix/batch-2026-10-31 (#1744 and #1745 first, as their own run, so R09 can rebase onto the landed drivers and drop its /var/tmp copy).
7. dekobon commented on Nov 1, 2026

   

   dekobon

   on Nov 1, 2026

   OwnerAuthor

   More actions

   Pointer update: #1744 landed the step this issue carries for it on branch fix/issue-1744 (pending merge; commits 2783e686 and 603e902e, output submodule c51ca7fa). The checklist line above now says so; ir_check.sh and snapdiff.py are recorded there as one-off and not landed.
8. dekobon commented on Nov 1, 2026

   

   dekobon

   on Nov 1, 2026

   OwnerAuthor

   More actions

   Follow-ups from R09 (#1699)

   Recorded here because they are DSL or close-out questions, not R09's migration (which is zero-drift by contract). Each came out of a CP3 review of refactor/1699-doc-roles.

   1. Per-role emission profile (architecture review). #![allow(dead_code)] in roles/mod.rs cannot narrow after R09: with it off, 58 items the macro emits uniformly on every row are unread — canonical, the pays_* family on owner-only roles, every predicate of an absent row. The @named table already maps role → module; extend it to role → shape (recognition for the R06/R09 roles, paying for R04/R05's, control for R07's) so a recognition row emits OWNERS, EXCLUDED, TEXT_GATED, owns, owns_kind, lists_kind and nothing else. A consumer reaching for comment::pays_at_leaf then fails to compile, which is the stronger property.
   2. CHAIN_GATED / child_gated_by (correctness + architecture reviews). The byte-less forwards (impl_doc_comment_from_row!, impl_doc_run_directive_from_row! with Ancestors::known(&[]); holds_doc_in_body_kind! and doxygen_entry with &[]) assert !TEXT_GATED at build time but nothing pins "no gate reads the chain": a future chain-reading DocComment gate answers false silently on nested comments, caught only by a fixture that nests one. gated_by cannot distinguish a gate reading the node's own subtree (Rust's has_doc_marker_child, has_body, field_declares, first_declarator_binds_a_name) from one reading ancestors. A third keyword child_gated_by for the former, with CHAIN_GATED = any gated_by, lets the forwards assert !CHAIN_GATED and documents on every row which gates are O(1) on the node.
   3. not_under [K…] owner attribute (architecture review). Three R09 gates are the same parent test by hand — python.rs::under_case_clause (pre-existing), go.rs::interface_element_opens_with_a_name's parent_has_kind(InterfaceType), js_family.rs::under_enum_body — and tsc's export / ambient veto is its complement in the Checker. under exists only on payers.
   4. preproc_conditional reads five kind() strings per call (CP2 cost audit, cleanup review). The C-family rows now carry every conditional alias id, so lang_helpers::doxygen::preproc_conditional could be a per-dialect lists_kind-style read; conditional_declares also calls it twice per child on the miss path. Constant-factor only (conditionals_read is memoised and equals base).
   5. No Parameter row. checker/{typescript,tsx}.rs's after_code parameter subset is a hand list under allow(DocTarget) because no row names a parameter kind; decide whether one belongs in the DSL or the bypass stays.
   6. HoldsDocInBody is a residual by name only. The row's definition ("beyond the callables and closures", roles/mod.rs) is policy-flavoured for a crate that names no policy; consider HoldsDocBeyondCallables, or declaring the full set and letting the Checker subtract.
   7. Test-file house style (cleanup review, deferred as consistent with siblings): the 12-copy #[cfg(any(…))] union per shared helper in each *_tests file; check's O(nodes × |expected|) selector; the [bool; N] Answers shape that prints unlabeled on mismatch; shared fixture consts for the cpp / mozcpp and javascript / mozjs parity pairs (space_tree_tests.rs does this; doc_tests/mod.rs copies).
   8. roles/mod.rs module doc now carries a "For the documentation epic" section and the reworded role-set table from R09; fold both into the doc rewrite this issue owns rather than restating them.
   9. Oracle driver homes (tier 1 per .claude/rules/temp-files.md): R09 reused R07's oracle.py / compare.py / recovery_oracle.py / perturb.py (one change: .h → objc in the recovery populations) and added a per-node doxygen-oracle crate that walks error-recovered trees with the real chain and prints Checker::is_doc_target per node. Four packages have now rebuilt the same drift harness out of tree; this close-out should decide whether it lands under utils/ with a *-test.py.
9. dekobon commented on Nov 4, 2026

   

   dekobon

   on Nov 4, 2026

   OwnerAuthor

   More actions

   R11 commit 1 landed on fix/issue-1701 (63d8ce57): decisions and evidence

   Decisions (confirmed with the maintainer 2026-11-04):

   • §2 Switch-body projection: (B) — the named is_switch_body_kind beside the table stays; the convention is documented (a consumer's structural projection of an owner's child lives as a named fn beside the table, with a row test pinning the parent/child relation). Since R09, Java's switch_block is claimed by HoldsDocInBody, so the old "no row claims it" wording is corrected; Groovy's was still coverage debt and is now excludes [SwitchBlock "…"] on its Switch row (coverage baseline −1, id 215).
   • Bypass-at-zero: the four Elixir entries (checker/elixir.rs::is_func_space, getter/elixir.rs::get_space_kind, restating Source and AnonymousFunction) are reclassified as three // role-bypass: allow exemptions under the lint's documented subset clause — the byte-less spelling lists the ungated owners of the FuncOpener / Container rows, the one answer it can give. Not a forward to opens_space: promotes_to_func_space_with_code calls the byte-less checker spelling per node before its single keyword read, and the forward would run both text-gated Call clauses (two extra elixir_call_keyword reads per call, calls + defs → 3·calls + defs in promotion_reads_each_calls_keyword_once). Reported as four findings reclassified as three documented subset exemptions, not as four duplicated lists removed. What the exemption no longer sees — a kind added inside either span — every_checker_forward_agrees_with_its_row_or_is_named does on every corpus node (perturbed below).
   • R09 item 9 (doxygen-oracle): not recoverable — not under /var/tmp/bca/, not in the refactor(cognitive,nexits): declare Branch, Switch, Loop, Catch, ElseIf and Exit #1697 records zip, not attached to refactor(checker): declare DocTarget, DocComment and Docstring rows ahead of the documentation epic #1699. Recorded as one-off; the claim it supported (7,675,318 node lines byte-identical) is on refactor(checker): declare DocTarget, DocComment and Docstring rows ahead of the documentation epic #1699 with its counts, and the derived suite asks the DocTarget rows on the corpus.
   • R09 items 1–7: filed as seven separate issues (next comment).

   Zero drift, commit 1. Structural: git diff origin/main -- src big-code-analysis-ast/src | rg '^[+-][^+-]' | rg -v '^[+-]\s*//' shows the excludes entry (emits a const) and test code only. Measured: utils/drift-oracle.py --bca <bca> --out <dir> --lang groovy --lang java --lang elixir --corpus tests/repositories --root cf=tests/repositories/big-code-analysis-output/control_flow --jobs 8 on a base bca from d96b6fca (build inputs 1213b4c0…) and a head bca from commit 1 (ab739ce4…), both built in the capture step; utils/drift-compare.py → 0 of 273 outputs differ (21 files, 13 views, 0 key problems).

   Gates on commit 1: role-census 0 pending; role-coverage OK 5,998 (the stale id-215 entry produced the expected note: before --update, not a failure); role-bypass OK, 0 baselined sites; role-parity-corpus pass; check-test-lang-gates OK; ast lib 643/643; root parity|roles|cognitive|elixir|groovy 759/759; .bca-baseline.toml unchanged after self-scan-write-baseline-headroom.

   Perturbations (utils/perturb.py, in-memory restore, old != new and count == 1 asserted; tree clean after):

   case	edit	verdict
   (a)	drop Groovy excludes [SwitchBlock …]	caught: groovy::is_else_if_is_the_row_and_the_switch_body_is_the_owners_child
   (a) by hand	same edit, make role-coverage	FAIL: Groovy: kind 215 switch_block … claimed by no role, excluded by none, and not baselined
   (b) by hand	strip allow(FuncOpener, Container) from the checker comment, make role-bypass	FAIL: 2 new bypass sites at checker/elixir.rs:31
   (b′)	| Elixir::Call added to the checker matches!	caught: promotion_reads_each_calls_keyword_once and every_checker_forward_agrees_with_its_row_or_is_named
   (b′)	| Elixir::DoBlock added	caught: every_checker_forward_agrees_with_its_row_or_is_named (unnamed disagreement)
   (c)	SwitchBlock removed from Java HoldsDocInBody	caught: java::switch_body_is_the_owners_child_and_not_the_switch_rows
10. dekobon commented on Nov 4, 2026

    

    dekobon

    on Nov 4, 2026

    OwnerAuthor

    More actions

    R11 implemented on fix/issue-1701 (five commits, c1506be3; awaiting the PR)

    commit	subject
    9614c253	refactor(roles): name the switch-body convention and exempt Elixir's byte-less space table
    65ee3d24	docs(ast): rewrite the roles module doc as the contributor reference (carries the roles/mod.rs loc.sloc baseline refresh, 1789 → 2014)
    588998cf	docs(rules): bring the dispatch, macro-comment and AGENTS rules onto the role tables
    19fd630e	docs(book): point the metrics and new-language pages at the role rows
    c1506be3	docs(lessons): sharing the rule left the list at every caller; sharing the list leaves the judgement (lesson 115)

    Checklist against the body: module doc ✓ (roles/mod.rs:1-540, nine sections, "What the mechanism cannot see" is the one list); grammar-dispatch.md §1/§4/§5/§7 ✓, macro-comments.md row-comment table ✓, AGENTS.md layout bullet + gate paragraph ✓; lesson 115 ✓; book metrics.md + new-language.md ✓ (README.md untouched, nothing there names roles); CHANGELOG ✓ (commit 1 carries the code change's entry per #1690 §9, commit 4 the closing entry); pays [] mode ✓ (#1745, verified: mode=split emitted by both readers; the doc names the three modes and quotes no count); Switch-body decision ✓ ((B), recorded in commit 1 and the previous comment); drift drivers ✓ (#1744).

    Reviews (all findings applied and fixed into their commits): plan-stage @sol-review (3), @security (3), @astra-review (7); implementation @astra-review ×3 (pass 1 on commit 1 + the seven issue bodies: nothing in the commit, six corrections to the bodies; pass 2 on the docs: 7, all precision of claims against source; pass 3 on the final tree: one number), a doc-author fresh-eyes prose review (25), an under-reviewed-areas + architecture audit (6, incl. the Groovydoc asymmetry now stated on roles/groovy.rs:155-170 and #1793 re-sized to 53 references in 17 files). simplify-rust / rust-optimize / performance audit: n/a — the Rust diff is one excludes entry (a const) and two test assertions.

    Gates on c1506be3: make pre-commit → BCA_GATE: pass (gate=pre-commit), 0 gpg failed to sign; make check-test-lang-gates-compare COMPARE_REF=origin/main OK; make role-parity-corpus pass (1,861 files, 83 recovered); role-census 0 pending / role-coverage 5,998 / role-bypass 0; make self-scan-baseline-fresh 0/0/0/0; RUSTDOCFLAGS=-D warnings cargo doc clean; rumdl clean on the seven .md files; ast lib 645/645.

    Follow-ups filed outside the series: #1792 (per-role emission profile), #1793 (child_gated_by / CHAIN_GATED; sized at 53 references in 17 files), #1794 (owner under / not_under), #1795 (preproc_conditional by id), #1796 (Parameter row — recommended decided-(b), awaiting the maintainer), #1797 (HoldsDocInBody name), #1798 (role test-file house style). R09's doxygen-oracle crate is recorded as one-off (not recoverable; the claim it supported is on #1699 with counts).

    The completion comment for #1690 follows the merge.