docs: re-measure #75 on today's base, and delimit the external-marker technique

[ghedwards]

Refs #75 — documentation only, no grammar or scanner change. The issue stays open and stays rejected; this records why, with fresh numbers and one new finding.

Why re-measure at all

#98 was parked on a cost that had expired: +591 states, measured before new_expression's arguments became required, and +14 after. That makes every other parked number worth re-taking rather than trusting.

#75's did not move.

route	STATE_COUNT	parser.c
base	5,315	19.5 MB
$.if_statement in the arrow body choice	10,704 (+101%)	39.3 MB
the same, gated on an external zero-width marker	10,758	39.4 MB
a purpose-built conditional whose branches are expressions, at 'member'	7,330 (+38%)	27.5 MB

The middle row is the finding

The external zero-width marker is the technique that made savecontent as an expression (#82) and the empty arrow body (#116) affordable — the scanner looks ahead one token where the grammar cannot. Here it buys nothing, and slightly less than nothing.

The reason is worth stating because it tells you when to reach for it: the cost is not ambiguity at the decision point. It is the if_statement subtree becoming reachable in expression position at all, which duplicates states across the whole expression hierarchy. A marker prunes a choice; it does not un-reach a rule.

Reach for the marker when the problem is one token of lookahead. It does not help when a rule category becomes reachable in a new context.

The third row is the same lesson from the other side: even a hand-written conditional that admits only expressions in its branches costs +38% and +8 MB, because it still mentions $.expression in a new position.

Verdict unchanged

For one corpus file — Lucee's LDEV1819/test2.cfm — none of the three is worth it. LIMITATIONS.md and docs/FAILING-PATTERNS.md now carry the table and the delimitation; the working tree is back on master's parser, with STATE_COUNT at 5,315, 346/346 tests and probes 50/56 with no drift.

🤖 Generated with Claude Code

https://claude.ai/code/session_0117EvdX3EgLgDevVNrWZ5hr

---

Generated by Claude Code