\u{...} is JavaScript-only, so ten rules never compiled outside the TS engine - #509
Merged
Merged
Conversation
… could not compile ten rules
Sixteen regex conditions across ten rules used \u{XXXXX}. That syntax needs the
RegExp u flag and exists only in JavaScript: Python re wants \U000E0000, Go wants
\x{E0000}, and neither accepts the JS form. Any consumer outside the TypeScript
engine simply dropped those rules.
Measured on this tree: of 3,353 regex conditions, 28 fail to compile under Python
re. Sixteen of them are this. Rewriting them as literal characters -- the one
spelling all three engines accept -- takes the affected rule count from 15 to 5.
What remains is variable-width lookbehind (11 conditions, 4 rules) and one
backreference, which are different problems.
A literal astral character needs the u flag too, or JS reads a surrogate pair and
a class range becomes a SyntaxError. So the flag decision moves from "does the
pattern contain \u{" to "does the pattern need u", in engine.ts and in
tests/validate-rules.ts, which had its own copy of that test.
Turning u on where it was previously off is stricter about escapes, and it found
one: ATR-2026-00391 had \" inside a character class, which is a valid identity
escape without u and invalid with it. The backslash was never needed.
Behaviour on the JavaScript side is unchanged, and that is checked rather than
assumed: every benign and malicious skill sample evaluated at two event shapes,
896 rows of rule-id sets, diffed against origin/main -- zero differences.
Contributor
ATR Rule Quality ReportStatus: PASS Validation
Test Cases
Quality Checklist
Label: |
…nothing
The first pass changed the flag test in src/engine.ts and tests/validate-rules.ts
and looked done: typecheck, build, rule validation and RE2 portability all green
locally. CI then failed three gates, and the reason was that four more copies of
the same expression exist.
scripts/eval-generalization.ts was the damaging one. Still testing for the
literal string \u{, it stopped adding u once the patterns became literals, so
every astral class range threw, the throw was caught as no-match, and eight rules
were reported as no longer matching their own authored true positives. The rules
were fine. The gate had gone blind.
There is now one definition — needsUnicodeFlag, exported from src/engine.ts — and
five consumers import it: eval-generalization, fn-mine-llm,
verify-re2-equivalence, lib/visibility-verify, and validate-rules.
fn-mine-llm also instructed the model to emit \u{XXXX} for astral codepoints,
promising the engine would add the flag. Every rule mined under that prompt would
have reintroduced exactly this. It now asks for the literal character and says
why.
Measurement re-run for the eleven touched rules: fp_count identical on all
eleven, only the fingerprints moved. That is the shape a rewrite that preserves
behaviour should have, and it restores the evidence the eligibility gate reads,
so the rules keep the response tiers they had already earned.
Worth recording for the next person: the 896-row corpus A/B in the previous
commit showed zero differences and was not enough. The benign and malicious skill
samples contain none of the unicode these rules exist to match, so the diff was
empty for the wrong reason. gate:generalization caught it because it tests each
rule against the payload its own author wrote.
Contributor
ATR Rule Quality ReportStatus: PASS Validation
Test Cases
Quality Checklist
Label: |
Two things CI caught that local gates did not.
The needsUnicodeFlag import in verify-re2-equivalence.ts landed at line 240,
after a later occurrence of the word import, so tsc -p tsconfig.scripts.json
failed on an undefined name while npm run typecheck passed. Different tsconfig,
different file set — running only the one is not running the check.
data/re2-equivalence.json was stale, which is the point of that artifact: making
ten rules portable changed which rules diverge under RE2, and the committed
record still described the old set. Regenerated, and it moved in the intended
direction — 37 lines out, 23 in, four divergent rules remaining, all of them
lookbehind or backreference rather than \u{...}.
Contributor
ATR Rule Quality ReportStatus: PASS Validation
Test Cases
Quality Checklist
Label: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sixteen regex conditions across ten rules used
\u{XXXXX}. That syntax requiresthe RegExp
uflag and exists only in JavaScript. Pythonrewants\U000E0000, Go wants\x{E0000}, and neither accepts the JS form — so everyconsumer outside the TypeScript engine dropped those rules silently.
Measured on this tree: of 3,353 regex conditions, 28 fail to compile under Python
re, and sixteen of them are this. Rewriting them as literal characters — theone spelling all three engines accept — takes the affected rule count from 15 to
5. What is left is variable-width lookbehind (11 conditions across 4 rules) and a
single backreference to a non-capturing group, which are separate problems and
not addressed here.
A literal astral character needs the
uflag too: without it JavaScript reads asurrogate pair, and a class range written with literals is a SyntaxError rather
than a range. So the flag decision changes from "does the pattern contain
\u{"to "does the pattern need
u". That test lived in two places —src/engine.tsand
tests/validate-rules.ts, which had its own copy — and both are updated. Thevalidator copy is why the first attempt at this passed typecheck and build and
still failed rule validation.
Turning
uon where it was previously off is stricter about escapes, and itsurfaced one: ATR-2026-00391 carried
\"inside a character class, which is apermitted identity escape without
uand invalid with it. The backslash wasnever needed.
BEHAVIOUR ON THE JS SIDE IS UNCHANGED, AND CHECKED
Every benign and malicious skill sample evaluated at two event shapes, 896 rows
of sorted rule-id sets, diffed against
origin/main: zero differences. The probeaborts before emitting anything if its control fails, so an empty diff cannot
come from a harness that never ran.
This is the source fix promised in microsoft/PyRIT#1893 — the Python scorer work
needs these rules to compile, and until now six of them dropped entirely.