Repository navigation
Conversation
This was referenced Oct 19, 2026
eeee2345
added a commit
that referenced
this pull request
Oct 19, 2026
Eight conditions in seven rules took time quadratic in a long run of one character, in the TypeScript engine and under Python re alike. Through evaluate(), with 40,000 such characters in the fields it reads, each of the seven rules took 1.3 to 5.7 seconds (V8, this laptop, median of 3), and the engine evaluates fields up to 100,000 characters; a synchronous regular expression cannot be cut short, so a tool output built that way stalls the guard. Each condition was listed in data/redos-baseline.json, which lets a known slow condition pass the ReDoS gate, so CI stayed green; before #696 the gate's inputs, 1,500 to 3,000 characters, were too short to see them. None is among the conditions open PR #704 rewrites, and #708 removes ATR-2026-00290#3. Found by timing every condition the engine compiles (3,458, on main with #704 and #708 applied) on runs of 30 characters at 5,000 and 20,000: in V8 these eight took 0.2 to 1.6 s at 20,000, and in V8 and Python re each doubling of the run costs them about four times as much. ATR-2026-00233#1 is quadratic too, but the rule is deprecated and runs in no lane; three others that looked superlinear at that size (00419#1, 02681#1, 01771#3) grow linearly from 10,000 to 80,000 characters. The rewrites, each a shape #704 names: - ATR-2026-00265#2 and 01612#1: (?:^|\n)\s* rescanned every following newline from every newline. The whitespace after the line start is now [^\S\n]*, and 00265's \s*\n\s* is [^\S\n]*\n\s*. - ATR-2026-01865#0: \[+ and \]+ are \[ and \]; a search that starts at the last bracket of a run finds the same matches. - ATR-2026-00066#1: '?\s*; is ;, since the optional quote and spaces before the semicolon add nothing to a search. - ATR-2026-00066#3: \{\{.*\}\} starts at the first brace of a run, (?:^|[^{])\{\{.*\}\}. - ATR-2026-00426#1: the link text [^\]]* excludes [ as well, so a match starts at the last [ before the ]. - ATR-2026-01756#1: .{15,} and then [.!?]?\s+ split a trailing whitespace run. It is now .{15}(?:.*\S)?, the first 15 characters and then up to the last non-space, and \s*\.?\s*$ is \s*(?:\.\s*)?$. - ATR-2026-02261#0: the captured identifier ([A-Za-z_$][\w$]*) could start inside another identifier, at every character of it. It now starts after a character that cannot continue one, (?:^|[^\w$]). Two behaviour changes, both deliberate: - ATR-2026-02261#0 narrows. The back-reference \1 can no longer be the tail of the assigned name, so "xtoken = os.environ[...]" followed by "b64encode(token)", two different variables, no longer matches. In the engine (JavaScript, no u flag) the character before the name may not be [A-Za-z0-9_$]; Python re's \w also counts other letters. #704 left this condition out because shortening the capture changes what \1 can equal; this is that change, made on purpose. - ATR-2026-00426 sets suppress_in_code_blocks, and the engine suppresses by where the first match starts. That is now the link's own [ rather than the first of an earlier run with no ] in between, so when a code span or fence lies between the two the decision follows the link: a stray [ inside code no longer hides a link outside it, and a link inside code is no longer reported because of a [ outside it. Checked old against new, in Python re (the gate's flags) and in the engine's compiled RegExp: 0 differences for the other six over 32,555 corpus and test-case strings (the benign measurement corpus, research mentions, PINT, the dual-use and test corpora, evasion payloads, every rule's own cases), over 40,000 strings per condition drawn from each pattern's tokens and mutated from its rule's cases, and over 40,000 more for each of 00265#2, 01612#1 and 01756#1 built from its grammar with whitespace that crosses lines (7,808, 13,630 and 6,100 match in Python; 00265#2 matches none of the corpus, of its rule's cases or of its token fuzz). 00426#1 is the same for a search; through evaluate() it differs on none of those strings, and on 3,345 of 20,000 built to put code spans and stray brackets around links, every one with both patterns matching and their first matches starting at different places. 02261#0: 0 in the corpus, 79 of 40,000 fuzz strings old-only, each with \1 starting right after an identifier character. After: at 40,000 characters each condition takes at most 0.4 ms in V8 (it took 1.4 to 4.8 s) and at most 4 ms in Python (1.2 to 74 s), and each rule takes at most 1 ms through evaluate(). data/redos-baseline.json, measured with the gate's own measure() and confirm() the way --update-baseline records an entry: 00066#1, 00265#2, 01612#1, 01756#1 and 01865#0 leave it; 00426#1 (0.16 s) and 02261#0 (0.18 s) stay, under their new fingerprints, as under-budget entries the margin keeps; 00066#3 stays over the budget, re-recorded at 0.90 s (main records 0.77 s; the old and new patterns took the same 1.9 s raw side by side here) on tiled ${ input, which this rewrite does not address. Only these eight keys change: on main the file goes from 437 to 432 entries. #704 changes the same file at adjacent keys, so whichever lands second takes both sides of the conflict. Not addressed: input tiled from several characters. At 100,000 characters the new patterns still take 10.7 s (00066#3 on "${"), 4.0 s (00066#1 on "/*x") and 1.0 s (00426#1 on a tiled link prefix); the old patterns took the same at 20,000 and 40,000. tests/rules-linear-long-runs.test.ts checks that each rewritten condition still fires, then evaluates its rule alone on the 40,000-character run and requires it under 250 ms; on main all eight cases fail. data/benign-fp-measurement.json carries the seven rules' new detection fingerprints, each fp_count unchanged (remeasure-benign-fp.ts --base-ref origin/main; the four unrelated records it also refreshed are left out, as in #704, and so are its generated_at and commit, which no gate reads). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
@dc-tw thanks for following this through from #331, and sorry it sat for a week. I reviewed it against current main and the change itself holds up.
One thing is still needed. data/pyrit-digest-exclusions.json line 5 still lists ATR-2026-00290#3, and data/pyrit-digest.json carries the same entry under "excluded". CI does not run the export, but on the merged tree |
Contributor
|
@dc-tw one more thing, separate from this PR and from #752; nothing in either review depends on it. I have sent you a repository invitation with the triage role, for a reviewer trial through 26 November if you want it. What it gives you: you can be requested as a reviewer, and you can label, close and reopen issues and pull requests. It does not include write access, and merging stays with me. What I would ask: review pull requests in the areas you already know well (the quality gates, the converters, pyATR), and say plainly in each review what you checked and what you did not. Around 26 November I will send you a short written look back at those reviews, and we decide together whether to continue, widen it, or stop. No obligation either way. GitHub lets the invitation expire after a week if you would rather not. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The removed pattern references
\1without a capturing group. pyATR rejects it under bothreandregex; V8 accepts it only as legacy U+0001 semantics, so it never implemented the described repeated-word detection. Adding the apparent group would introduce a catastrophic, non-RE2 backreference. The remaining three conditions cover all five existing true positives.Output-side repetition detection should be handled as a separate rule with its own event shape and test cases.
Refs #331
Validation
npm run validate— 836/836npm run gate:generalization— PASSnpx tsx scripts/gate-action-eligibility.ts— PASStool_responseevent shape — 5/5 TP, 5/5 TN, 0 compile failurespython scripts/verify_rule_redos.py rules/context-exfiltration/ATR-2026-00290-divergence-repeat-word-training-extraction.yaml --base upstream/main— PASSnpm test -- --maxWorkers=4— 125 files passed, 2 skipped; 2,216 tests passed, 14 skippedThe Go/RE2 oracle was unavailable locally; the pinned oracle will run in CI.