Carry every field in the PyRIT digest, and name the safe scope - #511
Merged
Merged
Conversation
…e scope
The digest shipped only the conditions written against agent_output and content
-- 366 of 778 rules. That was the right set for a text scorer and the wrong
artifact: a consumer holding tool-call content had no way to get the rules
written for it, and the count looked like a silent 53% cut of the rule set.
Every condition is now emitted carrying its `field`, and `default_fields` names
the scope a consumer that cannot route by field should select. 772 of 778 rules
are present, against 366 before.
The scoping is not dropped, it moves to the consumer, and the reason is
measured. scripts/measure-digest-parity.py runs the arms over the same rules
from one checkout, so rule freshness cannot be mistaken for a digest effect:
benign conversation, n=600
engine (what microsoft/PyRIT#1893 shipped) flagged 4 (0.7%)
digest, default_fields flagged 4 (0.7%) 600/600 agree
digest, every field, no severity floor flagged 130 (21.7%) 474/600
digest, every field, medium floor flagged 45 (7.5%) 559/600
The every-field arm is reported at two floors because the gap between them is
almost entirely ATR-2026-00099, a low-severity rule whose conditions name
tool_name and tool_args -- over English prose "execute" and "delete" are
everywhere. Quoting either figure alone misstates the cost, so the harness
prints both and the docstring names the floor.
The harness is checked in rather than the number alone, because a claim in a
comment that nobody can re-run is how stale numbers survive.
There is deliberately no flat {name: pattern} map in the artifact any more. The
shape that is convenient to hand straight to a RegexScorer is exactly the shape
that produces those numbers, and leaving it there was a loaded gun.
Three further changes fall out of emitting the other fields:
- The exclusion baseline grows from 13 to 16. The three additions were always
broken, just never candidates: ATR-2026-00290#3 (backreference to a
non-capturing group, tool_response) and ATR-2026-00442#0/#1 (variable-width
lookbehind, user_input). The ratchet caught them, which is what it is for.
- The inert-rule test now gates on status alone, matching src/engine.ts and
pyatr exactly. The extra `maturity === 'deprecated'` clause matched nothing
today but would have made the digest quietly stricter than the engine it
stands in for.
- A condition that opts out of case folding is emitted with
`case_sensitive: true`, so a pattern arriving without an inline flag is always
attributable. No rule sets it today; the test asserts zero unattributable
patterns rather than tolerating a fraction.
Tests move with the shape and were checked by mutation: widening default_fields
to every field, dropping a field tag, removing an inline flag, miscounting
conditions_by_field, and setting an unrankable severity each fail exactly one
assertion.
eeee2345
force-pushed
the
feat/pyrit-digest-all-fields
branch
from
August 23, 2026 20:12
54d3e13 to
0858180
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The digest shipped only the conditions written against
agent_outputandcontent-- 366 of 778 rules. That was the right set for a text scorer and the wrong artifact: a consumer holding tool-call content had no way to reach the rules written for it, and the count read as a silent 53% cut of the rule set.Every condition now carries its
field, anddefault_fieldsnames the scope a consumer that cannot route by field should select. 772 of 778 rules are present, against 366 before.The scoping is not dropped, it moves to the consumer
scripts/measure-digest-parity.pyruns the arms over the same rules from one checkout, so rule freshness cannot be mistaken for a digest effect. Controls run first and abort on failure -- an engine that silently loaded no rules flags nothing, and "0 versus 0" would otherwise read as perfect agreement.The every-field arm is reported at two floors because the gap between them is almost entirely one rule:
ATR-2026-00099is low severity and its conditions nametool_nameandtool_args, so over English prose it matches "execute" and "delete" everywhere. Quoting either figure alone misstates the cost, so the harness prints both.The harness is checked in rather than the number alone. A claim in a comment that nobody can re-run is how stale numbers survive:
No flat pattern map any more
The artifact no longer publishes a ready-made
{name: pattern}dict. That shape is convenient to hand straight to aRegexScorer, and it is exactly the shape that produces the numbers above. Leaving it in the file was a loaded gun.What fell out of emitting the other fields
The exclusion baseline grows from 13 to 16. The three additions were always broken, just never candidates:
ATR-2026-00290#3(backreference to a non-capturing group,tool_response) andATR-2026-00442#0/#1(variable-width lookbehind,user_input). The ratchet added in the previous commit caught them on the first run, which is what it is for.The inert-rule test now gates on
statusalone, matchingsrc/engine.tsandpyatrexactly. The extramaturity === 'deprecated'clause matched nothing today but would have made the digest quietly stricter than the engine it stands in for.A condition that opts out of case folding is emitted with
case_sensitive: true. A pattern arriving without an inline flag is therefore always attributable. No rule sets it today, so the test asserts zero unattributable patterns rather than tolerating a fraction.Verification
Tests move with the shape and were checked by mutation -- each of these fails exactly one assertion:
default_fieldswidened to every fieldfieldblanked(?i)removed with no opt-out markerconditions_by_fieldmiscounted by onecondition_counturgentLocal:
typecheck,typecheck:scripts,build,validate,gate:rule-status, and the 8 digest tests all pass.Prerequisite for the replacement scorer promised in microsoft/PyRIT#1893.