Skip to content

Carry every field in the PyRIT digest, and name the safe scope - #511

Merged
eeee2345 merged 1 commit into
mainfrom
feat/pyrit-digest-all-fields
Aug 23, 2026
Merged

eeee2345 merged 1 commit into
mainfrom
feat/pyrit-digest-all-fields

Conversation

@eeee2345

@eeee2345 eeee2345 commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

The digest shipped only the conditions written against agent_output and content -- 366 of 778 rules. That was the right set for a text scorer and the wrong artifact: a consumer holding tool-call content had no way to reach the rules written for it, and the count read as a silent 53% cut of the rule set.

Every condition now carries its field, and default_fields names the scope a consumer that cannot route by field should select. 772 of 778 rules are present, against 366 before.

The scoping is not dropped, it moves to the consumer

scripts/measure-digest-parity.py runs the arms over the same rules from one checkout, so rule freshness cannot be mistaken for a digest effect. Controls run first and abort on failure -- an engine that silently loaded no rules flags nothing, and "0 versus 0" would otherwise read as perfect agreement.

benign conversation, n=600, seed=20260823
  engine (what microsoft/PyRIT#1893 shipped)  flagged 4 (0.7%)    --
  digest, default_fields                      flagged 4 (0.7%)    600/600 agree
  digest, every field, no severity floor      flagged 130 (21.7%) 474/600 agree
  digest, every field, medium floor           flagged 45 (7.5%)   559/600 agree

The every-field arm is reported at two floors because the gap between them is almost entirely one rule: ATR-2026-00099 is low severity and its conditions name tool_name and tool_args, so over English prose it matches "execute" and "delete" everywhere. Quoting either figure alone misstates the cost, so the harness prints both.

The harness is checked in rather than the number alone. A claim in a comment that nobody can re-run is how stale numbers survive:

PYTHONPATH=python python3 scripts/measure-digest-parity.py 600

No flat pattern map any more

The artifact no longer publishes a ready-made {name: pattern} dict. That shape is convenient to hand straight to a RegexScorer, and it is exactly the shape that produces the numbers above. Leaving it in the file was a loaded gun.

What fell out of emitting the other fields

The exclusion baseline grows from 13 to 16. The three additions were always broken, just never candidates: ATR-2026-00290#3 (backreference to a non-capturing group, tool_response) and ATR-2026-00442#0/#1 (variable-width lookbehind, user_input). The ratchet added in the previous commit caught them on the first run, which is what it is for.

The inert-rule test now gates on status alone, matching src/engine.ts and pyatr exactly. The extra maturity === 'deprecated' clause matched nothing today but would have made the digest quietly stricter than the engine it stands in for.

A condition that opts out of case folding is emitted with case_sensitive: true. A pattern arriving without an inline flag is therefore always attributable. No rule sets it today, so the test asserts zero unattributable patterns rather than tolerating a fraction.

Verification

Tests move with the shape and were checked by mutation -- each of these fails exactly one assertion:

mutation caught by
default_fields widened to every field the default scope must be a real narrowing
a condition's field blanked every condition must be routable
an inline (?i) removed with no opt-out marker no unattributable case behaviour
conditions_by_field miscounted by one the per-field counts must total condition_count
a severity set to urgent every severity must be rankable
an uncompilable pattern injected compiles under the flags the engine would pick

Local: typecheck, typecheck:scripts, build, validate, gate:rule-status, and the 8 digest tests all pass.

Prerequisite for the replacement scorer promised in microsoft/PyRIT#1893.

…e scope

The digest shipped only the conditions written against agent_output and content
-- 366 of 778 rules. That was the right set for a text scorer and the wrong
artifact: a consumer holding tool-call content had no way to get the rules
written for it, and the count looked like a silent 53% cut of the rule set.

Every condition is now emitted carrying its `field`, and `default_fields` names
the scope a consumer that cannot route by field should select. 772 of 778 rules
are present, against 366 before.

The scoping is not dropped, it moves to the consumer, and the reason is
measured. scripts/measure-digest-parity.py runs the arms over the same rules
from one checkout, so rule freshness cannot be mistaken for a digest effect:

  benign conversation, n=600
    engine (what microsoft/PyRIT#1893 shipped)  flagged 4 (0.7%)
    digest, default_fields                      flagged 4 (0.7%)    600/600 agree
    digest, every field, no severity floor      flagged 130 (21.7%) 474/600
    digest, every field, medium floor           flagged 45 (7.5%)   559/600

The every-field arm is reported at two floors because the gap between them is
almost entirely ATR-2026-00099, a low-severity rule whose conditions name
tool_name and tool_args -- over English prose "execute" and "delete" are
everywhere. Quoting either figure alone misstates the cost, so the harness
prints both and the docstring names the floor.

The harness is checked in rather than the number alone, because a claim in a
comment that nobody can re-run is how stale numbers survive.

There is deliberately no flat {name: pattern} map in the artifact any more. The
shape that is convenient to hand straight to a RegexScorer is exactly the shape
that produces those numbers, and leaving it there was a loaded gun.

Three further changes fall out of emitting the other fields:

- The exclusion baseline grows from 13 to 16. The three additions were always
  broken, just never candidates: ATR-2026-00290#3 (backreference to a
  non-capturing group, tool_response) and ATR-2026-00442#0/#1 (variable-width
  lookbehind, user_input). The ratchet caught them, which is what it is for.

- The inert-rule test now gates on status alone, matching src/engine.ts and
  pyatr exactly. The extra `maturity === 'deprecated'` clause matched nothing
  today but would have made the digest quietly stricter than the engine it
  stands in for.

- A condition that opts out of case folding is emitted with
  `case_sensitive: true`, so a pattern arriving without an inline flag is always
  attributable. No rule sets it today; the test asserts zero unattributable
  patterns rather than tolerating a fraction.

Tests move with the shape and were checked by mutation: widening default_fields
to every field, dropping a field tag, removing an inline flag, miscounting
conditions_by_field, and setting an unrankable severity each fail exactly one
assertion.
@eeee2345
eeee2345 force-pushed the feat/pyrit-digest-all-fields branch from 54d3e13 to 0858180 Compare August 23, 2026 20:12
@eeee2345
eeee2345 merged commit 0db19b2 into main Aug 23, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant