FEAT: Adding Expectations to Attacks (Scorer Generalization Phase 4)

doc/code/executor/3_attack_configuration.ipynb

@@ -10,19 +10,48 @@
     "Every attack shares the same `execute_async` contract, so the inputs below work the same way no\n",
     "matter which executor you use.\n",
     "\n",
-    "`execute_async` accepts four standard arguments:\n",
+    "`execute_async` accepts these standard arguments:\n",
     "\n",
     "| Argument | Purpose |\n",
     "|---|---|\n",
-    "| `objective` | What you are trying to get the **objective target** (the system under test) to do. Drives scoring and multi-turn adversarial prompts. |\n",
+    "| `objective` | What you are trying to get the **objective target** to do. Drives attack prompts and supplies the default scoring context. |\n",
+    "| `expectation` | A per-execution `ScoringExpectation` for outcome scoring. Its objective may differ from the attack objective. |\n",
     "| `memory_labels` | A `dict[str, str]` tagged onto every prompt/response, so you can filter this run later in memory. |\n",
     "| `prepended_conversation` | A list of `Message`s to seed the conversation before the attack's own turns. This is also where the objective target's **system prompt** goes — `Message.from_system_prompt(...)` builds one (see below). |\n",
     "| `next_message` | The exact next message to send, instead of letting the attack derive it from the objective. Useful for multimodal or pre-built seeds. |\n",
     "\n",
     "Construction-time configuration objects — **adversarial**, **scoring**, and **converter** — are\n",
     "covered at the end and link out to their dedicated pages.\n",
     "\n",
-    "The examples here use `TextTarget`, which just records what would be sent — so they run instantly\n",
+    "## Scoring expectations\n",
+    "\n",
+    "`AttackScoringConfig` selects scorers and feedback policy, not execution criteria. For an attack\n",
+    "configured with an outcome scorer:\n",
+    "\n",
+    "```python\n",
+    "from pyrit.models import ScoringExpectation\n",
+    "\n",
+    "await attack.execute_async(\n",
+    "    objective=\"Identify who wrote Pride and Prejudice\",\n",
+    "    expectation=ScoringExpectation(objective=\"The answer identifies Jane Austen\"),\n",
+    ")\n",
+    "```\n",
+    "\n",
+    "A missing scoring objective defaults to the attack objective; supplied conditions stay unchanged.\n",
+    "Objective and auxiliary scorers receive the full expectation. Refusal, on-topic, and simulated\n",
+    "preparation checks keep their own criteria. Seeds are the intended main authoring source;\n",
+    "the execution parameter is transport. New expectation-bearing seed types are not implemented yet.\n",
+    "\n",
+    "`executor.execute_attack_from_seed_groups_async(attack=attack, seed_groups=groups, expectation=shared)`\n",
+    "broadcasts one expectation. Use `field_overrides=[{\"expectation\": first}, {\"expectation\": second}]`\n",
+    "for row-specific criteria; the list must match the seed-group count. A row override replaces the\n",
+    "whole expectation, and `None` uses that execution's objective fallback.\n",
+    "\n",
+    "**Behavior change:** `RedTeamingAttack` and `ChunkedRequestAttack` now run configured auxiliary\n",
+    "scorers that were previously skipped. This can add scoring requests and cost; leave the auxiliary\n",
+    "list empty to avoid them. Auxiliary results do not change the attack's success decision.\n",
+    "\n",
+    "The executable examples below use `TextTarget`, which just records what would be sent — so they run instantly\n",
     "and need no credentials."
    ]
   },
@@ -74,7 +103,9 @@
     "## Memory labels\n",
     "\n",
     "`memory_labels` tag every prompt and response this run produces. They don't change what is sent;\n",
-    "they make the run easy to find and group later in memory (e.g. by operation or operator)."
+    "they make the run easy to find and group later in memory (e.g. by operation or operator).\n",
+    "`AtomicAttack.run_async(memory_labels=...)` merges labels with its constructor labels.\n",
+    "Call-time values replace only shared keys; stored defaults stay unchanged."
    ]
   },
   {

doc/code/executor/3_attack_configuration.py

@@ -15,19 +15,48 @@
 # Every attack shares the same `execute_async` contract, so the inputs below work the same way no
 # matter which executor you use.
 #
-# `execute_async` accepts four standard arguments:
+# `execute_async` accepts these standard arguments:
 #
 # | Argument | Purpose |
 # |---|---|
-# | `objective` | What you are trying to get the **objective target** (the system under test) to do. Drives scoring and multi-turn adversarial prompts. |
+# | `objective` | What you are trying to get the **objective target** to do. Drives attack prompts and supplies the default scoring context. |
+# | `expectation` | A per-execution `ScoringExpectation` for outcome scoring. Its objective may differ from the attack objective. |
 # | `memory_labels` | A `dict[str, str]` tagged onto every prompt/response, so you can filter this run later in memory. |
 # | `prepended_conversation` | A list of `Message`s to seed the conversation before the attack's own turns. This is also where the objective target's **system prompt** goes — `Message.from_system_prompt(...)` builds one (see below). |
 # | `next_message` | The exact next message to send, instead of letting the attack derive it from the objective. Useful for multimodal or pre-built seeds. |
 #
 # Construction-time configuration objects — **adversarial**, **scoring**, and **converter** — are
 # covered at the end and link out to their dedicated pages.
 #
-# The examples here use `TextTarget`, which just records what would be sent — so they run instantly
+# ## Scoring expectations
+#
+# `AttackScoringConfig` selects scorers and feedback policy, not execution criteria. For an attack
+# configured with an outcome scorer:
+#
+# ```python
+# from pyrit.models import ScoringExpectation
+#
+# await attack.execute_async(
+#     objective="Identify who wrote Pride and Prejudice",
+#     expectation=ScoringExpectation(objective="The answer identifies Jane Austen"),
+# )
+# ```
+#
+# A missing scoring objective defaults to the attack objective; supplied conditions stay unchanged.
+# Objective and auxiliary scorers receive the full expectation. Refusal, on-topic, and simulated
+# preparation checks keep their own criteria. Seeds are the intended main authoring source;
+# the execution parameter is transport. New expectation-bearing seed types are not implemented yet.
+#
+# `executor.execute_attack_from_seed_groups_async(attack=attack, seed_groups=groups, expectation=shared)`
+# broadcasts one expectation. Use `field_overrides=[{"expectation": first}, {"expectation": second}]`
+# for row-specific criteria; the list must match the seed-group count. A row override replaces the
+# whole expectation, and `None` uses that execution's objective fallback.
+#
+# **Behavior change:** `RedTeamingAttack` and `ChunkedRequestAttack` now run configured auxiliary
+# scorers that were previously skipped. This can add scoring requests and cost; leave the auxiliary
+# list empty to avoid them. Auxiliary results do not change the attack's success decision.
+#
+# The executable examples below use `TextTarget`, which just records what would be sent — so they run instantly
 # and need no credentials.
 
 # %%
@@ -52,6 +81,8 @@
 #
 # `memory_labels` tag every prompt and response this run produces. They don't change what is sent;
 # they make the run easy to find and group later in memory (e.g. by operation or operator).
+# `AtomicAttack.run_async(memory_labels=...)` merges labels with its constructor labels.
+# Call-time values replace only shared keys; stored defaults stay unchanged.
 
 # %%
 result = await attack.execute_async(  # type: ignore

doc/code/executor/4_compound.ipynb

@@ -28,11 +28,19 @@
     "\n",
     "| Policy | Stops when | Envelope outcome |\n",
     "|---|---|---|\n",
-    "| `FIRST_SUCCESS` *(default)* | a child succeeds (continues past errors/failures) | SUCCESS if any child did |\n",
+    "| `FIRST_SUCCESS` *(default)* | a child succeeds (continues past all other outcomes) | SUCCESS if any child did |\n",
     "| `FIRST_DECISIVE` | a child succeeds **or** errors | SUCCESS if any child did |\n",
     "| `STRICT_ALL` | the first non-success | SUCCESS only if **every** child did (pipeline) |\n",
     "| `EXHAUSTIVE` | never (runs all) | SUCCESS if any child did |\n",
-    "| `LAST_RESULT` | never (runs all) | inherits the last child's outcome |"
+    "| `LAST_RESULT` | never (runs all) | inherits the last child's outcome |\n",
+    "\n",
+    "**Outcome correction:** An undecided child no longer becomes FAILURE in the compound result.\n",
+    "Without success, the any-success policies report ERROR when every child errored, UNDETERMINED\n",
+    "when any child is undecided, and otherwise FAILURE. `STRICT_ALL` stops at the first non-success\n",
+    "and reports that child's outcome: ERROR, FAILURE, or UNDETERMINED. If all children succeed, it reports SUCCESS.\n",
+    "A supplied execution expectation passes to each child unchanged. Otherwise, each child uses\n",
+    "its own preparation inputs and objective fallback, not the compound's display objective.\n",
+    "Compound implementations declare `DELEGATES_SCORING = True`; each child validates its own criteria."
    ]
   },
   {

doc/code/executor/4_compound.py

@@ -33,11 +33,19 @@
 #
 # | Policy | Stops when | Envelope outcome |
 # |---|---|---|
-# | `FIRST_SUCCESS` *(default)* | a child succeeds (continues past errors/failures) | SUCCESS if any child did |
+# | `FIRST_SUCCESS` *(default)* | a child succeeds (continues past all other outcomes) | SUCCESS if any child did |
 # | `FIRST_DECISIVE` | a child succeeds **or** errors | SUCCESS if any child did |
 # | `STRICT_ALL` | the first non-success | SUCCESS only if **every** child did (pipeline) |
 # | `EXHAUSTIVE` | never (runs all) | SUCCESS if any child did |
 # | `LAST_RESULT` | never (runs all) | inherits the last child's outcome |
+#
+# **Outcome correction:** An undecided child no longer becomes FAILURE in the compound result.
+# Without success, the any-success policies report ERROR when every child errored, UNDETERMINED
+# when any child is undecided, and otherwise FAILURE. `STRICT_ALL` stops at the first non-success
+# and reports that child's outcome: ERROR, FAILURE, or UNDETERMINED. If all children succeed, it reports SUCCESS.
+# A supplied execution expectation passes to each child unchanged. Otherwise, each child uses
+# its own preparation inputs and objective fallback, not the compound's display objective.
+# Compound implementations declare `DELEGATES_SCORING = True`; each child validates its own criteria.
 
 # %%
 import os

doc/code/framework.md

@@ -200,6 +200,8 @@ If you are contributing to PyRIT, that work will most likely land in one of the
 - Any branching decision (e.g. the next thing(s) to do is based on a previous result) should be an attack/executor.
 - Executors should always make use of other component's responsibilities. An executor should always branch based on a scorer and NOT a direct response. (e.g. was this prompt blocked? is a scorer responsibility, not an executor responsibility)
 - Executors should use scoring and target capabilities implicitly. Executors should support multi-modal.
+- Seeds author goals and criteria; execution parameters carry an optional `ScoringExpectation`
+  beside the attack objective. Attacks forward its conditions; scorers interpret them.
 - Compound attacks are possible, combining different attacks in different ways.
 - **Does not own**: packaging the attack. Those are passed in as configuration by the **attack technique**, not assembled here:
   - prepended / system prompts, role-play framing, the converter stack, or dataset selection (e.g. if an executor assembles its own prompt scaffolding for a simulated conversation, that is attack-technique work bleeding into the executor)

doc/code/scoring/0_scoring.ipynb

@@ -256,6 +256,16 @@
     "Cleanup and the reference removal share one transaction; shared observations remain available.\n",
     "Bulk SQL deletes do not use this ORM cleanup path.\n",
     "\n",
+    "Response helpers accept `expectation=`; their bare `objective=` input is deprecated until 2.0.\n",
+    "Objective and auxiliary scorers receive the complete expectation, with condition routing checked\n",
+    "across the group. Each scorer root keeps its own score/observation persistence boundary.\n",
+    "Direct scorers check required and duplicate criteria but ignore condition types they do not use.\n",
+    "Empty conditions retain legacy objective-only behavior and skip required-condition checks.\n",
+    "Data-bearing required conditions will need explicit validation before their scorer types are added.\n",
+    "Use a group helper, even with one scorer, when every condition must have a consumer.\n",
+    "`Scorer.score_with_scorers_async` accepts optional `scorer_roles`, one per scorer, for execution\n",
+    "context. Its result lists follow scorer input order, including empty lists.\n",
+    "\n",
     "Scoring APIs return `list[Score]`. An empty list means that the scorer does not apply to the\n",
     "evidence, such as a message with no supported role or data type. A non-empty list contains\n",
     "completed or undetermined scores.\n",

doc/code/scoring/0_scoring.py

@@ -158,6 +158,16 @@
 # Cleanup and the reference removal share one transaction; shared observations remain available.
 # Bulk SQL deletes do not use this ORM cleanup path.
 #
+# Response helpers accept `expectation=`; their bare `objective=` input is deprecated until 2.0.
+# Objective and auxiliary scorers receive the complete expectation, with condition routing checked
+# across the group. Each scorer root keeps its own score/observation persistence boundary.
+# Direct scorers check required and duplicate criteria but ignore condition types they do not use.
+# Empty conditions retain legacy objective-only behavior and skip required-condition checks.
+# Data-bearing required conditions will need explicit validation before their scorer types are added.
+# Use a group helper, even with one scorer, when every condition must have a consumer.
+# `Scorer.score_with_scorers_async` accepts optional `scorer_roles`, one per scorer, for execution
+# context. Its result lists follow scorer input order, including empty lists.
+#
 # Scoring APIs return `list[Score]`. An empty list means that the scorer does not apply to the
 # evidence, such as a message with no supported role or data type. A non-empty list contains
 # completed or undetermined scores.

pyrit/exceptions/__init__.py

@@ -34,6 +34,7 @@
         ExecutionContextManager,
         clear_execution_context,
         execution_context,
+        get_exception_execution_context,
         get_execution_context,
         set_execution_context,
     )
@@ -55,6 +56,7 @@
     "ExecutionContext": "pyrit.exceptions.exception_context",
     "ExecutionContextManager": "pyrit.exceptions.exception_context",
     "ExperimentalWarning": "pyrit.exceptions.exception_classes",
+    "get_exception_execution_context": "pyrit.exceptions.exception_context",
     "get_execution_context": "pyrit.exceptions.exception_context",
     "get_retry_collector": "pyrit.exceptions.retry_collector",
     "get_retry_max_num_attempts": "pyrit.exceptions.exception_classes",

pyrit/exceptions/exception_context.py

@@ -149,6 +149,27 @@ def get_execution_context() -> ExecutionContext | None:
     return _execution_context.get()
 
 
+def get_exception_execution_context(error: BaseException) -> ExecutionContext | None:
+    """
+    Find the component context attached to an exception or its exception chain.
+
+    Args:
+        error (BaseException): The failure, possibly raised in a child task.
+
+    Returns:
+        ExecutionContext | None: The recorded failure context, if present.
+    """
+    current: BaseException | None = error
+    visited: set[int] = set()
+    while current is not None and id(current) not in visited:
+        visited.add(id(current))
+        context = getattr(current, "_pyrit_execution_context", None)
+        if isinstance(context, ExecutionContext):
+            return context
+        current = current.__cause__ or (None if current.__suppress_context__ else current.__context__)
+    return None
+
+
 def set_execution_context(context: ExecutionContext) -> None:
     """
     Set the current execution context.
@@ -174,7 +195,8 @@ class ExecutionContextManager:
     execution context when entering and exiting a code block.
 
     On successful exit, the context is restored to its previous value.
-    On exception, the context is preserved so exception handlers can access it.
+    On exception, the context is preserved and attached to the failure so handlers
+    can access it even across task boundaries.
     """
 
     context: ExecutionContext
@@ -207,6 +229,9 @@ def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
         if exc_type is None:
             # No exception - restore previous context
             _execution_context.reset(self._token)
+        elif isinstance(exc_val, BaseException) and get_exception_execution_context(exc_val) is None:
+            # Keep the innermost recorded context, including when a caller wraps the failure.
+            exc_val.__dict__["_pyrit_execution_context"] = self.context
         # On exception, leave context in place for exception handlers to read