Merge branch 'main' into fix/krippendorff-unpairable-identical-ratings

Author: romanlutz

.pyrit_conf_example

@@ -111,6 +111,15 @@ operation: op_trash_panda
 # Applies only to the pyrit_backend server.
 max_concurrent_scenario_runs: 3
 
+# Live Reinitialization
+# ---------------------
+# Allows administrators to apply saved configuration changes without restarting
+# the backend. Enable only when the deployment has exactly one backend process
+# and one replica. Known worker and replica settings are checked as an additional
+# safeguard. Live apply rejects busy runtimes and requires a restart if
+# initialization fails after replacement begins.
+enable_live_reinitialization: false
+
 # Custom Initializer Registration (REST API)
 # -------------------------------------------
 # When true, the REST API accepts POST /api/initializers to register custom

build_scripts/validate_docs.py

@@ -102,6 +102,8 @@ def find_orphaned_files(toc_files: set[str], doc_root: Path) -> list[str]:
             continue
         # .py companion files for .ipynb are not orphaned
         if file_path.suffix == ".py":
+            if "supporting_assets" in file_path.parts:
+                continue
             notebook_version = file_path.with_suffix(".ipynb")
             if notebook_version.exists():
                 continue

doc/code/memory/8_seed_database.ipynb

@@ -286,7 +286,7 @@
     "\n",
     "For the most part these are user errors, but when you want to remove whole groups rather than individual seeds, use `remove_seed_groups_from_memory`. It applies the same filters, but removes every seed that shares a `prompt_group_id` with any match, so groups are never left partial. Note that it only affects seeds that belong to a group: a matching seed added individually (with no `prompt_group_id`) is skipped, so use `remove_seeds_from_memory` for those.\n",
     "\n",
-    "> **Note on deleting by `value`.** For the remove methods, the `value` filter defaults to full-string equality (`exact=True`), so `remove_seeds_from_memory(value=\"the\")` deletes only seeds whose value is exactly `\"the\"` — not everything containing it. This differs from `get_seeds`, which always matches `value` by substring. Pass `exact=False` to opt into substring deletion when you really want it. As a general rule, preview with the same filters via `get_seeds(...)` first and prefer a specific filter (such as `dataset_name` or `value_sha256`) for deletion.\n",
+    "> **Note on deleting by `value`.** For the remove methods, the `value` filter defaults to full-string equality (`exact=True`), so `remove_seeds_from_memory(value=\"the\")` deletes only seeds whose value is exactly `\"the\"` — not everything containing it. This differs from `get_seeds`, which always matches `value` by substring. The same applies to the list filters: `harm_categories`, `authors`, `groups` and `parameters` must match whole list elements (case-insensitive), so `remove_seeds_from_memory(harm_categories=[\"hate\"])` does not also remove seeds tagged `\"hate_speech\"`. Pass `exact=False` to opt into substring deletion when you really want it. As a general rule, preview with the same filters via `get_seeds(...)` first and prefer a specific filter (such as `dataset_name` or `value_sha256`) for deletion. Because `get_seeds` always matches by substring, the preview is a superset of what the remove methods delete: some previewed seeds will survive the deletion, which is expected.\n",
     "\n",
     "> **Note on file-backed seeds.** For `image_path`, `audio_path`, and `video_path` seeds, removal deletes only the database record; the serialized file on disk is left in place. Delete those files separately if they are no longer needed."
    ]

doc/code/memory/8_seed_database.py

@@ -145,6 +145,6 @@ def print_group(seed_group):
 #
 # For the most part these are user errors, but when you want to remove whole groups rather than individual seeds, use `remove_seed_groups_from_memory`. It applies the same filters, but removes every seed that shares a `prompt_group_id` with any match, so groups are never left partial. Note that it only affects seeds that belong to a group: a matching seed added individually (with no `prompt_group_id`) is skipped, so use `remove_seeds_from_memory` for those.
 #
-# > **Note on deleting by `value`.** For the remove methods, the `value` filter defaults to full-string equality (`exact=True`), so `remove_seeds_from_memory(value="the")` deletes only seeds whose value is exactly `"the"` — not everything containing it. This differs from `get_seeds`, which always matches `value` by substring. Pass `exact=False` to opt into substring deletion when you really want it. As a general rule, preview with the same filters via `get_seeds(...)` first and prefer a specific filter (such as `dataset_name` or `value_sha256`) for deletion.
+# > **Note on deleting by `value`.** For the remove methods, the `value` filter defaults to full-string equality (`exact=True`), so `remove_seeds_from_memory(value="the")` deletes only seeds whose value is exactly `"the"` — not everything containing it. This differs from `get_seeds`, which always matches `value` by substring. The same applies to the list filters: `harm_categories`, `authors`, `groups` and `parameters` must match whole list elements (case-insensitive), so `remove_seeds_from_memory(harm_categories=["hate"])` does not also remove seeds tagged `"hate_speech"`. Pass `exact=False` to opt into substring deletion when you really want it. As a general rule, preview with the same filters via `get_seeds(...)` first and prefer a specific filter (such as `dataset_name` or `value_sha256`) for deletion. Because `get_seeds` always matches by substring, the preview is a superset of what the remove methods delete: some previewed seeds will survive the deletion, which is expected.
 #
 # > **Note on file-backed seeds.** For `image_path`, `audio_path`, and `video_path` seeds, removal deletes only the database record; the serialized file on disk is left in place. Delete those files separately if they are no longer needed.

doc/code/scoring/3_combining_scorers.ipynb

@@ -58,6 +58,8 @@
     "        INV[\"TrueFalseInverterScorer<br/>negates one result\"]\n",
     "        CONV[\"create_conversation_scorer()<br/>scores concatenated history\"]\n",
     "        THRESH[\"FloatScaleThresholdScorer<br/>score ≥ threshold\"]\n",
+    "        FSFALL[\"FloatScaleFallbackScorer<br/>alternative float-scale judge\"]\n",
+    "        TFFALL[\"TrueFalseFallbackScorer<br/>alternative true/false judge\"]\n",
     "        CONV ~~~ THRESH\n",
     "    end\n",
     "\n",
@@ -70,20 +72,24 @@
     "    TF -->|\"1+ via scorers=\"| COMP\n",
     "    TF -->|\"1 via scorer=\"| INV\n",
     "    FS -->|\"1 via scorer=\"| THRESH\n",
+    "    FS -->|\"2 comparable single-result scorers\"| FSFALL\n",
+    "    TF -->|\"2 comparable single-result scorers\"| TFFALL\n",
     "    TF -->|\"1 scorer supporting text content\"| CONV\n",
     "    FS -->|\"1 scorer supporting text content\"| CONV\n",
     "\n",
     "    COMP -. is a .-> TFOUT\n",
     "    INV -. is a .-> TFOUT\n",
     "    THRESH -. is a .-> TFOUT\n",
+    "    FSFALL -. is a .-> FSOUT\n",
+    "    TFFALL -. is a .-> TFOUT\n",
     "    CONV -. \"for true/false input\" .-> TFOUT\n",
     "    CONV -. \"for float-scale input\" .-> FSOUT\n",
     "\n",
     "    classDef input fill:#e8f0fe,stroke:#4285f4,color:#15233a;\n",
     "    classDef wrapper fill:#fff4e5,stroke:#f29900,color:#3d2600;\n",
     "    classDef output fill:#e6f4ea,stroke:#34a853,color:#17351f;\n",
     "    class TF,FS input;\n",
-    "    class COMP,INV,THRESH,CONV wrapper;\n",
+    "    class COMP,INV,THRESH,CONV,FSFALL,TFFALL wrapper;\n",
     "    class TFOUT,FSOUT output;\n",
     "```"
    ]
@@ -99,8 +105,11 @@
     "`TrueFalseCompositeScorer` requires at least one `TrueFalseScorer` and combines their\n",
     "single results with `AND`, `OR`, or `MAJORITY`; `TrueFalseInverterScorer` accepts one\n",
     "`TrueFalseScorer`. `FloatScaleThresholdScorer` is the cross-kind adapter: it accepts one\n",
-    "`FloatScaleScorer` and produces a `TrueFalseScorer`. These generic wrappers forward the\n",
-    "same `Scorable` to their children, so each child must support that evidence kind.\n",
+    "`FloatScaleScorer` and produces a `TrueFalseScorer`. `FloatScaleFallbackScorer` accepts two\n",
+    "`FloatScaleScorer`s and produces a `FloatScaleScorer`. `TrueFalseFallbackScorer` does the\n",
+    "same for two `TrueFalseScorer`s. Each fallback wrapper tries the primary first and calls\n",
+    "the fallback only when the primary returns an undetermined score. These wrappers forward the same\n",
+    "`Scorable` to their children, so each child must support that evidence kind.\n",
     "\n",
     "`create_conversation_scorer()` accepts a true/false or float-scale scorer that supports\n",
     "text `ContentScorable` evidence. It returns a dynamic wrapper that remains the same scorer\n",
@@ -280,6 +289,58 @@
   {
    "cell_type": "markdown",
    "id": "11",
+   "metadata": {},
+   "source": [
+    "## Routing abstentions to a fallback scorer\n",
+    "\n",
+    "Both float-scale and true/false scorers can return `ScoreStatus.UNDETERMINED`.\n",
+    "A fallback wrapper evaluates the same evidence with a second scorer only when the\n",
+    "primary returns that status. A completed `False` or `0.0` is a valid judgment and does\n",
+    "not trigger fallback. This lets a fast primary handle most inputs without calling an\n",
+    "expensive second judge each time.\n",
+    "\n",
+    "Both children must belong to the same result family, support the same condition types,\n",
+    "and return exactly one score when applicable. The wrapper validates both children\n",
+    "before scoring and passes each its supported expectation. If fallback runs, the results\n",
+    "must refer to the same evidence and expectation; categories must match when both\n",
+    "results supply them. An unreadable judgment can have no category labels. Configure equivalent\n",
+    "criteria and, for float-scale scorers, comparable numeric meanings: matching types and\n",
+    "categories does not prove that two rubrics measure the same thing.\n",
+    "\n",
+    "With preconfigured scorers that meet these requirements:\n",
+    "\n",
+    "```python\n",
+    "from pyrit.score import FloatScaleFallbackScorer, TrueFalseFallbackScorer\n",
+    "\n",
+    "harm_scorer = FloatScaleFallbackScorer(\n",
+    "    scorer=primary_harm_scorer,\n",
+    "    fallback_scorer=secondary_harm_scorer,\n",
+    ")\n",
+    "objective_scorer = TrueFalseFallbackScorer(\n",
+    "    scorer=primary_objective_scorer,\n",
+    "    fallback_scorer=secondary_objective_scorer,\n",
+    ")\n",
+    "```\n",
+    "\n",
+    "A non-applicable primary (`[]`) returns `[]` without calling the fallback. A non-applicable\n",
+    "fallback leaves the primary's undetermined judgment in place. If both abstain, the result\n",
+    "remains undetermined. Exceptions propagate; they are not treated as abstentions.\n",
+    "\n",
+    "Multiple child results are rejected rather than paired by position or discarded.\n",
+    "Aggregate them explicitly before using fallback if a single aggregate is meaningful.\n",
+    "\n",
+    "Only the root score is persisted. The wrapper creates a new score without modifying\n",
+    "either child result and retains their observation links. Metadata records `resolved_by`\n",
+    "as `\"primary\"` or `\"fallback\"`. Child metadata keys are prefixed with `primary.` and\n",
+    "`fallback.`, so duplicate keys and nested fallback details are not overwritten.\n",
+    "When fallback is attempted, `primary_rationale` is retained and `fallback_status` records\n",
+    "`\"complete\"`, `\"undetermined\"`, or `\"not_applicable\"`. A fallback judgment also supplies\n",
+    "`fallback_rationale`; the combined rationale explains both attempts."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12",
    "metadata": {
     "lines_to_next_cell": 0
    },
@@ -299,7 +360,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "12",
+   "id": "13",
    "metadata": {},
    "outputs": [
     {
@@ -341,7 +402,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "13",
+   "id": "14",
    "metadata": {},
    "source": [
     "For a richer, real-world example, wrap a `SelfAskLikertScorer` with the\n",

doc/code/scoring/3_combining_scorers.py

@@ -48,6 +48,8 @@
 #         INV["TrueFalseInverterScorer<br/>negates one result"]
 #         CONV["create_conversation_scorer()<br/>scores concatenated history"]
 #         THRESH["FloatScaleThresholdScorer<br/>score ≥ threshold"]
+#         FSFALL["FloatScaleFallbackScorer<br/>alternative float-scale judge"]
+#         TFFALL["TrueFalseFallbackScorer<br/>alternative true/false judge"]
 #         CONV ~~~ THRESH
 #     end
 #
@@ -60,20 +62,24 @@
 #     TF -->|"1+ via scorers="| COMP
 #     TF -->|"1 via scorer="| INV
 #     FS -->|"1 via scorer="| THRESH
+#     FS -->|"2 comparable single-result scorers"| FSFALL
+#     TF -->|"2 comparable single-result scorers"| TFFALL
 #     TF -->|"1 scorer supporting text content"| CONV
 #     FS -->|"1 scorer supporting text content"| CONV
 #
 #     COMP -. is a .-> TFOUT
 #     INV -. is a .-> TFOUT
 #     THRESH -. is a .-> TFOUT
+#     FSFALL -. is a .-> FSOUT
+#     TFFALL -. is a .-> TFOUT
 #     CONV -. "for true/false input" .-> TFOUT
 #     CONV -. "for float-scale input" .-> FSOUT
 #
 #     classDef input fill:#e8f0fe,stroke:#4285f4,color:#15233a;
 #     classDef wrapper fill:#fff4e5,stroke:#f29900,color:#3d2600;
 #     classDef output fill:#e6f4ea,stroke:#34a853,color:#17351f;
 #     class TF,FS input;
-#     class COMP,INV,THRESH,CONV wrapper;
+#     class COMP,INV,THRESH,CONV,FSFALL,TFFALL wrapper;
 #     class TFOUT,FSOUT output;
 # ```
 
@@ -82,8 +88,11 @@
 # `TrueFalseCompositeScorer` requires at least one `TrueFalseScorer` and combines their
 # single results with `AND`, `OR`, or `MAJORITY`; `TrueFalseInverterScorer` accepts one
 # `TrueFalseScorer`. `FloatScaleThresholdScorer` is the cross-kind adapter: it accepts one
-# `FloatScaleScorer` and produces a `TrueFalseScorer`. These generic wrappers forward the
-# same `Scorable` to their children, so each child must support that evidence kind.
+# `FloatScaleScorer` and produces a `TrueFalseScorer`. `FloatScaleFallbackScorer` accepts two
+# `FloatScaleScorer`s and produces a `FloatScaleScorer`. `TrueFalseFallbackScorer` does the
+# same for two `TrueFalseScorer`s. Each fallback wrapper tries the primary first and calls
+# the fallback only when the primary returns an undetermined score. These wrappers forward the same
+# `Scorable` to their children, so each child must support that evidence kind.
 #
 # `create_conversation_scorer()` accepts a true/false or float-scale scorer that supports
 # text `ContentScorable` evidence. It returns a dynamic wrapper that remains the same scorer
@@ -163,6 +172,53 @@
 print(f"[threshold] near-copy   -> {near_copy.get_value()}")
 print(f"[threshold] independent -> {original.get_value()}")
 
+# %% [markdown]
+# ## Routing abstentions to a fallback scorer
+#
+# Both float-scale and true/false scorers can return `ScoreStatus.UNDETERMINED`.
+# A fallback wrapper evaluates the same evidence with a second scorer only when the
+# primary returns that status. A completed `False` or `0.0` is a valid judgment and does
+# not trigger fallback. This lets a fast primary handle most inputs without calling an
+# expensive second judge each time.
+#
+# Both children must belong to the same result family, support the same condition types,
+# and return exactly one score when applicable. The wrapper validates both children
+# before scoring and passes each its supported expectation. If fallback runs, the results
+# must refer to the same evidence and expectation; categories must match when both
+# results supply them. An unreadable judgment can have no category labels. Configure equivalent
+# criteria and, for float-scale scorers, comparable numeric meanings: matching types and
+# categories does not prove that two rubrics measure the same thing.
+#
+# With preconfigured scorers that meet these requirements:
+#
+# ```python
+# from pyrit.score import FloatScaleFallbackScorer, TrueFalseFallbackScorer
+#
+# harm_scorer = FloatScaleFallbackScorer(
+#     scorer=primary_harm_scorer,
+#     fallback_scorer=secondary_harm_scorer,
+# )
+# objective_scorer = TrueFalseFallbackScorer(
+#     scorer=primary_objective_scorer,
+#     fallback_scorer=secondary_objective_scorer,
+# )
+# ```
+#
+# A non-applicable primary (`[]`) returns `[]` without calling the fallback. A non-applicable
+# fallback leaves the primary's undetermined judgment in place. If both abstain, the result
+# remains undetermined. Exceptions propagate; they are not treated as abstentions.
+#
+# Multiple child results are rejected rather than paired by position or discarded.
+# Aggregate them explicitly before using fallback if a single aggregate is meaningful.
+#
+# Only the root score is persisted. The wrapper creates a new score without modifying
+# either child result and retains their observation links. Metadata records `resolved_by`
+# as `"primary"` or `"fallback"`. Child metadata keys are prefixed with `primary.` and
+# `fallback.`, so duplicate keys and nested fallback details are not overwritten.
+# When fallback is attempted, `primary_rationale` is retained and `fallback_status` records
+# `"complete"`, `"undetermined"`, or `"not_applicable"`. A fallback judgment also supplies
+# `fallback_rationale`; the combined rationale explains both attempts.
+
 # %% [markdown]
 # ## Scoring a whole conversation
 #

doc/code/scoring/4_scorer_metrics.ipynb

@@ -132,7 +132,19 @@
     "Three alpha values are reported:\n",
     "- **`krippendorff_alpha_humans`**: Agreement among human evaluators (baseline quality of labels)\n",
     "- **`krippendorff_alpha_model`**: Agreement across multiple model scoring trials (model consistency)\n",
-    "- **`krippendorff_alpha_combined`**: Overall agreement between humans and model"
+    "- **`krippendorff_alpha_combined`**: Overall agreement between humans and model\n",
+    "\n",
+    "### Error Split by Rater Agreement\n",
+    "\n",
+    "When a gold set has more than one human rater, the harm metrics also report the mean absolute error\n",
+    "separately for **unanimous** responses (every rater on the same side of `contested_threshold`, 0.5) and\n",
+    "**contested** responses (the raters split across it, so the gold label rests on a 2-1 vote rather than a\n",
+    "consensus). The aggregate MAE spends part of the scorer's error budget on the contested rows, so a scorer\n",
+    "can look strong overall while sitting near chance on exactly the responses humans found hard.\n",
+    "- **`mean_absolute_error_unanimous`** and **`num_unanimous_responses`**\n",
+    "- **`mean_absolute_error_contested`** and **`num_contested_responses`**\n",
+    "\n",
+    "These are `None` for single-rater gold sets, where there is no disagreement to measure."
    ]
   },
   {

doc/code/scoring/4_scorer_metrics.py

@@ -92,6 +92,18 @@
 # - **`krippendorff_alpha_humans`**: Agreement among human evaluators (baseline quality of labels)
 # - **`krippendorff_alpha_model`**: Agreement across multiple model scoring trials (model consistency)
 # - **`krippendorff_alpha_combined`**: Overall agreement between humans and model
+#
+# ### Error Split by Rater Agreement
+#
+# When a gold set has more than one human rater, the harm metrics also report the mean absolute error
+# separately for **unanimous** responses (every rater on the same side of `contested_threshold`, 0.5) and
+# **contested** responses (the raters split across it, so the gold label rests on a 2-1 vote rather than a
+# consensus). The aggregate MAE spends part of the scorer's error budget on the contested rows, so a scorer
+# can look strong overall while sitting near chance on exactly the responses humans found hard.
+# - **`mean_absolute_error_unanimous`** and **`num_unanimous_responses`**
+# - **`mean_absolute_error_contested`** and **`num_contested_responses`**
+#
+# These are `None` for single-rater gold sets, where there is no disagreement to measure.
 
 # %% [markdown]
 # ## Retrieving Scorer Metrics