Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/code/framework.md
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,7 @@ The below talks about responsibilities of most modules in the PyRIT library
- This is where cross-run analysis belongs: e.g. "which attack performed best for this objective?", "how often did a technique succeed?", or "which responses match known content?".
- **Does not own**: live, in-attack decisions — any decision made *during* an attack is a scorer's job. Analytics only operates on stored results, after the fact.
- Today it includes `ConversationAnalytics` (inspecting conversation history), `analyze_results` / `AttackStats` (aggregating outcomes across techniques), and text-matching strategies (`ExactTextMatching`, `ApproximateTextMatching`).
- `compute_scenario_statistics` calculates scenario success statistics. It owns execution-unit identity (atomic attack, technique configuration, and seed group), latest-attempt selection, counts, denominators, and rounding. SDK callers, the GUI backend's run detail and progress views, and the console, JSON, and HTML reports all present its results (`ScenarioExecutionStatistics`, `ScenarioExecutionUnit`, and `ScenarioProgressCounts` in `pyrit.models`) instead of calculating their own. The one exception is the GUI run-history list, which aggregates the same statistics in SQL (`MemoryInterface._build_scenario_history_aggregate_statement`) so it can page over many runs; `tests/unit/analytics/test_scenario_statistics_parity.py` keeps the two implementations in agreement.
- Shared analytics contracts (filters, dimensions, typed values, reports, facets, result pages, and `AttackStats`) live in `pyrit.models.analytics`. They validate data without querying memory or calculating statistics. `AttackResultSelection` defines selection modes without changing existing callers.
- Filter-bound cursor and label-normalization helpers live in `pyrit.common.pagination`. The backend pagination module retains compatibility exports, including History's invalid-cursor first-page fallback.

Expand Down
2 changes: 2 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,8 @@ filterwarnings = [
# Keep the in-repo suite on the scorable/expectation contract. Tests that cover the
# shim itself opt back in with pytest.warns.
"error:Scorer\\.score_async:DeprecationWarning",
# Rates come from pyrit.analytics; tests that cover the deprecated method opt back in with pytest.warns.
"error:ScenarioResult\\.objective_achieved_rate:DeprecationWarning",
]

[tool.ty]
Expand Down
2 changes: 2 additions & 0 deletions pyrit/analytics/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,14 @@
if TYPE_CHECKING:
from pyrit.analytics.conversation_analytics import ConversationAnalytics
from pyrit.analytics.result_analysis import AttackStats, analyze_results, get_cached_results_for_technique
from pyrit.analytics.scenario_statistics import compute_scenario_statistics
from pyrit.analytics.text_matching import ApproximateTextMatching, ExactTextMatching, TextMatching

_LAZY_EXPORTS: dict[str, str | tuple[str, str | None]] = {
"analyze_results": "pyrit.analytics.result_analysis",
"ApproximateTextMatching": "pyrit.analytics.text_matching",
"AttackStats": "pyrit.analytics.result_analysis",
"compute_scenario_statistics": "pyrit.analytics.scenario_statistics",
"ConversationAnalytics": "pyrit.analytics.conversation_analytics",
"ExactTextMatching": "pyrit.analytics.text_matching",
"get_cached_results_for_technique": "pyrit.analytics.result_analysis",
Expand Down
Loading