Skip to content
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/code/framework.md
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,7 @@ The below talks about responsibilities of most modules in the PyRIT library
- This is where cross-run analysis belongs: e.g. "which attack performed best for this objective?", "how often did a technique succeed?", or "which responses match known content?".
- **Does not own**: live, in-attack decisions — any decision made *during* an attack is a scorer's job. Analytics only operates on stored results, after the fact.
- Today it includes `ConversationAnalytics` (inspecting conversation history), `analyze_results` / `AttackStats` (aggregating outcomes across techniques), and text-matching strategies (`ExactTextMatching`, `ApproximateTextMatching`).
- `compute_scenario_statistics` is the single calculation of scenario success statistics. It owns execution-unit identity (atomic attack, technique configuration, and seed group), latest-attempt selection, counts, denominators, and rounding. `ScenarioResult`'s SDK callers, the GUI backend, and the console, JSON, and HTML reports all present its results (`ScenarioExecutionStatistics`, `ScenarioExecutionUnit`, and `ScenarioProgressCounts` in `pyrit.models`) instead of calculating their own.
Comment thread
hannahwestra25 marked this conversation as resolved.
Outdated
- Shared analytics contracts (filters, dimensions, typed values, reports, facets, result pages, and `AttackStats`) live in `pyrit.models.analytics`. They validate data without querying memory or calculating statistics. `AttackResultSelection` defines selection modes without changing existing callers.
- Filter-bound cursor and label-normalization helpers live in `pyrit.common.pagination`. The backend pagination module retains compatibility exports, including History's invalid-cursor first-page fallback.

Expand Down
2 changes: 2 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,8 @@ filterwarnings = [
# Keep the in-repo suite on the scorable/expectation contract. Tests that cover the
# shim itself opt back in with pytest.warns.
"error:Scorer\\.score_async:DeprecationWarning",
# Rates come from pyrit.analytics; tests that cover the deprecated method opt back in with pytest.warns.
"error:ScenarioResult\\.objective_achieved_rate:DeprecationWarning",
]

[tool.ty]
Expand Down
2 changes: 2 additions & 0 deletions pyrit/analytics/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,14 @@
if TYPE_CHECKING:
from pyrit.analytics.conversation_analytics import ConversationAnalytics
from pyrit.analytics.result_analysis import AttackStats, analyze_results, get_cached_results_for_technique
from pyrit.analytics.scenario_statistics import compute_scenario_statistics
from pyrit.analytics.text_matching import ApproximateTextMatching, ExactTextMatching, TextMatching

_LAZY_EXPORTS: dict[str, str | tuple[str, str | None]] = {
"analyze_results": "pyrit.analytics.result_analysis",
"ApproximateTextMatching": "pyrit.analytics.text_matching",
"AttackStats": "pyrit.analytics.result_analysis",
"compute_scenario_statistics": "pyrit.analytics.scenario_statistics",
"ConversationAnalytics": "pyrit.analytics.conversation_analytics",
"ExactTextMatching": "pyrit.analytics.text_matching",
"get_cached_results_for_technique": "pyrit.analytics.result_analysis",
Expand Down
Loading
Loading