diff --git a/AGENTS.md b/AGENTS.md index 78880852..378f4845 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,6 +39,7 @@ strands-evals depends on **`strands-agents`** (the Python SDK) for all LLM inter - `strands.tools.decorator` (`DecoratedFunctionTool`, `FunctionToolMetadata`) and the `@tool` decorator from `strands` - `strands.types.content.Message` - `strands.types.exceptions` (`EventLoopException`, `ModelThrottledException`) + - `strands.Skill` (public, exported in `strands.__all__` from `strands.vended_plugins.skills.skill`), used by the skill extractors to read `SKILL.md` frontmatter - **Not used as SDK imports (do not add to the list without verifying a real import):** - `strands.types.traces` — this repo defines its own `strands_evals.types.trace` for session/span modeling; the SDK's trace types are not imported. - `strands.telemetry` — the evals repo has its own `strands_evals.telemetry` module. The only reference to `strands.telemetry` is the literal string `"strands.telemetry.tracer"` used as an OpenTelemetry scope name in `mappers/constants.py`. diff --git a/README.md b/README.md index 583da35c..6a669acf 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,7 @@ Strands Evaluation is a powerful framework for evaluating AI agents and LLM appl - **Multiple Evaluation Types**: Output evaluation, trajectory analysis, tool usage assessment, and interaction evaluation - **Multimodal Evaluation**: MLLM-as-a-Judge evaluators for image-to-text tasks with built-in rubrics +- **Skill Evaluation**: Assess which skills a skill-equipped agent selected and whether it followed their instructions - **Dynamic Simulators**: Multi-turn conversation simulation with realistic user behavior, goal-oriented interactions, and LLM-powered tool simulation with shared state - **LLM-as-a-Judge**: Built-in evaluators using language models for sophisticated assessment with structured scoring - **Trace-based Evaluation**: Analyze agent behavior through OpenTelemetry execution traces @@ -517,6 +518,39 @@ tool_parameter_evaluator = ToolParameterAccuracyEvaluator( ) ``` +### Skill Selection and Instruction Following + +A skill is an instruction file (usually `SKILL.md`) that the harness offers to the agent at +runtime; the agent decides which, if any, to load. Evaluate both halves of that behavior, +which skill the agent picked and whether it then followed the skill's steps: + +```python +from strands_evals.evaluators import ( + SkillInstructionFollowingEvaluator, + SkillInvoked, + SkillSelectionAccuracyEvaluator, +) + +# Was each invoked skill an appropriate pick? Binary, one result per invoked skill. +# A run that invoked nothing has no selection to judge and yields a not-applicable row. +selection_evaluator = SkillSelectionAccuracyEvaluator() + +# Did the agent follow the invoked skill's steps? Five-level rating, one result per +# invoked skill, grounded in per-step covered/partial/skipped evidence. +following_evaluator = SkillInstructionFollowingEvaluator() + +# Deterministic presence check, no model. +invoked_check = SkillInvoked(skill_name="pdf-processing") +``` + +Both judges read the trajectory only: pass a `Session` or a raw message list as +`actual_trajectory`. Skill signals are recognized for the Strands `AgentSkills` plugin, Claude +Code, Codex, Gemini CLI, OpenHands, and Google ADK, plus the generic case of an agent reading a +`SKILL.md` from disk. A harness whose skill calls match none of those yields empty results rather +than an error, so confirm `parse_available_skills(trajectory)` returns your skills before trusting +a score. The signals are recovered by `parse_available_skills` and `extract_selected_skills` from +`strands_evals.extractors`. + ### Multimodal Evaluation (MLLM-as-a-Judge) Evaluate multimodal agent responses involving images and text using MLLM-as-a-Judge: diff --git a/SKILL.md b/SKILL.md index 42b95ec2..16b66c8b 100644 --- a/SKILL.md +++ b/SKILL.md @@ -131,11 +131,12 @@ Pick the evaluator by scope: | Scope | Evaluators | | --- | --- | | Tool-level | `ToolSelectionAccuracyEvaluator`, `ToolParameterAccuracyEvaluator` | +| Skill-level (per invoked skill) | `SkillSelectionAccuracyEvaluator`, `SkillInstructionFollowingEvaluator`, `SkillInvoked` (deterministic) | | Trace-level (last turn) | `CorrectnessEvaluator`, `HelpfulnessEvaluator`, `FaithfulnessEvaluator`, `CoherenceEvaluator`, `ConcisenessEvaluator`, `ResponseRelevanceEvaluator`, `HarmfulnessEvaluator`, `RefusalEvaluator`, `StereotypingEvaluator`, `InstructionFollowingEvaluator` | | Session-level (full conversation) | `GoalSuccessRateEvaluator` | | Multi-agent interactions and handoffs | `InteractionsEvaluator` (output-based) | -Helpfulness uses a seven-level scale, 0.0 Not helpful to 1.0 Above and beyond. Correctness uses a three-level rubric in basic mode, or binary CORRECT/INCORRECT in reference mode when `expected_assertion` is set on the case. Conciseness uses three levels. Coherence uses five levels. Harmfulness, Refusal, Stereotyping, InstructionFollowing are binary. +Helpfulness uses a seven-level scale, 0.0 Not helpful to 1.0 Above and beyond. Correctness uses a three-level rubric in basic mode, or binary CORRECT/INCORRECT in reference mode when `expected_assertion` is set on the case. Conciseness uses three levels. Coherence uses five levels. Harmfulness, Refusal, Stereotyping, InstructionFollowing are binary. SkillInstructionFollowing uses a five-level scale, 0.0 Not Followed to 1.0 Fully Followed, and returns one result per invoked skill. SkillSelectionAccuracy is binary and also returns one result per invoked skill. For traces from external systems pick the matching mapper: - `CloudWatchSessionMapper` paired with `CloudWatchProvider` and `CloudWatchLogsParser` diff --git a/pyproject.toml b/pyproject.toml index aefd376e..4c838016 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -32,6 +32,7 @@ dependencies = [ "opentelemetry-instrumentation-threading>=0.51b0,<1.00b0", "boto3>=1.26.0", "tenacity>=8.0.0,<10.0.0", + "pyyaml>=6.0.0,<7.0.0", ] [tool.hatch.build.targets.wheel] diff --git a/src/strands_evals/cli/_entrypoint.py b/src/strands_evals/cli/_entrypoint.py index 4093f38e..5f0d7059 100644 --- a/src/strands_evals/cli/_entrypoint.py +++ b/src/strands_evals/cli/_entrypoint.py @@ -29,6 +29,8 @@ InstructionFollowingEvaluator, RefusalEvaluator, ResponseRelevanceEvaluator, + SkillInstructionFollowingEvaluator, + SkillSelectionAccuracyEvaluator, StereotypingEvaluator, ToolParameterAccuracyEvaluator, ToolSelectionAccuracyEvaluator, @@ -53,6 +55,10 @@ "instruction-following": InstructionFollowingEvaluator, "refusal": RefusalEvaluator, "response-relevance": ResponseRelevanceEvaluator, + # `skill-invoked` is absent for the same reason as `tool-called`: `SkillInvoked` + # requires a `skill_name`, so it belongs in an experiment file. + "skill-instruction-following": SkillInstructionFollowingEvaluator, + "skill-selection-accuracy": SkillSelectionAccuracyEvaluator, "stereotyping": StereotypingEvaluator, "tool-parameter-accuracy": ToolParameterAccuracyEvaluator, "tool-selection-accuracy": ToolSelectionAccuracyEvaluator, diff --git a/src/strands_evals/cli/commands/run.py b/src/strands_evals/cli/commands/run.py index b82bfce2..e0196a17 100644 --- a/src/strands_evals/cli/commands/run.py +++ b/src/strands_evals/cli/commands/run.py @@ -222,13 +222,21 @@ def _print_summary(report: EvaluationReport) -> None: `run_evaluations_async` returns a single flattened report whose `cases` rows are tagged with an `evaluator` key; regroup by that tag so the summary still reads `: P/T passed (avg)` per evaluator. + + Each average is computed the same way as `report.overall_score`, dropping cases that had + nothing to judge. Averaging their placeholder 0.0 instead would print a lower per-evaluator + number than the overall score computed from the very same rows. """ by_eval: dict[str, list[int]] = {} by_eval_scores: dict[str, list[float]] = {} for i, case in enumerate(report.cases): name = case.get("evaluator", "unknown") by_eval.setdefault(name, []).append(int(report.test_passes[i])) - by_eval_scores.setdefault(name, []).append(report.scores[i]) + outputs = report.detailed_results[i] if i < len(report.detailed_results) else [] + if EvaluationReport.is_applicable(outputs): + by_eval_scores.setdefault(name, []).append(report.scores[i]) + else: + by_eval_scores.setdefault(name, []) parts: list[str] = ["strands-evals run"] for name, passes in by_eval.items(): @@ -271,7 +279,10 @@ def _display_expanded( details: dict[str, Any] = {"name": case.get("name", f"Test {i + 1}")} if "evaluator" in case: details["evaluator"] = case["evaluator"] - details["score"] = f"{report.scores[i]:.2f}" + outputs = report.detailed_results[i] if i < len(report.detailed_results) else [] + # A case with nothing to judge is excluded from every average, so printing its + # placeholder 0.00 in the score column would read as the worst possible verdict. + details["score"] = f"{report.scores[i]:.2f}" if EvaluationReport.is_applicable(outputs) else "n/a" details["test_pass"] = report.test_passes[i] if i < len(report.test_passes) else False details["reason"] = report.reasons[i] if i < len(report.reasons) else "" details["input"] = EvaluationReport.format_input_for_display(case.get("input")) diff --git a/src/strands_evals/evaluators/__init__.py b/src/strands_evals/evaluators/__init__.py index c163ec8e..870509ae 100644 --- a/src/strands_evals/evaluators/__init__.py +++ b/src/strands_evals/evaluators/__init__.py @@ -1,7 +1,7 @@ from .coherence_evaluator import CoherenceEvaluator from .conciseness_evaluator import ConcisenessEvaluator from .correctness_evaluator import CorrectnessEvaluator -from .deterministic import Contains, Equals, StartsWith, StateEquals, ToolCalled +from .deterministic import Contains, Equals, SkillInvoked, StartsWith, StateEquals, ToolCalled from .evaluator import Evaluator from .faithfulness_evaluator import FaithfulnessEvaluator from .goal_success_rate_evaluator import GoalSuccessRateEvaluator @@ -17,12 +17,17 @@ from .output_evaluator import OutputEvaluator from .refusal_evaluator import RefusalEvaluator from .response_relevance_evaluator import ResponseRelevanceEvaluator +from .skill_instruction_following_evaluator import SkillInstructionFollowingEvaluator +from .skill_selection_accuracy_evaluator import SkillSelectionAccuracyEvaluator from .stereotyping_evaluator import StereotypingEvaluator from .tool_parameter_accuracy_evaluator import ToolParameterAccuracyEvaluator from .tool_selection_accuracy_evaluator import ToolSelectionAccuracyEvaluator from .trajectory_evaluator import TrajectoryEvaluator __all__ = [ + "SkillSelectionAccuracyEvaluator", + "SkillInstructionFollowingEvaluator", + "SkillInvoked", "Evaluator", "OutputEvaluator", "MultimodalOutputEvaluator", diff --git a/src/strands_evals/evaluators/deterministic/__init__.py b/src/strands_evals/evaluators/deterministic/__init__.py index 66cba320..8a44d549 100644 --- a/src/strands_evals/evaluators/deterministic/__init__.py +++ b/src/strands_evals/evaluators/deterministic/__init__.py @@ -1,8 +1,10 @@ from .environment_state import StateEquals from .output import Contains, Equals, StartsWith +from .skill_invoked import SkillInvoked from .trajectory import ToolCalled __all__ = [ + "SkillInvoked", "Contains", "Equals", "StartsWith", diff --git a/src/strands_evals/evaluators/deterministic/skill_invoked.py b/src/strands_evals/evaluators/deterministic/skill_invoked.py new file mode 100644 index 00000000..8627e4ca --- /dev/null +++ b/src/strands_evals/evaluators/deterministic/skill_invoked.py @@ -0,0 +1,50 @@ +from ...extractors.skills import extract_skill_load_events +from ...types.evaluation import EvaluationData, EvaluationOutput, InputT, OutputT +from ..evaluator import Evaluator + + +class SkillInvoked(Evaluator[InputT, OutputT]): + """Checks if a specific skill was invoked in the trajectory.""" + + def __init__(self, skill_name: str, name: str | None = None): + super().__init__(name=name) + self.skill_name = skill_name + + def evaluate(self, evaluation_case: EvaluationData[InputT, OutputT]) -> list[EvaluationOutput]: + trajectory = evaluation_case.actual_trajectory + if trajectory is None: + return [EvaluationOutput(score=0.0, test_pass=False, reason="no trajectory provided")] + + # Read the individual attempts rather than the per-skill summary, so this check applies its + # own definition of "invoked": a refused load does not count, because the agent never + # received the skill and an assertion that it was used is false. The summary folds a + # refusal and a later success into one loaded row, which is right for judging the choice + # and wrong for asserting the skill was in play. + attempts = [e for e in extract_skill_load_events(trajectory) if e.name == self.skill_name] + found = any(e.status == "loaded" for e in attempts) + refusal = next((e for e in attempts if e.status == "failed"), None) + if found: + reason = f"skill '{self.skill_name}' was invoked" + elif refusal is not None: + # Report what the harness said: a check that fails on a misspelled skill name and one + # that fails because nothing was mounted call for different fixes. + detail = f": {refusal.error}" if refusal.error else "" + attempted = f" ({len(attempts)} attempts)" if len(attempts) > 1 else "" + reason = f"skill '{self.skill_name}' was requested but the load failed{attempted}{detail}" + elif attempts: + # Every attempt was made and none has a recorded outcome, so the trajectory does not + # say whether the skill was received. Reported as not invoked, since the check asserts + # use and use was not observed, but named apart from never asking. + reason = f"skill '{self.skill_name}' was requested but the trajectory records no outcome" + else: + reason = f"skill '{self.skill_name}' was not invoked" + return [ + EvaluationOutput( + score=1.0 if found else 0.0, + test_pass=found, + reason=reason, + ) + ] + + async def evaluate_async(self, evaluation_case: EvaluationData[InputT, OutputT]) -> list[EvaluationOutput]: + return self.evaluate(evaluation_case) diff --git a/src/strands_evals/evaluators/evaluator.py b/src/strands_evals/evaluators/evaluator.py index 4da8c479..694a9870 100644 --- a/src/strands_evals/evaluators/evaluator.py +++ b/src/strands_evals/evaluators/evaluator.py @@ -88,6 +88,26 @@ def _default_aggregator(outputs: list[EvaluationOutput]) -> tuple[float, bool, s combined_reason = " | ".join(o.reason for o in outputs if o.reason) return avg_score, all_pass, combined_reason + @staticmethod + def _aggregate_dropping_na(outputs: list[EvaluationOutput]) -> tuple[float, bool, str]: + """Average only the rows that carry a verdict. + + For evaluators that emit one row per decision and can find some of those decisions + unjudgeable, the not-applicable rows score 0.0 as a placeholder. Averaging that in would + report a case with one perfectly judged decision and one unjudgeable one as half right. + Set `self.aggregator` to this in `__init__` to opt in. + """ + scored = [o for o in outputs if not o.not_applicable] + if not scored: + reason = " | ".join(o.reason for o in outputs if o.reason) or "not applicable" + # Carry the rows' own verdicts: "nothing to judge" passes, but absent data fails. + all_pass = all(o.test_pass for o in outputs) if outputs else True + return (0.0, all_pass, reason) + avg = sum(o.score for o in scored) / len(scored) + all_pass = all(o.test_pass for o in scored) + reason = " | ".join(o.reason for o in scored if o.reason) + return avg, all_pass, reason + def evaluate(self, evaluation_case: EvaluationData[InputT, OutputT]) -> list[EvaluationOutput]: """ Evaluate the performance of the task on the given test cases. diff --git a/src/strands_evals/evaluators/prompt_templates/skill_instruction_following/__init__.py b/src/strands_evals/evaluators/prompt_templates/skill_instruction_following/__init__.py new file mode 100644 index 00000000..9e87a1ef --- /dev/null +++ b/src/strands_evals/evaluators/prompt_templates/skill_instruction_following/__init__.py @@ -0,0 +1,11 @@ +from . import skill_instruction_following_v0 + +VERSIONS = { + "v0": skill_instruction_following_v0, +} + +DEFAULT_VERSION = "v0" + + +def get_template(version: str = DEFAULT_VERSION): + return VERSIONS[version] diff --git a/src/strands_evals/evaluators/prompt_templates/skill_instruction_following/skill_instruction_following_v0.py b/src/strands_evals/evaluators/prompt_templates/skill_instruction_following/skill_instruction_following_v0.py new file mode 100644 index 00000000..76a37620 --- /dev/null +++ b/src/strands_evals/evaluators/prompt_templates/skill_instruction_following/skill_instruction_following_v0.py @@ -0,0 +1,42 @@ +SYSTEM_PROMPT = """You are an objective judge evaluating whether an AI agent followed the \ +instructions of a skill it loaded. + +A skill is an instruction file (SKILL.md) with prescribed steps. You are given: +- the SKILL.md body (the steps the agent was supposed to follow), +- the agent's run (its actions, tool calls, and outputs). + +## Evaluation Task +Identify the prescribed steps in the SKILL.md body, then, for EACH step, judge from the agent's +run whether the step was: +- "covered": the run clearly shows the agent carried out the step, +- "partial": the run shows incomplete or ambiguous adherence, +- "skipped": the run shows no evidence the step was carried out. + +After the per-step judgments, give an overall five-point rating of how fully the agent +followed the skill's steps: +- "Fully Followed": essentially every step carried out; no meaningful gaps. +- "Mostly Followed": the great majority of steps carried out; only minor gaps. +- "Partially Followed": a mix of carried-out and skipped steps. +- "Minimally Followed": most steps skipped; only a few carried out. +- "Not Followed": the skill's steps were essentially ignored. + +The overall rating must be consistent with your per-step statuses: it should track the share +of steps that were covered (a run with nearly all steps covered is "Fully Followed"; one with +nearly all skipped is "Not Followed"). + +## Guidelines +- Judge only against the steps this skill prescribes, not generic task quality. +- Ground each per-step judgment in specific evidence from the run. +- A plan, unexecuted code snippet, or claim that a step will be done is not evidence + that an executable step was carried out. Require an action or result in the trajectory. +- If the skill has no clearly enumerable steps, treat its core instructions as the steps. +- If the skill body prescribes nothing at all (it is reference material rather than instructions), + return an empty step list rather than inventing steps. +- A skill written as a decision tree prescribes one path per run, not every branch. List only the + steps on the path this run's situation called for; leave out a branch the run correctly did not + enter rather than listing it as skipped. + +## Output Format +Provide, for each step, its status (covered / partial / skipped) with a short evidence note; +brief overall reasoning; and the overall five-point rating. +""" diff --git a/src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/__init__.py b/src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/__init__.py new file mode 100644 index 00000000..da47da30 --- /dev/null +++ b/src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/__init__.py @@ -0,0 +1,11 @@ +from . import skill_selection_accuracy_v0 + +VERSIONS = { + "v0": skill_selection_accuracy_v0, +} + +DEFAULT_VERSION = "v0" + + +def get_template(version: str = DEFAULT_VERSION): + return VERSIONS[version] diff --git a/src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/skill_selection_accuracy_v0.py b/src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/skill_selection_accuracy_v0.py new file mode 100644 index 00000000..f717cda4 --- /dev/null +++ b/src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/skill_selection_accuracy_v0.py @@ -0,0 +1,28 @@ +SYSTEM_PROMPT = """You are an objective judge evaluating whether an AI agent made an \ +appropriate skill-selection decision for a task. + +A skill is a reusable instruction file the agent may load to help with a task. At runtime the +agent is shown a list of available skills (each with a name and description) and decides which, +if any, to load. You are given: +- the task the agent was asked to do, +- the list of available skills (name + description), +- one skill the agent invoked, which is the decision under evaluation, +- the agent's run. + +## Evaluation Question +Judge only the one skill named as the decision under evaluation. +- "Yes" if that skill's description fits the task (it was a reasonable skill to load). +- "No" if that skill does not fit the task (an inappropriate pick). +- On a task that needs several skills, invoking any one skill that genuinely fits is + appropriate on its own; judge this skill on its own merits, not on whether the agent also + loaded the other skills it needed. + +## Guidelines +- Judge the selection decision, not how well the agent then executed the skill. +- Base the decision on the skill descriptions and the task, not on the outcome. + +## Output Format +First give brief step-by-step reasoning, then a single verdict: +- "Yes" if the decision was appropriate, +- "No" if it was not. +""" diff --git a/src/strands_evals/evaluators/prompt_templates/trajectory_prompt_template.py b/src/strands_evals/evaluators/prompt_templates/trajectory_prompt_template.py new file mode 100644 index 00000000..741a0b9c --- /dev/null +++ b/src/strands_evals/evaluators/prompt_templates/trajectory_prompt_template.py @@ -0,0 +1,49 @@ +"""Rendering a trajectory into judge-prompt text.""" + +from __future__ import annotations + +import json +from typing import Any + +from pydantic import BaseModel + +from ...types.trace import Session + +# Cap on serialized trajectory size in judge prompts, ~150k tokens at 4 chars/token. +MAX_TRAJECTORY_CHARS = 600_000 + + +def serialize_trajectory(trajectory: Session | list[Any] | None, max_chars: int = MAX_TRAJECTORY_CHARS) -> str: + """Serialize a trajectory into stable JSON, for use in judge prompts. + + The middle of a long run is dropped: a real trajectory can reach millions of tokens (one read + of a large artifact is enough), which overflows any judge context window. The head and tail are + kept because skills are loaded early and the outcome lands late. + + Args: + trajectory: A `Session` or a raw message list, or None. + max_chars: Size ceiling for the returned text. Pass 0 to keep the whole trajectory. + + Returns: + str: The serialized trajectory, "(no trajectory)" when None, with an inline note naming the + character count where the middle was omitted. + """ + if trajectory is None: + return "(no trajectory)" + if isinstance(trajectory, Session): + value: Any = trajectory.model_dump(mode="json") + else: + value = [item.model_dump(mode="json") if isinstance(item, BaseModel) else item for item in trajectory] + text = json.dumps(value, indent=2, default=str) + if max_chars <= 0 or len(text) <= max_chars: + return text + keep = max_chars // 2 + # The note says what a judge should infer from the gap, not just that there is one. The + # instruction-following rubric offers "skipped" for a step with no visible evidence, so a long + # but correct run would otherwise be marked down for the evidence that fell in the middle. + return ( + f"{text[:keep]}\n\n" + f"... [{len(text) - 2 * keep} characters omitted; evidence for a step may lie in this gap, " + f"so do not treat a step as skipped solely because nothing here shows it] ...\n\n" + f"{text[-keep:]}" + ) diff --git a/src/strands_evals/evaluators/skill_instruction_following_evaluator.py b/src/strands_evals/evaluators/skill_instruction_following_evaluator.py new file mode 100644 index 00000000..89299120 --- /dev/null +++ b/src/strands_evals/evaluators/skill_instruction_following_evaluator.py @@ -0,0 +1,232 @@ +from enum import Enum +from typing import Literal, cast + +from pydantic import BaseModel, Field +from strands import Agent +from strands.models.model import Model + +from ..extractors.skills import InvokedSkill, extract_selected_skills +from ..types.evaluation import NOT_APPLICABLE, EvaluationData, EvaluationOutput, InputT, OutputT +from .evaluator import Evaluator +from .prompt_templates.skill_instruction_following import get_template +from .prompt_templates.trajectory_prompt_template import serialize_trajectory + + +class SkillFollowingScore(str, Enum): + """Five-point ordinal rating for how fully the agent followed a skill's steps. + + Mirrors the five-point scale used by the coherence, faithfulness, and response + relevance evaluators (the framework's convention for graded quality judgments). + """ + + FULLY_FOLLOWED = "Fully Followed" + MOSTLY_FOLLOWED = "Mostly Followed" + PARTIALLY_FOLLOWED = "Partially Followed" + MINIMALLY_FOLLOWED = "Minimally Followed" + NOT_FOLLOWED = "Not Followed" + + +# The fields the AgentSkills plugin generates after the skill's own text, used to tell its runtime +# block apart from a Markdown rule inside real instructions. +_HARNESS_METADATA_FIELDS = ("Location:", "Allowed tools:", "Compatibility:", "Available resources:") + + +def _strip_frontmatter(body: str) -> str: + """Drop a leading YAML frontmatter block (`---\\n ... \\n---`) so the judge sees only steps. + + Some harnesses return the raw SKILL.md (frontmatter included); others return the + body alone. Stripping is a no-op when there is no frontmatter. + """ + if not body.startswith("---"): + return body + lines = body.splitlines() + # find the closing '---' after the opening one + for i in range(1, len(lines)): + if lines[i].strip() == "---": + return "\n".join(lines[i + 1 :]).lstrip("\n") + return body + + +def _strip_harness_metadata(body: str) -> str: + """Drop the runtime block the harness appends after the skill's own instructions. + + The Strands AgentSkills plugin ends a filesystem-skill result with a `---` rule followed by + lines it generated rather than the skill author: `Location:`, `Allowed tools:`, + `Compatibility:`, and an `Available resources:` list. The prompt labels this whole string + "SKILL.md instructions", so a judge can read `Available resources: scripts/extract.py` as a + prescribed step nobody wrote, or `Allowed tools:` as a constraint from the skill. + + Keyed on those field names rather than on the `---` alone, because a rule is legal Markdown + inside real instructions and splitting on it would truncate them. + """ + marker = "\n---\n" + index = body.rfind(marker) + if index == -1: + return body + tail = body[index + len(marker) :] + if not tail.strip(): + return body + first = tail.lstrip().split("\n", 1)[0] + if any(first.startswith(field) for field in _HARNESS_METADATA_FIELDS): + return body[:index].rstrip("\n") + return body + + +class SkillFollowingRating(BaseModel): + """Structured output for skill instruction following evaluation.""" + + reasoning: str = Field(description="Brief overall reasoning about adherence to the skill") + # Still required, but deliberately without `min_length=1`. This model is the + # structured-output schema, and a skill body with nothing prescriptive in it (reference + # material, frontmatter only) makes an empty list the correct answer. Rejecting it sends the + # judge back to re-emit the same answer until the retry loop dies of recursion depth. The + # empty case is handled in `_rating_to_output` instead. + steps: list["SkillStepRating"] = Field( + description="One status and evidence record per prescribed step, in instruction order", + ) + score: SkillFollowingScore = Field( + description=( + "Overall five-point rating of how fully the skill's steps were followed, " + "consistent with the per-step statuses" + ) + ) + + @property + def coverage(self) -> float: + """Derive coverage from structured statuses instead of trusting model arithmetic. + + Zero when there are no steps, which is the vacuous case rather than a failure; callers + distinguish the two by checking `steps` themselves. + """ + if not self.steps: + return 0.0 + weights = {"covered": 1.0, "partial": 0.5, "skipped": 0.0} + return sum(weights[step.status] for step in self.steps) / len(self.steps) + + +class SkillStepRating(BaseModel): + """Judge result for one prescribed skill step.""" + + step: str = Field(description="The prescribed step being evaluated") + status: Literal["covered", "partial", "skipped"] + evidence: str = Field(description="Concrete trajectory evidence for the status") + + +class SkillInstructionFollowingEvaluator(Evaluator[InputT, OutputT]): + """Evaluates whether the agent followed the steps of each skill it invoked. + + Returns one `EvaluationOutput` per invoked skill, scored on a five-point rating. When no + skill was invoked there is nothing to follow, so a single not-applicable row is returned + and dropped from the aggregated mean. + """ + + _score_mapping = { + SkillFollowingScore.FULLY_FOLLOWED: 1.0, + SkillFollowingScore.MOSTLY_FOLLOWED: 0.75, + SkillFollowingScore.PARTIALLY_FOLLOWED: 0.5, + SkillFollowingScore.MINIMALLY_FOLLOWED: 0.25, + SkillFollowingScore.NOT_FOLLOWED: 0.0, + } + + def __init__( + self, + version: str = "v0", + model: Model | str | None = None, + system_prompt: str | None = None, + name: str | None = None, + ): + super().__init__(name=name) + self.system_prompt = system_prompt if system_prompt is not None else get_template(version).SYSTEM_PROMPT + self.version = version + self.model = model + # Drop not-applicable rows from the aggregate so no-skill runs don't deflate the mean. + self.aggregator = self._aggregate_dropping_na + + def _not_applicable_row(self, reason: str, test_pass: bool = True) -> EvaluationOutput: + return EvaluationOutput(score=0.0, test_pass=test_pass, reason=reason, label=NOT_APPLICABLE) + + def _missing_trajectory_row(self) -> EvaluationOutput: + """A missing trajectory is absent data, not a run that had nothing to follow.""" + return self._not_applicable_row("no trajectory provided", test_pass=False) + + @staticmethod + def _unscorable_reason(skill: InvokedSkill) -> str | None: + """Why this skill cannot be scored for adherence, or None when it can be. + + A refused load is reported separately from a missing body: the agent never received any + instructions, so there was nothing it could have followed. Both are not-applicable, but + conflating them hides a broken harness behind what looks like a capture gap. + """ + if skill.status == "failed": + # The harness's own message says which refusal it was, and so what to fix: a + # misspelled skill name in the agent's call, or a harness that mounted none. + refusal = f" ({skill.error})" if skill.error else "" + return f"{skill.name}: the harness refused the load{refusal}, so no instructions were received" + if not skill.body: + return f"{skill.name}: skill body unavailable" + return None + + def _build_prompt(self, skill: InvokedSkill, evaluation_case: EvaluationData[InputT, OutputT]) -> str: + body = _strip_harness_metadata(_strip_frontmatter(skill.body or "")) + return ( + f"## Skill: {skill.name}\n\n" + f"## SKILL.md instructions\n{body}\n\n" + f"## Agent trajectory\n{serialize_trajectory(evaluation_case.actual_trajectory)}\n\n" + f"## Agent's final response\n{evaluation_case.actual_output}" + ) + + def _rating_to_output(self, skill: InvokedSkill, rating: SkillFollowingRating) -> EvaluationOutput: + # A skill that prescribes nothing has nothing to follow, so scoring it either way would be + # arbitrary: it is the same vacuous case as "no skill invoked", not a failure to adhere. + if not rating.steps: + return self._not_applicable_row(f"{skill.name}: no prescribed steps found in the skill body") + # Score off the five-point ordinal rating via `_score_mapping`, following the + # framework's graded-quality judges. The per-step statuses ground that rating and + # are preserved in `reason` (a plain field), so the base output schema is untouched. + # `label` carries the ordinal rating (its enum value), consistent with the other judges. + normalized_score = self._score_mapping[rating.score] + step_evidence = "\n".join(f"- {step.step}: {step.status}; evidence: {step.evidence}" for step in rating.steps) + return EvaluationOutput( + score=normalized_score, + # A skill's steps are prescriptive, so the bar is "Mostly Followed" rather than the + # mid-scale 0.5 the open-ended quality judges use. + test_pass=normalized_score >= 0.75, + reason=(f"{skill.name}: {rating.reasoning}\nCoverage: {rating.coverage:.2f}\nSteps:\n{step_evidence}"), + label=rating.score.value, + ) + + def evaluate(self, evaluation_case: EvaluationData[InputT, OutputT]) -> list[EvaluationOutput]: + if evaluation_case.actual_trajectory is None: + return [self._missing_trajectory_row()] + invoked = extract_selected_skills(evaluation_case.actual_trajectory) + if not invoked: + return [self._not_applicable_row("no skill invoked")] + results = [] + for skill in invoked: + if reason := self._unscorable_reason(skill): + results.append(self._not_applicable_row(reason)) + continue + prompt = self._build_prompt(skill, evaluation_case) + evaluator_agent = Agent(model=self.model, system_prompt=self.system_prompt, callback_handler=None) + result = evaluator_agent(prompt, structured_output_model=SkillFollowingRating) + rating = cast(SkillFollowingRating, result.structured_output) + results.append(self._rating_to_output(skill, rating)) + return results + + async def evaluate_async(self, evaluation_case: EvaluationData[InputT, OutputT]) -> list[EvaluationOutput]: + if evaluation_case.actual_trajectory is None: + return [self._missing_trajectory_row()] + invoked = extract_selected_skills(evaluation_case.actual_trajectory) + if not invoked: + return [self._not_applicable_row("no skill invoked")] + results = [] + for skill in invoked: + if reason := self._unscorable_reason(skill): + results.append(self._not_applicable_row(reason)) + continue + prompt = self._build_prompt(skill, evaluation_case) + evaluator_agent = Agent(model=self.model, system_prompt=self.system_prompt, callback_handler=None) + result = await evaluator_agent.invoke_async(prompt, structured_output_model=SkillFollowingRating) + rating = cast(SkillFollowingRating, result.structured_output) + results.append(self._rating_to_output(skill, rating)) + return results diff --git a/src/strands_evals/evaluators/skill_selection_accuracy_evaluator.py b/src/strands_evals/evaluators/skill_selection_accuracy_evaluator.py new file mode 100644 index 00000000..9835a564 --- /dev/null +++ b/src/strands_evals/evaluators/skill_selection_accuracy_evaluator.py @@ -0,0 +1,203 @@ +from enum import Enum +from typing import cast + +from pydantic import BaseModel, Field +from strands import Agent +from strands.models.model import Model + +from ..extractors.skills import ( + InvokedSkill, + advertised_a_catalog, + extract_selected_skills, + parse_available_skills, +) +from ..types.evaluation import NOT_APPLICABLE, EvaluationData, EvaluationOutput, InputT, OutputT +from .evaluator import Evaluator +from .prompt_templates.skill_selection_accuracy import get_template +from .prompt_templates.trajectory_prompt_template import serialize_trajectory + + +class SkillSelectionScore(str, Enum): + """Binary skill selection appropriateness ratings.""" + + YES = "Yes" + NO = "No" + + +class SkillSelectionRating(BaseModel): + """Structured output for skill selection accuracy evaluation.""" + + reasoning: str = Field(description="Step by step reasoning to derive the final score") + score: SkillSelectionScore = Field(description="Score should be one of 'Yes' or 'No'") + + +class SkillSelectionAccuracyEvaluator(Evaluator[InputT, OutputT]): + """Evaluates whether each skill the agent invoked was an appropriate selection. + + Returns one `EvaluationOutput` per invoked skill. A run that invoked nothing has no + selection to judge and yields a single not-applicable row. Whether declining was correct + is a question about the whole offered set rather than about any one invocation, so it is + out of scope here and belongs to a session-level check. + """ + + _score_mapping = { + SkillSelectionScore.YES: 1.0, + SkillSelectionScore.NO: 0.0, + } + + def __init__( + self, + version: str = "v0", + model: Model | str | None = None, + system_prompt: str | None = None, + name: str | None = None, + ): + super().__init__(name=name) + self.system_prompt = system_prompt if system_prompt is not None else get_template(version).SYSTEM_PROMPT + self.version = version + self.model = model + # A case with nothing to select from contributes a placeholder 0.0 row; averaging it in + # would report a run that had no decision to make as a failed one. + self.aggregator = self._aggregate_dropping_na + + def _available_str(self, evaluation_case: EvaluationData[InputT, OutputT]) -> str: + """The offered skills as the judge sees them, or why the list is empty. + + The two empty cases have to read differently. A harness that advertises no skills is a + fact about the run, and "none" is the honest rendering. A harness that never records what + it offered is a gap in the telemetry, and rendering it as an empty set invites the judge to + conclude the invoked skill did not exist and mark a correct pick wrong. Claude Code and the + Claude Agent SDK never emit the block, so that is the common case, not a corner one. + """ + available = parse_available_skills(evaluation_case.actual_trajectory) + if available: + return "\n".join(f"- {s.name}: {s.description}" for s in available) + if advertised_a_catalog(evaluation_case.actual_trajectory): + return "(none: this harness advertised no skills)" + return "(not recorded by this harness)" + + def _has_catalog(self, evaluation_case: EvaluationData[InputT, OutputT]) -> bool: + return bool(parse_available_skills(evaluation_case.actual_trajectory)) + + def _case_context(self, evaluation_case: EvaluationData[InputT, OutputT]) -> tuple[str, str]: + """The two halves of the prompt that do not depend on which decision is being judged. + + Built once per case: the skill catalog and the serialized trajectory are the same for + every invoked skill, and serializing a long trajectory once per skill is wasted work. + """ + head = f"## Task\n{evaluation_case.input}\n\n## Available skills\n{self._available_str(evaluation_case)}\n\n" + tail = ( + f"## Agent trajectory\n{serialize_trajectory(evaluation_case.actual_trajectory)}\n\n" + f"## Agent's final response\n{evaluation_case.actual_output}" + ) + return head, tail + + @staticmethod + def _prompt_for(context: tuple[str, str], focus_skill: InvokedSkill) -> str: + """Prompt judging one focal decision: whether invoking `focus_skill` was appropriate.""" + head, tail = context + decision = f"## Decision under evaluation\nThe agent invoked the skill: {focus_skill.name}\n" + if focus_skill.status == "failed": + # What is being judged is the choice, not the outcome. Without this the judge sees + # an error in the trajectory and marks a correct selection wrong for failing. + decision += ( + "The harness refused the load, so the agent never received the skill. " + "Judge whether asking for this skill was the right choice, not whether it worked.\n" + ) + if focus_skill.error: + # The refusal text can still bear on the choice: a name the harness did not + # recognize is a worse pick than a right one the harness could not mount. + decision += f"The harness said: {focus_skill.error}\n" + return f"{head}{decision}\n{tail}" + + def _build_prompt( + self, + evaluation_case: EvaluationData[InputT, OutputT], + focus_skill: InvokedSkill, + ) -> str: + """Prompt judging one focal decision: whether invoking `focus_skill` was appropriate.""" + return self._prompt_for(self._case_context(evaluation_case), focus_skill) + + def _rating_to_output(self, rating: SkillSelectionRating, decision: str) -> EvaluationOutput: + """One row for one decision. + + `label` carries the judge's rating, as every other judge in the framework does, so a + consumer reading labels across evaluators sees verdicts rather than a mix of verdicts and + skill names. Which decision the row is about is named in `reason` instead, since a case + with several invoked skills produces several rows. + """ + normalized_score = self._score_mapping[rating.score] + return EvaluationOutput( + score=normalized_score, + test_pass=normalized_score == 1.0, + reason=f"{decision}: {rating.reasoning}", + label=rating.score.value, + ) + + def _new_judge(self) -> Agent: + """A fresh judge per decision. + + Each skill is judged independently, so reusing one `Agent` across the loop would both + carry the previous verdicts into the next prompt as conversation history and resend the + whole trajectory on top of it, growing every request. + """ + return Agent(model=self.model, system_prompt=self.system_prompt, callback_handler=None) + + def _judge(self, prompt: str) -> SkillSelectionRating: + result = self._new_judge()(prompt, structured_output_model=SkillSelectionRating) + return cast(SkillSelectionRating, result.structured_output) + + async def _judge_async(self, prompt: str) -> SkillSelectionRating: + result = await self._new_judge().invoke_async(prompt, structured_output_model=SkillSelectionRating) + return cast(SkillSelectionRating, result.structured_output) + + @staticmethod + def _not_applicable_row(reason: str, test_pass: bool) -> EvaluationOutput: + return EvaluationOutput(score=0.0, test_pass=test_pass, reason=reason, label=NOT_APPLICABLE) + + @classmethod + def _missing_trajectory_row(cls) -> EvaluationOutput: + """A missing trajectory is absent data, not a correct abstention, so it is not scored.""" + return cls._not_applicable_row("no trajectory provided", test_pass=False) + + @classmethod + def _no_invocation_row(cls, has_catalog: bool) -> EvaluationOutput: + """No invoked skill means no selection decision for this evaluator to judge. + + Whether declining was correct depends on the whole offered set, not on any one + invocation, so it is a session-level question and out of scope here. The reason + still distinguishes the two cases, because "nothing was on offer" and "something was + on offer and none was taken" are different facts even when neither is scored. + """ + reason = ( + "no skill invoked; whether declining was correct is not judged here" + if has_catalog + else "no skills were available to select from" + ) + return cls._not_applicable_row(reason, test_pass=True) + + def evaluate(self, evaluation_case: EvaluationData[InputT, OutputT]) -> list[EvaluationOutput]: + if evaluation_case.actual_trajectory is None: + return [self._missing_trajectory_row()] + invoked = extract_selected_skills(evaluation_case.actual_trajectory) + if not invoked: + return [self._no_invocation_row(self._has_catalog(evaluation_case))] + context = self._case_context(evaluation_case) + results = [] + for skill in invoked: + rating = self._judge(self._prompt_for(context, skill)) + results.append(self._rating_to_output(rating, decision=skill.name)) + return results + + async def evaluate_async(self, evaluation_case: EvaluationData[InputT, OutputT]) -> list[EvaluationOutput]: + if evaluation_case.actual_trajectory is None: + return [self._missing_trajectory_row()] + invoked = extract_selected_skills(evaluation_case.actual_trajectory) + if not invoked: + return [self._no_invocation_row(self._has_catalog(evaluation_case))] + context = self._case_context(evaluation_case) + results = [] + for skill in invoked: + rating = await self._judge_async(self._prompt_for(context, skill)) + results.append(self._rating_to_output(rating, decision=skill.name)) + return results diff --git a/src/strands_evals/experiment.py b/src/strands_evals/experiment.py index de42b984..2fed4958 100644 --- a/src/strands_evals/experiment.py +++ b/src/strands_evals/experiment.py @@ -22,7 +22,7 @@ from .evaluators.coherence_evaluator import CoherenceEvaluator from .evaluators.conciseness_evaluator import ConcisenessEvaluator from .evaluators.correctness_evaluator import CorrectnessEvaluator -from .evaluators.deterministic import Contains, Equals, StartsWith, StateEquals, ToolCalled +from .evaluators.deterministic import Contains, Equals, SkillInvoked, StartsWith, StateEquals, ToolCalled from .evaluators.evaluator import Evaluator from .evaluators.faithfulness_evaluator import FaithfulnessEvaluator from .evaluators.goal_success_rate_evaluator import GoalSuccessRateEvaluator @@ -38,6 +38,8 @@ from .evaluators.output_evaluator import OutputEvaluator from .evaluators.refusal_evaluator import RefusalEvaluator from .evaluators.response_relevance_evaluator import ResponseRelevanceEvaluator +from .evaluators.skill_instruction_following_evaluator import SkillInstructionFollowingEvaluator +from .evaluators.skill_selection_accuracy_evaluator import SkillSelectionAccuracyEvaluator from .evaluators.stereotyping_evaluator import StereotypingEvaluator from .evaluators.tool_parameter_accuracy_evaluator import ToolParameterAccuracyEvaluator from .evaluators.tool_selection_accuracy_evaluator import ToolSelectionAccuracyEvaluator @@ -45,7 +47,7 @@ from .telemetry import get_tracer, serialize from .telemetry._cloudwatch_logger import _send_to_cloudwatch from .types.detector import DiagnosisConfig -from .types.evaluation import EvaluationData, InputT, OutputT +from .types.evaluation import NOT_APPLICABLE, EvaluationData, EvaluationOutput, InputT, OutputT from .types.evaluation_report import EvaluationReport from .types.trace import Session from .utils import is_throttling_error @@ -59,7 +61,11 @@ _MAX_RETRY_DELAY = 240 # 4 minutes -def _get_label_from_score(evaluator: Evaluator, score: float) -> str: +def _get_label_from_score( + evaluator: Evaluator, + score: float, + outputs: list[EvaluationOutput] | None = None, +) -> str: """ Get the label from score using evaluator's _score_mapping if available. If no mapping exists, returns "YES" for scores >= 0.5, "NO" otherwise. @@ -67,11 +73,17 @@ def _get_label_from_score(evaluator: Evaluator, score: float) -> str: Args: evaluator: The evaluator instance score: The numeric score - default_label: Default label to return if provided and no mapping found + outputs: The rows the score was aggregated from, when available. A case whose every row + was not-applicable has no verdict to report, and its 0.0 is a placeholder rather than + a score, so reverse-mapping it would emit the mapping's worst label for a case that + was never judged. Returns: The label corresponding to the score """ + if outputs is not None and not EvaluationReport.is_applicable(outputs): + return NOT_APPLICABLE + if hasattr(evaluator, "_score_mapping") and evaluator._score_mapping: # Create reverse mapping from score to label reverse_mapping = {v: k for k, v in evaluator._score_mapping.items()} @@ -366,7 +378,7 @@ async def _evaluate_with_retry(evaluator=evaluator, evaluation_context=evaluatio ) = await _evaluate_with_retry() try: - label = _get_label_from_score(evaluator, aggregate_score) + label = _get_label_from_score(evaluator, aggregate_score, evaluation_outputs) except Exception: label = "UNKNOWN" @@ -685,7 +697,10 @@ async def run_evaluations_async( data = evaluator_data[eval_name] scores = data["scores"] report = EvaluationReport( - overall_score=sum(scores) / len(scores) if scores else 0, + overall_score=EvaluationReport.calculate_overall_score( + scores, + data["detailed_results"], + ), scores=scores, test_passes=data["test_passes"], cases=data["cases"], @@ -775,6 +790,8 @@ def from_dict(cls, data: dict, custom_evaluators: list[type[Evaluator]] | None = "InstructionFollowingEvaluator": InstructionFollowingEvaluator, "RefusalEvaluator": RefusalEvaluator, "ResponseRelevanceEvaluator": ResponseRelevanceEvaluator, + "SkillInstructionFollowingEvaluator": SkillInstructionFollowingEvaluator, + "SkillSelectionAccuracyEvaluator": SkillSelectionAccuracyEvaluator, "StereotypingEvaluator": StereotypingEvaluator, "ToolParameterAccuracyEvaluator": ToolParameterAccuracyEvaluator, "ToolSelectionAccuracyEvaluator": ToolSelectionAccuracyEvaluator, @@ -788,6 +805,7 @@ def from_dict(cls, data: dict, custom_evaluators: list[type[Evaluator]] | None = "StartsWith": StartsWith, "StateEquals": StateEquals, "ToolCalled": ToolCalled, + "SkillInvoked": SkillInvoked, } all_evaluators: dict[str, type[Evaluator]] = { **default_evaluators, diff --git a/src/strands_evals/extractors/__init__.py b/src/strands_evals/extractors/__init__.py index 3e76b551..c56a4291 100644 --- a/src/strands_evals/extractors/__init__.py +++ b/src/strands_evals/extractors/__init__.py @@ -1,3 +1,21 @@ +from .skills import ( + AvailableSkill, + InvokedSkill, + SkillLoadEvent, + advertised_a_catalog, + extract_selected_skills, + extract_skill_load_events, + parse_available_skills, +) from .trace_extractor import TraceExtractor -__all__ = ["TraceExtractor"] +__all__ = [ + "TraceExtractor", + "AvailableSkill", + "advertised_a_catalog", + "InvokedSkill", + "SkillLoadEvent", + "parse_available_skills", + "extract_selected_skills", + "extract_skill_load_events", +] diff --git a/src/strands_evals/extractors/skills/__init__.py b/src/strands_evals/extractors/skills/__init__.py new file mode 100644 index 00000000..98ae8145 --- /dev/null +++ b/src/strands_evals/extractors/skills/__init__.py @@ -0,0 +1,37 @@ +"""Skill trajectory parsing helpers. + +`parse_available_skills` recovers the skills exposed to the agent (name plus +description) from the harness-injected `` block. +`extract_skill_load_events` recovers every load attempt in trajectory order, and +`extract_selected_skills` folds those into one row per skill, each with its +`SKILL.md` body when the trajectory carried it. All accept a `Session` or a raw +message list. + +Skills are not first-class in the trace schema the way tools are: there is no +`AgentInvocationSpan.available_skills` field, and a skill invocation surfaces as +an ordinary tool call. A load is detected either by a reserved skill-tool name and +its skill-name argument, or by a read of a known `SKILL.md` path. + +The work is split three ways: `models` holds what the extractors return, `adapters` +holds the per-harness block shapes, and `extractor` holds the harness-independent +decisions about which calls are skill loads. `_patterns` and `_normalize` are the +literals and primitives the other two share. +""" + +from .extractor import ( + advertised_a_catalog, + extract_selected_skills, + extract_skill_load_events, + parse_available_skills, +) +from .models import AvailableSkill, InvokedSkill, SkillLoadEvent + +__all__ = [ + "AvailableSkill", + "advertised_a_catalog", + "InvokedSkill", + "SkillLoadEvent", + "extract_selected_skills", + "extract_skill_load_events", + "parse_available_skills", +] diff --git a/src/strands_evals/extractors/skills/_normalize.py b/src/strands_evals/extractors/skills/_normalize.py new file mode 100644 index 00000000..1811247c --- /dev/null +++ b/src/strands_evals/extractors/skills/_normalize.py @@ -0,0 +1,266 @@ +"""Reading text, statuses and skill names out of whatever shape a harness used. + +These are the primitives the adapters and the extraction policy share: flattening content +wrappers, deciding whether a result failed, and naming a skill from its body or its path. +""" + +from __future__ import annotations + +import html +import json +from typing import Any + +import yaml +from pydantic import BaseModel +from strands import Skill + +from ._patterns import ( + _ACKNOWLEDGEMENT, + _AVAILABLE_BLOCK, + _AVAILABLE_MARKDOWN, + _AVAILABLE_MARKDOWN_ENTRY, + _DESC_TAG, + _FAILED_STATUSES, + _FILE_LOCATOR, + _HARNESS_TOOLS, + _LOAD_ERROR, + _NAME_TAG, + _SKILL_ENTRY, + _SKILL_PATH, + _SUCCESS_EXIT_CODES, +) +from .models import AvailableSkill + + +def _as_dict(value: Any) -> dict[str, Any] | None: + """Normalize raw dictionaries and strands_evals Pydantic trace objects.""" + if isinstance(value, dict): + return value + if isinstance(value, BaseModel): + return value.model_dump(mode="json") + return None + + +def _content_text(content: Any) -> str: + """Flatten common text/content wrappers into text.""" + if isinstance(content, str): + return content + if isinstance(content, list): + parts = [_content_text(item) for item in content] + return "\n".join(part for part in parts if part) + item = _as_dict(content) + if item is not None: + for key in ( + "instructions", + "llmContent", + "content", + "output", + "aggregated_output", + "text", + # Google ADK nests its tool payload under `response`, with plain tool output + # under `result`; both must be traversed to reach the skill body or catalog. + "response", + "result", + ): + if key in item: + text = _content_text(item[key]) + if text: + return text + return "" + + +def _result_failed(result: Any) -> bool: + value = _as_dict(result) + if value is None: + return False + status = str(value.get("status", "")).casefold() + exit_code = value.get("exit_code") + failed = ( + status in _FAILED_STATUSES + or value.get("is_error") is True + or value.get("error") not in (None, "", False) + or exit_code not in _SUCCESS_EXIT_CODES + ) + if failed: + return True + return any( + _result_failed(value[key]) + for key in ("response", "result") + if key in value and _as_dict(value[key]) is not None + ) + + +def _load_refused(result: Any) -> bool: + """Whether the load was refused, including refusals the harness marks successful. + + A plugin can report a lookup failure in the payload rather than in the status. The Strands + AgentSkills plugin returns "Skill 'x' not found. ..." as a plain string from an `@tool` + function, and `@tool` reports a plain string return as `status="success"`, so the only signal + that no skill was loaded is the text. Recognizing it here is what separates a refused load + from a load whose body simply was not captured: both have `body=None`, but only the refusal + means the agent never received instructions. + """ + return _result_failed(result) or bool(_LOAD_ERROR.fullmatch(_content_text(result).strip())) + + +def _refusal_message(result: Any) -> str | None: + """What the harness said when it refused, or None when it said nothing usable. + + Reported alongside the refusal because which refusal it was decides what to fix, and the + text is the only place that distinction survives: "Skill 'pdf-procesing' not found" is a + misspelled name in the agent's call, while "Available skills: (none)" is a harness that + mounted nothing. A structured failure can carry no message at all, hence the None. + + The harness's own `error` field is preferred over the result text, including when it is + nested a level down the way `_result_failed` finds it, since that is where a harness that + keeps its diagnostics apart from tool output puts them. + """ + error = _error_field(result) + if error is not None: + return error + return _content_text(result).strip() or None + + +def _error_field(result: Any) -> str | None: + """The harness's `error` message, at the top level or nested under `response`/`result`.""" + value = _as_dict(result) + if value is None: + return None + error = value.get("error") + if isinstance(error, str) and error.strip(): + return error.strip() + for key in ("response", "result"): + if key in value: + nested = _error_field(value[key]) + if nested is not None: + return nested + return None + + +def _body_from_result(result: Any) -> str | None: + """Return actual skill instructions, excluding errors and load acknowledgements.""" + if _result_failed(result): + return None + text = _content_text(result).strip() + if not text or _ACKNOWLEDGEMENT.fullmatch(text) or _LOAD_ERROR.fullmatch(text): + return None + + # Some tool integrations JSON-encode their structured result. + if text.startswith(("{", "[")): + try: + decoded = json.loads(text) + except json.JSONDecodeError: + pass + else: + if decoded != result: + return _body_from_result(decoded) + return text + + +def _last_path_segment(path: str) -> str: + """The final component of a directory path, with separators normalized.""" + return path.replace("\\", "/").rstrip("/").rsplit("/", 1)[-1] + + +def _skill_name_from_path(path: str) -> str: + """The directory a `SKILL.md` sits in, which is the skill's name by convention. + + Returns "" when the path has no usable parent directory. A bare `SKILL.md` or `./SKILL.md` + names no skill, and answering `SKILL.md` or `.` would put a name that is wrong on its face in + front of the judge. Callers treat the empty result as "not a skill read". + """ + normalized = path.replace("\\", "/").rstrip("/") + parts = normalized.split("/") + if len(parts) < 2: + return "" + parent = parts[-2] + return "" if parent in {"", ".", ".."} else parent + + +def _canonical_skill_key(name: str) -> str: + """Fold the naming variants that refer to one skill. + + An agent may read the same `SKILL.md` more than once, and a partial read (a paged `sed` + window that misses the frontmatter) falls back to the directory name while a full read + reports the frontmatter name. A directory named `pdf_processing` holding a skill whose + frontmatter says `pdf-processing` is the same skill, so the two separators fold together. + `.` is left alone: it is legal in a skill name, so folding it would merge `data.clean` + and `data-clean`, which are two different skills. + """ + return name.casefold().replace("_", "-") + + +def _skill_name_from_body(body: str, path: str) -> str: + """Prefer the runtime-visible frontmatter name over a directory alias. + + `Skill.from_content` parses the `SKILL.md` YAML frontmatter. Bodies are + read from arbitrary on-disk files, so malformed frontmatter is expected; + `yaml` raises `YAMLError` (not a `ValueError`) on bad structure. Both + fall back to the directory-derived name rather than aborting extraction. + + Returns "" when neither source yields a name, which callers treat as "not a skill read". A + file literally named `SKILL.md` with no frontmatter and no parent directory carries nothing + that identifies a skill, and any name invented for it would be wrong. + """ + try: + return Skill.from_content(body).name + except (ValueError, yaml.YAMLError): + return _skill_name_from_path(path) + + +def _skill_path_from_text(value: str) -> str | None: + match = _SKILL_PATH.search(value) + if not match: + return None + return next(group for group in match.groups() if group is not None) + + +def _skill_name_from_args(tool_name: str, arguments: dict[str, Any]) -> str | None: + """Read the skill name from a reserved tool's input arguments.""" + key = _HARNESS_TOOLS.get(tool_name) + if key is None and tool_name.casefold().endswith("_load_skill"): + key = "skill_name" # Google ADK permits a tool_name_prefix. + if key is None: + return None + value = arguments.get(key) + # A missing or empty name is a malformed call rather than a selection, and it yields no event: + # there is no skill to attribute the attempt to. Harnesses do refuse these for real (the Strands + # plugin answers "Error: skill_name is required.", Google ADK answers with an INVALID_ARGUMENTS + # error code), but reporting the attempt would mean inventing a skill named "" and counting it + # against the agent's selection accuracy, which is worse than not seeing the fumbled call. + return str(value) if value else None + + +def _parse_available_block(text: str) -> list[AvailableSkill]: + """Parse an XML or Markdown available-skills section from prompt text. + + Skills missing a name are skipped. A missing description yields an empty + description rather than dropping the skill. + """ + block_match = _AVAILABLE_BLOCK.search(text or "") + if block_match: + out: list[AvailableSkill] = [] + for entry in _SKILL_ENTRY.finditer(block_match.group(1)): + body = entry.group(1) + name_m = _NAME_TAG.search(body) + if not name_m: + continue + desc_m = _DESC_TAG.search(body) + out.append( + AvailableSkill( + name=html.unescape(name_m.group(1).strip()), + description=html.unescape(desc_m.group(1).strip()) if desc_m else "", + ) + ) + return out + + markdown_match = _AVAILABLE_MARKDOWN.search(text or "") + if not markdown_match: + return [] + return [ + AvailableSkill( + name=match.group("name").strip(), + description=_FILE_LOCATOR.sub("", match.group("description")).strip(), + ) + for match in _AVAILABLE_MARKDOWN_ENTRY.finditer(markdown_match.group("body")) + ] diff --git a/src/strands_evals/extractors/skills/_patterns.py b/src/strands_evals/extractors/skills/_patterns.py new file mode 100644 index 00000000..bbffbe28 --- /dev/null +++ b/src/strands_evals/extractors/skills/_patterns.py @@ -0,0 +1,113 @@ +"""The literal text and shapes the harnesses emit. + +Everything here is a fact about some harness's output format, keyed by the design-doc B.1 +table and verified against real runs. Kept apart from the extraction policy so that adding a +harness is a change to this module and its adapter, not to the traversal logic. +""" + +from __future__ import annotations + +import re + +# Reserved skill-tool name -> the input-argument key that holds the skill name. +_HARNESS_TOOLS: dict[str, str] = { + "skills": "skill_name", # Strands AgentSkills plugin + "Skill": "skill", # Claude Code / Claude Agent SDK + "load_skill": "skill_name", # OpenAI Agents SDK, Google ADK + "activate_skill": "name", # Gemini CLI + "invoke_skill": "name", # OpenHands +} + +# Claude Code injects the skill body as a user message headed by the skill's directory. +_CLAUDE_BASE_DIR = re.compile(r"Base directory for this skill:\s*(?P\S+)", re.IGNORECASE) + +# The available-skills block the harness injects into the system prompt. +_AVAILABLE_BLOCK = re.compile(r"(.*?)", re.DOTALL | re.IGNORECASE) +_SKILL_ENTRY = re.compile(r"(.*?)", re.DOTALL | re.IGNORECASE) +_NAME_TAG = re.compile(r"(.*?)", re.DOTALL | re.IGNORECASE) +_DESC_TAG = re.compile(r"(.*?)", re.DOTALL | re.IGNORECASE) +_AVAILABLE_MARKDOWN = re.compile( + r"^### Available skills\s*$\n(?P.*?)(?=^###\s|\Z)", + re.DOTALL | re.IGNORECASE | re.MULTILINE, +) +_AVAILABLE_MARKDOWN_ENTRY = re.compile( + r"^\s*-\s+(?P[^:\n]+):\s*(?P.*?)\s*$", + re.MULTILINE, +) +_FILE_LOCATOR = re.compile(r"\s+\(file:\s*.+?\)\s*$", re.IGNORECASE) +_SKILL_PATH = re.compile( + r'"([^"\n]*SKILL\.md)"|\'([^\'\n]*SKILL\.md)\'|([^\s"\'=;|<>]*SKILL\.md)', + re.IGNORECASE, +) +_READ_VERBS = {"cat", "sed", "head", "tail", "bat", "less", "type", "get-content"} +# `bash -lc "..."`, `/bin/sh -c '...'`: the real command is inside the quotes. +_SHELL_WRAPPER = re.compile( + r"^\s*(?:\S*/)?(?:ba|da|k|z)?sh\s+(?:-\w+\s+)*(?P['\"])(?P.*)(?P=quote)\s*$", + re.DOTALL, +) +# Command separators. Splitting inside a quoted argument is harmless here: the pieces are only +# used to locate a read verb and a path, and a quoted separator does not produce either. +_SHELL_SEPARATOR = re.compile(r"\|\||&&|[;|\n]") +# Leading noise before the verb: `sudo`, `env`, `FOO=bar`, `command`, `time`. +_SHELL_PREFIX = re.compile(r"^(?:sudo|env|command|time|nohup|\S+=\S*)\s+", re.IGNORECASE) +_SED_IN_PLACE = re.compile(r"(?:^|\s)(?:-i\S*|--in-place\b)") +_READ_TOOL_NAMES = { + "read", + "read_file", + "file_read", + "filesystem_read", + "read_text_file", +} +_SHELL_TOOL_NAMES = {"bash", "shell", "terminal", "command", "execute_command", "exec_command", "run_command"} +# Codex wraps shell output in a fixed preamble before the output itself, e.g. +# Chunk ID: 92cb1e +# Wall time: 0.0631 seconds +# Process exited with code 0 +# Original token count: 63 +# Output: +# --- +# name: pdf-processing +# Leaving the preamble attached would make the skill body start with the wall time, so the +# frontmatter would not parse and the judge would score the envelope as instructions. +_CODEX_EXEC_OUTPUT = re.compile( + r"\A(?P(?:[A-Za-z][A-Za-z ]*:.*\n|Process exited with code\s+-?\d+\n)*?)Output:\n(?P.*)\Z", + re.DOTALL, +) +_CODEX_EXIT_CODE = re.compile(r"^Process exited with code\s+(?P-?\d+)\s*$", re.MULTILINE) +_DISCOVERY_TOOL_NAMES = {"list_skills", "search_skills"} +_FAILED_STATUSES = {"error", "errored", "fail", "failed", "failure", "cancelled", "canceled"} +# Exit codes that still leave usable output on stdout. 141 is SIGPIPE, which is what a paged read +# reports: `cat SKILL.md | head -20` exits 141 once head closes the pipe, having printed the part +# of the file the agent actually saw. Treating that as a failed load discards a real skill body. +_SUCCESS_EXIT_CODES = {None, 0, "0", 141, "141"} +_ACKNOWLEDGEMENT = re.compile( + # A load acknowledgement is not the skill body. Harnesses word this either way round + # ("Launching skill: x", or Gemini CLI's "Skill activated. Resources loaded from ..."), + # and mistaking one for the body would have the judge score a status line as instructions. + # The optional group is the skill name some harnesses interpose, e.g. the Strands + # AgentSkills plugin's "Skill 'x' activated (no instructions available).". + # Anchored to a single line, and deliberately not with `$`/`.*`: `[^\n]*\Z` is what keeps a + # body whose first line is an acknowledgement. `\s*` matches newlines, so a trailing `\s*.*$` + # reaches past the first line and discards a short real body along with its status line. + r"^(?:(?:Launching|Loading|Activating|Loaded|Activated)\s+skill" + r"|skill\s+(?:'[^']*'|\"[^\"]*\"|[\w.-]+)?[ \t]*(?:activated|loaded|launched))" + r"\b(?:[ \t]*[:.]?[^\n]*)?\Z", + re.IGNORECASE, +) +_LOAD_ERROR = re.compile( + # A refused load is not the skill body either, and it does not arrive marked as an error. + # The Strands AgentSkills plugin returns these as plain strings from an `@tool` function, + # and `@tool` reports a plain string return as `status="success"`, so `_result_failed` sees + # nothing wrong and the judge would be handed "Skill 'x' not found. Available skills: ..." + # as the instructions the agent was supposed to follow. + # Anchored to a single line, like `_ACKNOWLEDGEMENT` and for the same reason: a refusal-shaped + # first line must not carry away the real body that follows it. Reporting that body's skill as + # a failed load is the worse half of the bug, since the harness did activate it. + r"^(?:" + r"error\s*:\s*skill_name\s+is\s+required" + r"|skill\s+(?:'[^']*'|\"[^\"]*\"|[\w.-]+)\s+(?:was\s+|is\s+)?not\s+found" + r"|(?:unknown|unrecognized)\s+skill\b" + r"|no\s+such\s+skill\b" + r")[ \t]*[:.]?[^\n]*\Z", + re.IGNORECASE, +) diff --git a/src/strands_evals/extractors/skills/adapters/__init__.py b/src/strands_evals/extractors/skills/adapters/__init__.py new file mode 100644 index 00000000..d9175a19 --- /dev/null +++ b/src/strands_evals/extractors/skills/adapters/__init__.py @@ -0,0 +1,20 @@ +"""Per-harness block shapes. + +Each harness writes a tool call and a tool result its own way. These modules know only those +shapes: given one content block, they say whether it is a call or a result and pull out the +identifier, the name and the arguments or payload. Deciding what a call *means* (a skill load, a +file read, a refusal) is the extractor's job, not theirs. + +One module per harness (`strands`, `claude`, `gemini`, `codex`, `openhands`), `registry` for the +order they are tried in and the `ToolCallBlock` / `ToolResultBlock` they produce, and `_common` for +what they share. A harness whose skill load is not a tool call at all (Codex reads `SKILL.md` with a +shell command) contributes only its result envelope here; recognizing the load is policy and lives +in `extractor`. +""" + +from ._common import ToolCallBlock, ToolResultBlock + +__all__ = [ + "ToolCallBlock", + "ToolResultBlock", +] diff --git a/src/strands_evals/extractors/skills/adapters/_common.py b/src/strands_evals/extractors/skills/adapters/_common.py new file mode 100644 index 00000000..7dbf161e --- /dev/null +++ b/src/strands_evals/extractors/skills/adapters/_common.py @@ -0,0 +1,90 @@ +"""What every adapter returns, and the block-flattening they all feed from. + +A recognizer is a function from one content block to either the shape it knows or None. Calls +report `(call_id, name, arguments)` and results report `(raw_payload, call_id)`, both unvalidated: +`registry._tool_call` drops a call whose name or arguments are not the expected type, and +`registry._tool_result` interprets the payload. Keeping recognizers this thin is what lets a single +harness module be read on its own. +""" + +from __future__ import annotations + +from typing import Any, NamedTuple + +from .._normalize import _as_dict + +# A recognized call, as its harness module returns it: (call_id, name, arguments). +CallMatch = tuple[Any, Any, Any] +# A recognized result: (raw payload, call_id). The payload is handed to `_normalize` to read. +ResultMatch = tuple[Any, Any] + + +class ToolCallBlock(NamedTuple): + """A tool call recovered from a trajectory, harness-independent.""" + + call_id: str | None + name: str + arguments: dict[str, Any] + + +class ToolResultBlock(NamedTuple): + """A tool result recovered from a trajectory, harness-independent.""" + + call_id: str | None + refused: bool + body: str | None + error: str | None + + +def _looks_like_block(value: dict[str, Any]) -> bool: + return ( + "toolUse" in value + or "toolResult" in value + or value.get("type") + in {"tool_use", "tool_result", "text", "command_execution", "function_call", "function_call_output"} + or "tool_name" in value + or ("name" in value and "args" in value) + or str(value.get("kind", "")).startswith("InvokeSkill") + ) + + +def _iter_indexed_blocks(messages: list[Any]) -> list[tuple[int, str | None, dict[str, Any]]]: + """Flatten raw, Claude stream, and Codex event wrappers into content blocks.""" + blocks: list[tuple[int, str | None, dict[str, Any]]] = [] + for index, item in enumerate(messages): + outer = _as_dict(item) + if outer is None: + continue + + if outer.get("type") == "item.completed" and (codex_item := _as_dict(outer.get("item"))): + blocks.append((index, None, codex_item)) + continue + if outer.get("type") in { + "tool_response", + "function_response", + # Responses API items, as a Codex session rollout records them: the item *is* the + # block, with no message envelope around it. + "function_call", + "function_call_output", + } or str(outer.get("kind", "")).startswith("InvokeSkill"): + blocks.append((index, None, outer)) + continue + + message = _as_dict(outer.get("message")) or outer + role = message.get("role") or outer.get("role") + # Google GenAI puts the blocks in `parts`, not `content`: a `types.Content` dumps to + # `{"role", "parts": [...]}`, which is what Gemini and Google ADK trajectories are made of. + content = message.get("content", message.get("parts")) + if isinstance(content, list): + blocks.extend( + (index, str(role) if role else None, block) + for content_item in content + if (block := _as_dict(content_item)) is not None + ) + elif (block := _as_dict(content)) is not None: + blocks.append((index, str(role) if role else None, block)) + elif isinstance(content, str): + blocks.append((index, str(role) if role else None, {"type": "text", "text": content})) + elif _looks_like_block(message): + blocks.append((index, str(role) if role else None, message)) + return blocks diff --git a/src/strands_evals/extractors/skills/adapters/claude.py b/src/strands_evals/extractors/skills/adapters/claude.py new file mode 100644 index 00000000..d4526e3b --- /dev/null +++ b/src/strands_evals/extractors/skills/adapters/claude.py @@ -0,0 +1,31 @@ +"""Anthropic / Claude Code block shapes. + +The Anthropic content-block shape: `{"type": "tool_use", "id", "name", "input"}` and +`{"type": "tool_result", "tool_use_id", "content"}`. Claude Code streams these inside its own +event envelope, which `_common._iter_indexed_blocks` unwraps before they reach here. + +Claude Code delivers a skill body as a separate injected user message rather than in the tool +result, so its calls arrive with no body attached. Recovering it is the extractor's job +(`_claude_body_after`), because it means matching a later message to this call rather than reading +one block. +""" + +from __future__ import annotations + +from typing import Any + +from ._common import CallMatch, ResultMatch + + +def _anthropic_call(block: dict[str, Any]) -> CallMatch | None: + """Anthropic-style content block: `{"type": "tool_use", "name", "input"}`.""" + if block.get("type") == "tool_use": + return block.get("id") or block.get("tool_use_id"), block.get("name"), block.get("input") + return None + + +def _anthropic_result(block: dict[str, Any]) -> ResultMatch | None: + """Anthropic-style content block: `{"type": "tool_result", "tool_use_id"}`.""" + if block.get("type") == "tool_result": + return block, block.get("tool_use_id") or block.get("id") + return None diff --git a/src/strands_evals/extractors/skills/adapters/codex.py b/src/strands_evals/extractors/skills/adapters/codex.py new file mode 100644 index 00000000..a4ce7cf1 --- /dev/null +++ b/src/strands_evals/extractors/skills/adapters/codex.py @@ -0,0 +1,75 @@ +"""Codex and OpenAI Agents event shapes. + +Both emit events rather than messages, and Codex emits them in two different envelopes depending +on how it was run: + +- `codex exec --json` streams `{"type": "item.completed", "item": {"type": "command_execution", ...}}`, + unwrapped by `_common._iter_indexed_blocks`. +- The interactive CLI writes a session rollout of OpenAI Responses API items, so the same shell + call arrives as `{"type": "function_call", "name": "exec_command", "arguments": ""}` with + its output in a matching `function_call_output`. + +Either way a Codex skill load is a shell read of a `SKILL.md` path rather than a reserved skill +tool, so recognizing the load is the extractor's job. OpenAI Agents exposes a real `load_skill` +tool, whose call rides the `{"name", "args"}` recognizer in `gemini`. + +What is left here are the call and result envelopes. +""" + +from __future__ import annotations + +import json +from typing import Any + +from .._patterns import _CODEX_EXEC_OUTPUT, _CODEX_EXIT_CODE +from ._common import CallMatch, ResultMatch + + +def _function_call(block: dict[str, Any]) -> CallMatch | None: + """A Responses API `function_call` item, as a Codex session rollout records one. + + `arguments` is a JSON string rather than an object. A call whose arguments do not decode to a + dict is reported with an empty one instead of being dropped, so the call still counts as an + attempt. + """ + if block.get("type") != "function_call": + return None + raw = block.get("arguments") + arguments: Any = raw + if isinstance(raw, str): + try: + arguments = json.loads(raw) + except json.JSONDecodeError: + arguments = {} + return block.get("call_id") or block.get("id"), block.get("name"), arguments if isinstance(arguments, dict) else {} + + +def _function_call_output(block: dict[str, Any]) -> ResultMatch | None: + """A Responses API `function_call_output` item, with Codex's shell preamble stripped. + + Codex prefixes the command's output with its own header lines (chunk id, wall time, exit code, + token count). The exit code is lifted out so a failed read is recognized as one, and the rest + of the preamble is dropped: it is the harness talking about the command, not the command's + output, and a body that starts with "Wall time" parses as neither frontmatter nor instructions. + """ + if block.get("type") != "function_call_output": + return None + payload: dict[str, Any] = {"tool_call_id": block.get("call_id") or block.get("id")} + output = block.get("output") + if isinstance(output, str): + if match := _CODEX_EXEC_OUTPUT.match(output): + payload["output"] = match.group("output") + if code := _CODEX_EXIT_CODE.search(match.group("preamble")): + payload["exit_code"] = int(code.group("code")) + else: + payload["output"] = output + else: + payload["output"] = output + return payload, payload["tool_call_id"] + + +def _event_result(block: dict[str, Any]) -> ResultMatch | None: + """A `tool_response` / `function_response` event, as OpenAI Agents and Codex emit.""" + if block.get("type") in {"tool_response", "function_response"}: + return block, block.get("tool_use_id") or block.get("id") + return None diff --git a/src/strands_evals/extractors/skills/adapters/gemini.py b/src/strands_evals/extractors/skills/adapters/gemini.py new file mode 100644 index 00000000..7ad8a1a8 --- /dev/null +++ b/src/strands_evals/extractors/skills/adapters/gemini.py @@ -0,0 +1,47 @@ +"""Gemini CLI and Google ADK block shapes. + +Both are built on the Google GenAI content parts, `{"functionCall": {"id", "name", "args"}}` and +`{"functionResponse": {"id", "response"}}`, so one pair of recognizers serves them. Gemini CLI's +stream events use a flatter `{"tool_name", "parameters"}` instead, and ADK event items a bare +`{"name", "args"}`; both are here rather than in a separate module because they are the same two +products emitting the same call in a different envelope. + +The `{"name", "args"}` recognizer is the loosest in the registry and is tried last for that reason: +any block with a string `name` and a dict `args` matches it. +""" + +from __future__ import annotations + +from typing import Any + +from ._common import CallMatch, ResultMatch + + +def _gemini_call(block: dict[str, Any]) -> CallMatch | None: + """Gemini CLI / Google ADK content parts: `{"functionCall": {"id", "name", "args"}}`.""" + raw = block.get("functionCall") or block.get("function_call") + if isinstance(raw, dict): + return raw.get("id"), raw.get("name"), raw.get("args") + return None + + +def _gemini_result(block: dict[str, Any]) -> ResultMatch | None: + """Gemini CLI / Google ADK content parts: payload nests under `response`.""" + raw = block.get("functionResponse") or block.get("function_response") + if isinstance(raw, dict): + return raw, raw.get("id") + return None + + +def _named_tool_call(block: dict[str, Any]) -> CallMatch | None: + """Gemini CLI stream events: `{"tool_name", "parameters"}`.""" + if block.get("tool_name"): + return block.get("id"), block.get("tool_name"), block.get("parameters") + return None + + +def _args_call(block: dict[str, Any]) -> CallMatch | None: + """A bare `{"name", "args"}` call, as Google ADK and Codex event items emit.""" + if isinstance(block.get("name"), str) and isinstance(block.get("args"), dict): + return block.get("id"), block.get("name"), block.get("args") + return None diff --git a/src/strands_evals/extractors/skills/adapters/openhands.py b/src/strands_evals/extractors/skills/adapters/openhands.py new file mode 100644 index 00000000..fe5cec36 --- /dev/null +++ b/src/strands_evals/extractors/skills/adapters/openhands.py @@ -0,0 +1,28 @@ +"""OpenHands action and observation shapes. + +OpenHands models a skill load as a first-class event pair rather than a tool call: +`{"kind": "InvokeSkillAction", "name": ""}` and `{"kind": "InvokeSkillObservation"}`. The +skill name is the action's own `name`, not an argument, so the recognizer reports a synthetic +`invoke_skill` call carrying it as one. That keeps the extractor's skill-name lookup uniform across +harnesses instead of special-casing this one. +""" + +from __future__ import annotations + +from typing import Any + +from ._common import CallMatch, ResultMatch + + +def _openhands_call(block: dict[str, Any]) -> CallMatch | None: + """OpenHands: `{"kind": "InvokeSkillAction", "name"}`, with the skill name as the action name.""" + if block.get("kind") == "InvokeSkillAction": + return block.get("tool_call_id") or block.get("id"), "invoke_skill", {"name": block.get("name")} + return None + + +def _openhands_result(block: dict[str, Any]) -> ResultMatch | None: + """OpenHands: `{"kind": "InvokeSkillObservation"}`.""" + if block.get("kind") == "InvokeSkillObservation": + return block, block.get("tool_call_id") or block.get("id") + return None diff --git a/src/strands_evals/extractors/skills/adapters/registry.py b/src/strands_evals/extractors/skills/adapters/registry.py new file mode 100644 index 00000000..a7fc72a8 --- /dev/null +++ b/src/strands_evals/extractors/skills/adapters/registry.py @@ -0,0 +1,118 @@ +"""The order the harness recognizers are tried in, and the common blocks they produce. + +Each harness module knows one wire format and nothing about the others. This module is the only +place that knows they compete: a block is offered to each recognizer in turn and the first usable +match wins. A recognizer that matches the block's shape but cannot read a name and arguments out of +it does not end the search, because the same block can be a truncated call in one shape and a whole +one in another; see `_tool_call`. + +**The order is behavior, not style.** Shapes overlap, so a block can satisfy more than one +recognizer and the first one reached decides how it is read: + +- `strands._bedrock_*` before `claude._anthropic_*`: a harness can wrap a `toolUse` and also tag the + block `type: "tool_use"`, and the wrapper carries the identifier the flat block lacks. +- `gemini._args_call` last among the calls: `{"name", "args"}` is the loosest shape here, and any + block with a string name and a dict of arguments matches it, including blocks a more specific + recognizer would have read correctly. Order alone is not enough to protect it once the search can + continue past a match, so it is also withheld from any block that declares a harness + (`_UNTAGGED_CALL_ADAPTERS`). +- `openhands._openhands_call` after the rest: its blocks are `kind`-tagged and cannot collide, so + its position is free, but keeping it last leaves the loose recognizers' relative order intact. + +Adding a harness means adding a module and one entry here. Put it above `_args_call` unless its +blocks are tagged in a way nothing else matches. +""" + +from __future__ import annotations + +from typing import Any + +from .._normalize import _body_from_result, _load_refused, _refusal_message +from . import claude, codex, gemini, openhands, strands +from ._common import ToolCallBlock, ToolResultBlock + +_CALL_ADAPTERS = ( + strands._bedrock_call, + gemini._gemini_call, + strands._typed_call, + claude._anthropic_call, + codex._function_call, + gemini._named_tool_call, + gemini._args_call, + openhands._openhands_call, +) + + +_RESULT_ADAPTERS = ( + strands._bedrock_result, + gemini._gemini_result, + strands._typed_result, + claude._anthropic_result, + codex._function_call_output, + codex._event_result, + openhands._openhands_result, +) + +# The recognizers that match on shape alone, with no tag naming the harness they belong to. They +# are correct for the harnesses that emit a bare call, and wrong for anything else, so they are the +# ones `_tool_call` withholds from a block that already declares a harness. +_UNTAGGED_CALL_ADAPTERS = frozenset({gemini._args_call}) + +# The keys and `type` values that name the harness a block came from. A block carrying one is that +# harness's block, well-formed or not. +_HARNESS_TAGS = ("toolUse", "functionCall", "function_call", "content_type", "tool_name", "kind") +_HARNESS_TYPES = frozenset({"tool_use", "function_call"}) + + +def _declares_a_harness(block: dict[str, Any]) -> bool: + """Whether this block identifies which harness's call shape it is.""" + return any(key in block for key in _HARNESS_TAGS) or block.get("type") in _HARNESS_TYPES + + +def _tool_call(block: dict[str, Any]) -> ToolCallBlock | None: + """The tool call this block carries, or None if it is not one. + + A recognizer that matches but reports an unusable name or arguments does not end the search. + Shapes overlap, so the same block can be a malformed instance of one harness's call and a + well-formed instance of another's: the dual-tagged block in this module's docstring carries + both a `toolUse` wrapper and the flat `type: "tool_use"` fields, and a truncated wrapper there + should not hide the flat fields that did survive. + + Falling through would not be safe on its own. Every recognizer but `_args_call` is gated on a + tag that names its harness, so a second reading is a second reading of the same call. + `_args_call` is gated on nothing but the presence of a string `name` and a dict `args`, and + until now only its position at the end of the registry kept it away from blocks a specific + recognizer had already claimed. Reached after a malformed match it would read those two keys off + a block that is not its shape: `{"toolUse": {...}, "name": "other", "args": {...}}` is one + malformed Bedrock call, not a valid bare one, and reporting `other` would attribute a skill the + agent never asked for. So a block that declares a harness is offered only to the recognizers + that read tagged shapes. + """ + tagged_only = _declares_a_harness(block) + for adapter in _CALL_ADAPTERS: + if tagged_only and adapter in _UNTAGGED_CALL_ADAPTERS: + continue + matched = adapter(block) + if matched is None: + continue + call_id, name, arguments = matched + if not isinstance(name, str) or not isinstance(arguments, dict): + continue + return ToolCallBlock(str(call_id) if call_id is not None else None, name, arguments) + return None + + +def _tool_result(block: dict[str, Any]) -> ToolResultBlock | None: + """The tool result this block carries, or None if it is not one.""" + for adapter in _RESULT_ADAPTERS: + matched = adapter(block) + if matched is not None: + raw_result, result_id = matched + refused = _load_refused(raw_result) + return ToolResultBlock( + call_id=str(result_id) if result_id is not None else None, + refused=refused, + body=_body_from_result(raw_result), + error=_refusal_message(raw_result) if refused else None, + ) + return None diff --git a/src/strands_evals/extractors/skills/adapters/strands.py b/src/strands_evals/extractors/skills/adapters/strands.py new file mode 100644 index 00000000..a1f5b2fd --- /dev/null +++ b/src/strands_evals/extractors/skills/adapters/strands.py @@ -0,0 +1,42 @@ +"""Bedrock / Strands native block shapes, and the `strands_evals` typed messages. + +Strands' in-memory message shape is the Bedrock Converse one: an assistant message carries +`{"toolUse": {"toolUseId", "name", "input"}}` and the following user message carries +`{"toolResult": {"toolUseId", "content"}}`. `TraceExtractor` parses those into `ToolCallContent` / +`ToolResultContent`, which serialize to a flatter `content_type`-tagged dict, so both live here. +""" + +from __future__ import annotations + +from typing import Any + +from ._common import CallMatch, ResultMatch + + +def _bedrock_call(block: dict[str, Any]) -> CallMatch | None: + """Bedrock / Strands native: `{"toolUse": {"toolUseId", "name", "input"}}`.""" + raw = block.get("toolUse") + if isinstance(raw, dict): + return raw.get("toolUseId"), raw.get("name"), raw.get("input") + return None + + +def _bedrock_result(block: dict[str, Any]) -> ResultMatch | None: + """Bedrock / Strands native: `{"toolResult": {"toolUseId", "content"}}`.""" + if isinstance(block.get("toolResult"), dict): + return block["toolResult"], block["toolResult"].get("toolUseId") + return None + + +def _typed_call(block: dict[str, Any]) -> CallMatch | None: + """A strands_evals `ToolCallContent`, dumped to a dict.""" + if block.get("content_type") == "tool_use": + return block.get("tool_call_id"), block.get("name"), block.get("arguments") + return None + + +def _typed_result(block: dict[str, Any]) -> ResultMatch | None: + """A strands_evals `ToolResultContent`, dumped to a dict.""" + if block.get("content_type") == "tool_result": + return block, block.get("tool_call_id") + return None diff --git a/src/strands_evals/extractors/skills/extractor.py b/src/strands_evals/extractors/skills/extractor.py new file mode 100644 index 00000000..93d55c3e --- /dev/null +++ b/src/strands_evals/extractors/skills/extractor.py @@ -0,0 +1,690 @@ +"""What the trajectory says the agent was offered and what it loaded. + +This is the harness-independent half: given blocks the adapters have normalized, decide which +calls are skill loads, pair each with its result, and recover the body. That produces a +`SkillLoadEvent` per attempt, which `extract_selected_skills` then folds into one `InvokedSkill` +per skill. The public entry points are `parse_available_skills`, `extract_skill_load_events` and +`extract_selected_skills`. +""" + +from __future__ import annotations + +import json +import logging +from typing import Any + +from ...types.trace import ( + AgentInvocationSpan, + Session, + ToolExecutionSpan, +) +from ._normalize import ( + _as_dict, + _body_from_result, + _canonical_skill_key, + _content_text, + _last_path_segment, + _load_refused, + _parse_available_block, + _refusal_message, + _skill_name_from_args, + _skill_name_from_body, + _skill_path_from_text, +) +from ._patterns import ( + _AVAILABLE_BLOCK, + _AVAILABLE_MARKDOWN, + _CLAUDE_BASE_DIR, + _DISCOVERY_TOOL_NAMES, + _READ_TOOL_NAMES, + _READ_VERBS, + _SED_IN_PLACE, + _SHELL_PREFIX, + _SHELL_SEPARATOR, + _SHELL_TOOL_NAMES, + _SHELL_WRAPPER, +) +from .adapters._common import ToolResultBlock, _iter_indexed_blocks +from .adapters.registry import _tool_call, _tool_result +from .models import AvailableSkill, InvokedSkill, SkillLoadEvent + +logger = logging.getLogger(__name__) + + +# ---- Recognizing a skill read ------------------------------------------------ + + +def _shell_read_skill_path(command: str) -> str | None: + """Return the `SKILL.md` path a shell command reads, or None if it does not read one. + + The verb and the path have to belong to the same command, and the path has to be an operand + of that verb rather than anywhere in the line. Searching the whole command independently + turns writes and unrelated work into phantom skill loads that the judge then scores: + + cat draft.md > /skills/new/SKILL.md # creates a skill, does not load one + sed -i 's/a/b/' /skills/pdf/SKILL.md # edits it + cat data.csv; ls -l /skills/pdf/SKILL.md + """ + if wrapper := _SHELL_WRAPPER.match(command): + command = wrapper.group("inner") + + for segment in _SHELL_SEPARATOR.split(command): + # A redirection target is a write, not a read, whichever verb precedes it. + head = segment.split(">")[0] + while (stripped := _SHELL_PREFIX.sub("", head, count=1)) != head: + head = stripped + parts = head.split() + if not parts: + continue + verb = parts[0].rsplit("/", 1)[-1].casefold() + if verb not in _READ_VERBS: + continue + if verb == "sed" and _SED_IN_PLACE.search(head): + continue # `sed -i` rewrites the file + operands = head[len(parts[0]) :] + if path := _skill_path_from_text(operands): + return path + return None + + +def _skill_read_path(tool_name: str, arguments: dict[str, Any]) -> str | None: + """Return a SKILL.md path only for recognizable file-read operations.""" + lowered = tool_name.casefold() + if lowered in _READ_TOOL_NAMES or any(lowered.endswith(f".{name}") for name in _READ_TOOL_NAMES): + for key in ("path", "file_path", "filename"): + value = arguments.get(key) + if isinstance(value, str) and (path := _skill_path_from_text(value)): + return path + + if lowered in _SHELL_TOOL_NAMES: + command = arguments.get("command") or arguments.get("cmd") + if isinstance(command, str): + return _shell_read_skill_path(command) + return None + + +def _summarize_events(events: list[SkillLoadEvent]) -> list[InvokedSkill]: + """Fold load attempts into one row per skill, in first-attempt order. + + Args: + events: The attempts as they appeared in the trajectory. + + Returns: + list: One `InvokedSkill` per skill, carrying the fullest body recovered for it. + """ + out: list[InvokedSkill] = [] + index_by_key: dict[str, int] = {} + for event in events: + # An attempt whose outcome the trajectory never carried is still a selection the agent + # made, so it is reported. It reads as loaded-without-a-body here, since "the agent asked + # for this skill" is all a per-skill summary can say about it. + summary = InvokedSkill( + name=event.name, + body=event.body, + status="failed" if event.status == "failed" else "loaded", + error=event.error, + ) + key = _canonical_skill_key(event.name) + index = index_by_key.get(key) + if index is None: + index_by_key[key] = len(out) + out.append(summary) + continue + # One success anywhere in the run means the agent got the skill, so a retry after a + # refusal is reported as loaded. Only an all-refused skill stays failed. + if out[index].status == "failed" and summary.status == "loaded": + out[index] = summary + continue + if summary.status == "failed": + continue + # Prefer the fullest body, and with it the name that body declares. A later read only + # wins when it contains what was already recovered, which is what a re-read of the + # same file looks like: a paged window is contained in the whole file. Unrelated + # output that happened to be attributed to this skill is not, so it cannot displace + # a real body just by being longer. + kept = out[index].body or "" + candidate = summary.body or "" + if len(candidate) > len(kept) and kept in candidate: + out[index] = summary + return out + + +# ---- Session path ----------------------------------------------------------- + + +def _available_from_session(session: Session) -> list[AvailableSkill]: + """Recover available skills from the first AgentInvocationSpan.system_prompt that has the block.""" + for trace in session.traces: + for span in trace.spans: + if isinstance(span, AgentInvocationSpan) and span.system_prompt: + skills = _parse_available_block(span.system_prompt) + if skills: + return skills + return [] + + +def _agent_id_of(span: ToolExecutionSpan) -> str | None: + """Which agent ran the span, when the trace records it. + + Read from `metadata` rather than a schema field, since the trace types carry no agent identity + of their own. A multi-agent mapper that records one is honoured; the rest report None, which is + truthful about a single-agent run and about a mapper that dropped the attribution. + """ + metadata = span.metadata or {} + for key in ("agent_id", "agent_name", "agent"): + value = metadata.get(key) + if isinstance(value, str) and value.strip(): + return value.strip() + return None + + +def _events_from_session(session: Session) -> list[SkillLoadEvent]: + """Recover load attempts from ToolExecutionSpans with a reserved skill-tool name. + + The skill body is taken from the tool result content when present. (Some + harnesses put the body elsewhere, e.g. Claude Code's following message; those + are handled by their raw-list adapters and are follow-ups for the Session path.) + + Args: + session: The trace to read. + + Returns: + list: One event per attempt, `position` counting tool spans across the whole session so it + orders attempts made in different traces. + """ + out: list[SkillLoadEvent] = [] + position = 0 + for trace in session.traces: + for span in trace.spans: + if not isinstance(span, ToolExecutionSpan): + continue + position += 1 + call_id = span.tool_result.tool_call_id or span.tool_call.tool_call_id + agent_id = _agent_id_of(span) + failed = bool(span.tool_result.error) + skill_name = _skill_name_from_args(span.tool_call.name, span.tool_call.arguments) + if skill_name is not None: + if failed or _load_refused(span.tool_result.content): + out.append( + SkillLoadEvent( + name=skill_name, + status="failed", + error=span.tool_result.error or _refusal_message(span.tool_result.content), + call_id=call_id, + position=position, + agent_id=agent_id, + ) + ) + else: + out.append( + SkillLoadEvent( + name=skill_name, + status="loaded", + body=_body_from_result(span.tool_result.content), + call_id=call_id, + position=position, + agent_id=agent_id, + ) + ) + continue + + if failed: + continue + read_path = _skill_read_path(span.tool_call.name, span.tool_call.arguments) + body = _body_from_result(span.tool_result.content) + name = _skill_name_from_body(body, read_path) if read_path and body else "" + if name: + out.append( + SkillLoadEvent( + name=name, + status="loaded", + body=body, + call_id=call_id, + position=position, + agent_id=agent_id, + ) + ) + return out + + +# ---- Raw message-list path -------------------------------------------------- +# +# Strands' native in-memory message shape: assistant messages carry +# {"toolUse": {"name", "toolUseId", "input": {...}}} blocks, and the following +# user message carries {"toolResult": {"toolUseId", "content": [...]}}. We also +# accept already-parsed strands_evals message objects (UserMessage/AssistantMessage). + + +def _structured_available_skills(message: Any) -> list[AvailableSkill]: + """Recover structured catalogs, including discovery-tool response wrappers.""" + pending = [message] + seen: set[int] = set() + while pending: + candidate = pending.pop(0) + if id(candidate) in seen: + continue + seen.add(id(candidate)) + + if isinstance(candidate, str) and candidate.lstrip().startswith(("{", "[")): + try: + candidate = json.loads(candidate) + except json.JSONDecodeError: + continue + if isinstance(candidate, list): + pending.extend(candidate) + continue + + value = _as_dict(candidate) + if value is None: + continue + skills = value.get("skills") + if isinstance(skills, list): + out: list[AvailableSkill] = [] + for skill in skills: + if isinstance(skill, str): + out.append(AvailableSkill(skill, "")) + elif isinstance(skill, dict) and skill.get("name"): + out.append(AvailableSkill(str(skill["name"]), str(skill.get("description", "")))) + if out: + return out + pending.extend( + value[key] + for key in ( + "response", + "result", + "output", + "content", + "toolResult", + "functionResponse", + "function_response", + "toolResponse", + ) + if key in value + ) + return [] + + +def _text_candidates(value: Any) -> list[str]: + """Collect prompt/result text recursively without stringifying opaque objects.""" + if isinstance(value, str): + return [value] + if isinstance(value, list): + return [text for item in value for text in _text_candidates(item)] + item = _as_dict(value) + if item is None: + return [] + texts: list[str] = [] + for key in ( + "system_prompt", + # Result wrappers, so a discovery tool's catalog is reachable: harnesses nest the + # payload one level down (Bedrock `toolResult`, Gemini/ADK `functionResponse`). + "toolResult", + "functionResponse", + "function_response", + "toolResponse", + "content", + "text", + "output", + "aggregated_output", + "llmContent", + "instructions", + "message", + "response", + "result", + ): + if key in item: + texts.extend(_text_candidates(item[key])) + return texts + + +def _discovery_tool_name(block: dict[str, Any]) -> str | None: + for candidate in ( + block.get("name"), + block.get("tool_name"), + block.get("functionResponse"), + block.get("function_response"), + block.get("toolResponse"), + ): + if isinstance(candidate, str): + return candidate + value = _as_dict(candidate) + if value is not None and isinstance(value.get("name"), str): + return value["name"] + return None + + +def _result_id(block: dict[str, Any]) -> str | None: + candidates: list[Any] = [ + block.get("tool_call_id"), + block.get("tool_use_id"), + block.get("id"), + ] + for key in ("toolResult", "functionResponse", "function_response", "toolResponse"): + value = _as_dict(block.get(key)) + if value is not None: + candidates.extend( + ( + value.get("toolUseId"), + value.get("tool_call_id"), + value.get("tool_use_id"), + value.get("id"), + ) + ) + return next((str(candidate) for candidate in candidates if candidate is not None), None) + + +def _is_discovery_tool_name(name: str) -> bool: + lowered = name.casefold() + return lowered in _DISCOVERY_TOOL_NAMES or any( + lowered.endswith(f"_{discovery_name}") for discovery_name in _DISCOVERY_TOOL_NAMES + ) + + +def _available_from_list(messages: list[Any]) -> list[AvailableSkill]: + """Parse trusted system catalogs and skill-discovery tool results.""" + indexed_blocks = _iter_indexed_blocks(messages) + discovery_ids = { + call.call_id + for _, _, block in indexed_blocks + if (call := _tool_call(block)) is not None and _is_discovery_tool_name(call.name) and call.call_id is not None + } + + for msg in messages: + outer = _as_dict(msg) + if outer is None: + continue + message = _as_dict(outer.get("message")) or outer + role = str(message.get("role") or outer.get("role") or "").casefold() + is_system = role in {"system", "developer"} or str(outer.get("type", "")).casefold() == "system" + if is_system: + structured = _structured_available_skills(msg) + if structured: + return structured + for text in _text_candidates(msg): + skills = _parse_available_block(text) + if skills: + return skills + elif "system_prompt" in outer: + for text in _text_candidates(outer["system_prompt"]): + skills = _parse_available_block(text) + if skills: + return skills + + for _, _, block in indexed_blocks: + tool_name = _discovery_tool_name(block) + is_discovery_result = (isinstance(tool_name, str) and _is_discovery_tool_name(tool_name)) or _result_id( + block + ) in discovery_ids + if not is_discovery_result: + continue + structured = _structured_available_skills(block) + if structured: + return structured + for text in _text_candidates(block): + skills = _parse_available_block(text) + if skills: + return skills + return [] + + +def _claude_body_after( + indexed_blocks: list[tuple[int, str | None, dict[str, Any]]], + call_index: int, + skill_name: str, +) -> str | None: + """Find Claude Code's injected skill body after its launch acknowledgement. + + The body is matched to the call by the skill directory named on the `Base directory` line, + not by position: Claude Code can launch several skills in one assistant turn, and then the + injected bodies arrive in an order the call order does not fix. Taking the first block after + the call gives every skill in the turn the first skill's instructions, which the adherence + judge would then score against the wrong steps. + """ + candidates: list[tuple[str, str]] = [] # (base directory, full injected text) + for index, role, block in indexed_blocks: + if index <= call_index or role not in (None, "user"): + continue + text = _content_text(block.get("text") if block.get("type") == "text" else block) + if match := _CLAUDE_BASE_DIR.match(text.lstrip()): + candidates.append((_last_path_segment(match.group("path")), text)) + + wanted = _canonical_skill_key(skill_name) + for directory, text in candidates: + if _canonical_skill_key(directory) == wanted: + return text + # No directory matched. With one candidate that is still this call's body, since the directory + # can be an alias for the name in the frontmatter. With several it is unknowable which belongs + # to this call, and guessing would attribute another skill's instructions to it. + return candidates[0][1] if len(candidates) == 1 else None + + +def _events_from_list(messages: list[Any]) -> list[SkillLoadEvent]: + """Recover load attempts from raw or typed message lists. + + Matches assistant `toolUse` blocks with a reserved skill-tool name, then + pairs each with the `toolResult` block (by toolUseId) that carries the body. + Typed `ToolCallContent` / `ToolResultContent` blocks use the equivalent + `content_type` and `tool_call_id` fields. + + Args: + messages: The raw message list. + + Returns: + list: One event per attempt, `position` being the index in `messages` the attempt was + found at. An attempt whose result the list never carries is reported as "attempted". + """ + indexed_blocks = _iter_indexed_blocks(messages) + results_by_id: dict[str, ToolResultBlock] = {} + unkeyed_results: list[tuple[int, ToolResultBlock]] = [] + for result_index, _, block in indexed_blocks: + parsed_result = _tool_result(block) + if parsed_result is None: + continue + if parsed_result.call_id is not None: + results_by_id[parsed_result.call_id] = parsed_result + else: + unkeyed_results.append((result_index, parsed_result)) + + # Every tool call's position, so an unkeyed result is only paired with the call it follows + # directly. Without that bound the next unclaimed result wins, and an unrelated tool's output + # in between is attributed to the skill: the adherence judge then scores the agent against a + # weather report instead of the skill's steps. + call_indices = sorted({index for index, _, block in indexed_blocks if _tool_call(block) is not None}) + + out: list[SkillLoadEvent] = [] + used_unkeyed_results: set[int] = set() + for message_index, _, block in indexed_blocks: + if block.get("type") == "command_execution": + command = block.get("command") + body = _body_from_result(block) + path = _shell_read_skill_path(command) if isinstance(command, str) else None + name = _skill_name_from_body(body, path) if path and body else "" + if name: + out.append( + SkillLoadEvent( + name=name, + status="loaded", + body=body, + position=message_index, + ) + ) + continue + + call = _tool_call(block) + if call is None: + continue + matched_result = results_by_id.get(call.call_id) if call.call_id is not None else None + if matched_result is None and call.call_id is None: + next_call_index = next((index for index in call_indices if index > message_index), None) + unkeyed_match = next( + ( + (index, result) + for index, result in unkeyed_results + if message_index < index and index not in used_unkeyed_results + if next_call_index is None or index < next_call_index + ), + None, + ) + if unkeyed_match is not None: + used_unkeyed_results.add(unkeyed_match[0]) + matched_result = unkeyed_match[1] + + skill_name = _skill_name_from_args(call.name, call.arguments) + if skill_name is not None: + if matched_result is not None and matched_result.refused: + out.append( + SkillLoadEvent( + name=skill_name, + status="failed", + error=matched_result.error, + call_id=call.call_id, + position=message_index, + ) + ) + continue + body = matched_result.body if matched_result is not None else None + if call.name == "Skill" and body is None: + body = _claude_body_after(indexed_blocks, message_index, skill_name) + out.append( + SkillLoadEvent( + name=skill_name, + # No result and no injected body means the trajectory stops before the + # outcome, a different run from one that loaded and whose body went uncaptured. + status="attempted" if matched_result is None and body is None else "loaded", + body=body, + call_id=call.call_id, + position=message_index, + ) + ) + continue + + read_path = _skill_read_path(call.name, call.arguments) + if read_path is None or matched_result is None or matched_result.refused or not matched_result.body: + continue + # An unnamed read is not a skill load: see `_skill_name_from_body`. + name = _skill_name_from_body(matched_result.body, read_path) + if name: + out.append( + SkillLoadEvent( + name=name, + status="loaded", + body=matched_result.body, + call_id=call.call_id, + position=message_index, + ) + ) + return out + + +# ---- Public API ------------------------------------------------------------- + + +def parse_available_skills(trajectory: Session | list[Any] | str | None) -> list[AvailableSkill]: + """Return the skills exposed to the agent (name + description). + + Args: + trajectory: A `Session`, a raw message list, or a bare prompt string (e.g. a harness's + system prompt, which is where the block is injected but which some session mappers + store separately from the message list). + + Returns: + list: The advertised skills, or [] when no `` block is found. + """ + if isinstance(trajectory, Session): + return _available_from_session(trajectory) + if isinstance(trajectory, str): + return _parse_available_block(trajectory) + if isinstance(trajectory, list): + return _available_from_list(trajectory) + if trajectory is not None: + logger.debug("type=<%s> | unsupported trajectory type for available skills", type(trajectory).__name__) + return [] + + +def advertised_a_catalog(trajectory: Session | list[Any] | str | None) -> bool: + """Whether the run recorded an available-skills block at all, empty or not. + + `parse_available_skills` returns [] for two different runs: one where the harness advertised + no skills, and one where the harness never records what it offered. Callers that show the + offered set to a judge need to tell those apart, because an empty set is a claim about the run + while a missing one is a gap in the telemetry. + + Args: + trajectory: A `Session`, a raw message list, or a bare prompt string. + + Returns: + bool: True when a block was present, whether or not it listed any skills. + """ + for text in _catalog_texts(trajectory): + if _AVAILABLE_BLOCK.search(text) or _AVAILABLE_MARKDOWN.search(text): + return True + return False + + +def _catalog_texts(trajectory: Session | list[Any] | str | None) -> list[str]: + """Every place a harness is known to put the available-skills block.""" + if trajectory is None: + return [] + if isinstance(trajectory, str): + return [trajectory] + if isinstance(trajectory, Session): + return [ + span.system_prompt + for trace in trajectory.traces + for span in trace.spans + if isinstance(span, AgentInvocationSpan) and span.system_prompt + ] + if isinstance(trajectory, list): + texts: list[str] = [] + for message in trajectory: + outer = _as_dict(message) + if outer is None: + continue + content = outer.get("content") + if isinstance(content, str): + texts.append(content) + elif isinstance(content, list): + texts.extend( + block["text"] + for item in content + if (block := _as_dict(item)) is not None and isinstance(block.get("text"), str) + ) + return texts + return [] + + +def extract_skill_load_events(trajectory: Session | list[Any] | None) -> list[SkillLoadEvent]: + """Return every skill load attempt, in trajectory order. + + The harness-independent form, before any folding: repeated loads of one skill are separate + events, and a refusal followed by a successful retry is two. Use this where the individual + attempts matter (how often a skill was reloaded, which agent loaded it, whether an attempt's + outcome was ever recorded); `extract_selected_skills` gives the per-skill summary. + + Args: + trajectory: A `Session` or a raw message list. + + Returns: + list: One `SkillLoadEvent` per attempt, or [] when the trajectory carries none. + """ + if isinstance(trajectory, Session): + return _events_from_session(trajectory) + if isinstance(trajectory, list): + return _events_from_list(trajectory) + if trajectory is not None: + logger.debug("type=<%s> | unsupported trajectory type for skill load events", type(trajectory).__name__) + return [] + + +def extract_selected_skills(trajectory: Session | list[Any] | None) -> list[InvokedSkill]: + """Return the skills the agent selected, one row per skill, in first-attempt order. + + Args: + trajectory: A `Session` or a raw message list. + + Returns: + list: One `InvokedSkill` per skill, carrying the `SKILL.md` body when the trajectory + surfaced it (else `None`) and `status="failed"` with the harness's message when every + attempt was refused. + """ + return _summarize_events(extract_skill_load_events(trajectory)) diff --git a/src/strands_evals/extractors/skills/models.py b/src/strands_evals/extractors/skills/models.py new file mode 100644 index 00000000..497532c2 --- /dev/null +++ b/src/strands_evals/extractors/skills/models.py @@ -0,0 +1,57 @@ +"""The types the skill extractors return.""" + +from __future__ import annotations + +from typing import Literal, NamedTuple + + +class AvailableSkill(NamedTuple): + """A skill exposed to the agent at runtime.""" + + name: str + description: str + + +class SkillLoadEvent(NamedTuple): + """One load attempt, as it appeared in the trajectory. + + The harness-independent form every adapter converts its own messages into. One event per + attempt, in trajectory order, before any judgment about what the attempts add up to: two loads + of the same skill are two events, and a refusal followed by a retry is two events rather than + one outcome. `InvokedSkill` is the per-skill summary built from these, so evaluators that need + their own definition of selected, invoked, or followed read the events instead. + """ + + name: str + # "attempted" means the call was made and the trajectory never carried its outcome, which is + # not the same as a load that succeeded and whose body went uncaptured: one is a run we cannot + # see the end of, the other is a run we saw succeed. "failed" is a refusal the harness reported. + status: Literal["attempted", "loaded", "failed"] + body: str | None = None # SKILL.md text if captured from the trajectory, else None + error: str | None = None # the harness's message, on a refusal + call_id: str | None = None # the harness's tool-call identifier, when it keys its results + position: int | None = None # index of the message or span the attempt was found in + agent_id: str | None = None # which agent made the attempt, when the harness records it + + +class InvokedSkill(NamedTuple): + """A skill the agent selected during the run, whether or not the load succeeded. + + One row per skill, folded from the `SkillLoadEvent`s for that skill: repeated loads collapse, + the fullest body recovered wins, and one success anywhere makes the skill loaded. Read + `extract_skill_load_events` instead where the individual attempts matter. + """ + + name: str + body: str | None # SKILL.md text if captured from the trajectory, else None + # "failed" means the harness refused the load (unknown skill, sandbox error). Kept rather than + # dropped because a refused load and no attempt at all are different runs: the agent that asked + # for the right skill and was refused made a correct selection, and reporting it as an + # abstention credits or blames the wrong decision. + status: Literal["loaded", "failed"] = "loaded" + # The harness's own refusal message, on a failed load. Which refusal it was decides what to + # fix: "Skill 'pdf-procesing' not found. Available skills: pdf-processing" is a misspelled + # name in the agent's call, while "Available skills: (none)" is a harness that mounted no + # skills at all. Collapsing both into "the load failed" hides that difference from whoever + # reads the result. + error: str | None = None diff --git a/src/strands_evals/mappers/openinference_session_mapper.py b/src/strands_evals/mappers/openinference_session_mapper.py index 6352357d..7974fb10 100644 --- a/src/strands_evals/mappers/openinference_session_mapper.py +++ b/src/strands_evals/mappers/openinference_session_mapper.py @@ -299,6 +299,12 @@ def _build_trace(self, trace_id: str, spans: list[dict], session_id: str) -> Tra if isinstance(converted, AgentInvocationSpan) and not converted.available_tools: converted.available_tools = tools_list + system_prompt = self._trace_system_prompt_map.get(trace_id) + if system_prompt: + for converted in converted_spans: + if isinstance(converted, AgentInvocationSpan) and not converted.system_prompt: + converted.system_prompt = system_prompt + return Trace(spans=converted_spans, trace_id=trace_id, session_id=session_id) # ========================================================================= @@ -603,13 +609,18 @@ def _convert_agent_invocation_span(self, span: dict, session_id: str) -> AgentIn if isinstance(output_value, str) and output_value: agent_response = output_value - # LangGraph / ADOT: extract from structured messages - if not user_prompt or not agent_response: - input_messages, output_messages = self._get_messages_from_span_events(span) - if not user_prompt: - user_prompt = self._extract_user_prompt(input_messages, span) - if not agent_response: - agent_response = self._extract_agent_response(output_messages, span) + # LangGraph / ADOT: extract from structured messages. Parsed unconditionally + # (the result is cached) because the system prompt lives here even when the + # prompt and response were already recovered from smolagents attributes. + input_messages, output_messages = self._get_messages_from_span_events(span) + if not user_prompt: + user_prompt = self._extract_user_prompt(input_messages, span) + if not agent_response: + agent_response = self._extract_agent_response(output_messages, span) + + _, span_system_prompt = self._extract_user_contents(input_messages, span) + if span_system_prompt: + self._trace_system_prompt_map[trace_id] = span_system_prompt if not user_prompt: logger.warning(f"No user_prompt for agent span {span.get('span_id')}") @@ -638,6 +649,7 @@ def _convert_agent_invocation_span(self, span: dict, session_id: str) -> AgentIn user_prompt=user_prompt, agent_response=agent_response, available_tools=available_tools, + system_prompt=self._trace_system_prompt_map.get(trace_id) or None, metadata=metadata, ) @@ -759,6 +771,11 @@ def _extract_messages_from_span(self, span: dict) -> tuple[list[dict], list[dict event_name = event.get("event_name", "") if event_name in SCOPES_OPENINFERENCE_FAMILY: body = event.get("body", {}) + if not isinstance(body, dict): + # This path is now walked for every span, to reach the system prompt, so a + # malformed body here would raise and cost the caller the whole span rather + # than just its messages. Matches `cloudwatch_parser`'s own guard. + continue input_group = body.get("input", {}) output_group = body.get("output", {}) input_msgs = input_group.get("messages", []) diff --git a/src/strands_evals/mappers/strands_in_memory_session_mapper.py b/src/strands_evals/mappers/strands_in_memory_session_mapper.py index 54ac6910..e498b0d7 100644 --- a/src/strands_evals/mappers/strands_in_memory_session_mapper.py +++ b/src/strands_evals/mappers/strands_in_memory_session_mapper.py @@ -139,6 +139,10 @@ def _use_latest_conventions(self) -> bool: def _convert_trace(self, trace_id: str, otel_spans: list[ReadableSpan], session_id: str) -> Trace: converted_spans: list[InferenceSpan | ToolExecutionSpan | AgentInvocationSpan] = [] + system_prompt = next( + (prompt for span in otel_spans if (prompt := self._extract_system_prompt(span))), + None, + ) for span in otel_spans: try: @@ -164,6 +168,11 @@ def _convert_trace(self, trace_id: str, otel_spans: list[ReadableSpan], session_ bridge_parent_gaps(converted_spans, raw_parent_map) + if system_prompt: + for converted_span in converted_spans: + if isinstance(converted_span, AgentInvocationSpan) and not converted_span.system_prompt: + converted_span.system_prompt = system_prompt + return Trace(spans=converted_spans, trace_id=trace_id, session_id=session_id) def _create_span_info(self, span: ReadableSpan, session_id: str) -> SpanInfo: @@ -186,6 +195,43 @@ def _parse_json_attr(self, attributes: Any, key: str, default: str = "[]") -> An except (AttributeError, TypeError, json.JSONDecodeError): return json.loads(default) + def _prompt_content_to_text(self, value: Any) -> str: + """Flatten legacy content blocks or latest OTEL instruction parts.""" + if isinstance(value, str): + try: + decoded = json.loads(value) + except json.JSONDecodeError: + return value + return self._prompt_content_to_text(decoded) + if isinstance(value, list): + return "\n".join(part for item in value if (part := self._prompt_content_to_text(item))) + if isinstance(value, dict): + for key in ("text", "content"): + if key in value: + return self._prompt_content_to_text(value[key]) + return "" + + def _extract_system_prompt(self, span: ReadableSpan) -> str | None: + """Extract a model-visible system prompt from either GenAI convention.""" + attributes = span.attributes or {} + direct = attributes.get("gen_ai.system_instructions") + if direct: + prompt = self._prompt_content_to_text(direct).strip() + if prompt: + return prompt + + for event in span.events: + event_attributes = event.attributes or {} + if event.name == "gen_ai.system.message": + prompt = self._prompt_content_to_text(event_attributes.get("content")).strip() + elif event.name == "gen_ai.client.inference.operation.details": + prompt = self._prompt_content_to_text(event_attributes.get("gen_ai.system_instructions")).strip() + else: + continue + if prompt: + return prompt + return None + def _process_user_message(self, content_list: list[dict[str, Any]]) -> list[TextContent | ToolResultContent]: return [TextContent(text=item["text"]) for item in content_list if "text" in item] @@ -466,5 +512,6 @@ def _convert_agent_invocation_span(self, span: ReadableSpan, session_id: str) -> user_prompt=user_prompt, agent_response=agent_response, available_tools=available_tools, + system_prompt=self._extract_system_prompt(span), metadata={}, ) diff --git a/src/strands_evals/types/__init__.py b/src/strands_evals/types/__init__.py index 8d38173a..56d67122 100644 --- a/src/strands_evals/types/__init__.py +++ b/src/strands_evals/types/__init__.py @@ -7,11 +7,21 @@ RCAOutput, RCAStructuredOutput, ) -from .evaluation import EnvironmentState, EvaluationData, EvaluationOutput, InputT, Interaction, OutputT, TaskOutput +from .evaluation import ( + NOT_APPLICABLE, + EnvironmentState, + EvaluationData, + EvaluationOutput, + InputT, + Interaction, + OutputT, + TaskOutput, +) from .multimodal import AnyMediaData, ImageData, MultimodalInput, resolve_image_bytes from .simulation import ActorProfile, ActorResponse __all__ = [ + "NOT_APPLICABLE", "EnvironmentState", "Interaction", "TaskOutput", diff --git a/src/strands_evals/types/evaluation.py b/src/strands_evals/types/evaluation.py index 4e25f3b0..e7295f6b 100644 --- a/src/strands_evals/types/evaluation.py +++ b/src/strands_evals/types/evaluation.py @@ -109,6 +109,17 @@ class EvaluationData(BaseModel, Generic[InputT, OutputT]): expected_environment_state: list[EnvironmentState] | None = None +NOT_APPLICABLE = "not_applicable" +"""`EvaluationOutput.label` for a row that had nothing to judge. + +An evaluator emits this when the case gave it no decision to score, so the row carries a +diagnosis in `reason` rather than a verdict. Its `score` is 0.0 by convention and is not a +judgment: `EvaluationReport.calculate_overall_score` drops these rows so they do not deflate the +mean. Anything reporting an average has to drop them the same way, or the same rows read as two +different numbers depending on where they are displayed. +""" + + class EvaluationOutput(BaseModel): """ Structured output for LLM-based judge. @@ -117,10 +128,16 @@ class EvaluationOutput(BaseModel): score: The score of the test case. test_pass: Whether the test pass or fail. reason: The reason for the score for each test case. - label: The categorical label corresponding to the score. + label: The categorical label corresponding to the score, or `NOT_APPLICABLE` when there + was nothing to judge. """ score: float test_pass: bool reason: str | None = None label: str | None = None + + @property + def not_applicable(self) -> bool: + """Whether this row had nothing to judge, so its score is not a verdict.""" + return self.label == NOT_APPLICABLE diff --git a/src/strands_evals/types/evaluation_report.py b/src/strands_evals/types/evaluation_report.py index 8ec53a1e..f3ecec2e 100644 --- a/src/strands_evals/types/evaluation_report.py +++ b/src/strands_evals/types/evaluation_report.py @@ -29,6 +29,38 @@ class EvaluationReport(BaseModel): diagnoses: list[dict | None] = [] recommendations: list[str | None] = [] + @staticmethod + def is_applicable(outputs: list[EvaluationOutput]) -> bool: + """Whether a case's rows carry a verdict, so its score belongs in an average. + + A case is dropped only when every row both declined to judge and passed. One judged row is + enough to make the case's score a real number. + + `test_pass` is what separates the two things a not-applicable row can mean. Declining to + judge passes and is droppable: nothing was on offer, so there is no verdict to average. + Failing to judge does not pass, and dropping it would take a real failure out of every + average and report the run as better than it was. Both evaluators emit the second shape for + a missing trajectory. + + No rows at all is applicable for the same reason. An evaluator that returned nothing failed + to judge rather than declining to, and `_default_aggregator` scores that `test_pass=False`. + """ + return not outputs or any(not output.not_applicable or not output.test_pass for output in outputs) + + @classmethod + def calculate_overall_score( + cls, + scores: list[float], + detailed_results: list[list[EvaluationOutput]], + ) -> float: + """Average applicable rows while retaining N/A rows in report details.""" + applicable_scores = [ + score + for index, score in enumerate(scores) + if index >= len(detailed_results) or cls.is_applicable(detailed_results[index]) + ] + return sum(applicable_scores) / len(applicable_scores) if applicable_scores else 0.0 + @classmethod def flatten(cls, reports: list["EvaluationReport"]) -> "EvaluationReport": """Concatenate multiple evaluation reports into one. @@ -53,7 +85,7 @@ def flatten(cls, reports: list["EvaluationReport"]) -> "EvaluationReport": recs.append(report.recommendations[i] if i < len(report.recommendations) else None) return cls( - overall_score=sum(scores) / len(scores) if scores else 0.0, + overall_score=cls.calculate_overall_score(scores, detailed), scores=scores, cases=cases, test_passes=passes, diff --git a/tests/strands_evals/cli/test_entrypoint.py b/tests/strands_evals/cli/test_entrypoint.py index fd68fd4d..cca04d93 100644 --- a/tests/strands_evals/cli/test_entrypoint.py +++ b/tests/strands_evals/cli/test_entrypoint.py @@ -12,7 +12,9 @@ classify_agent, classify_task, import_attr, + resolve_evaluator_spec, ) +from strands_evals.evaluators import SkillInstructionFollowingEvaluator, SkillSelectionAccuracyEvaluator def test_import_attr_resolves_module_function(): @@ -235,3 +237,9 @@ def too_many(case, extra): with pytest.raises(EntryPointError, match="2 positional arguments"): classify_task(too_many, "x:y") + + +def test_resolve_evaluator_spec_skill_shortnames(): + """The skill judges are reachable from `--evaluator` like the other zero-arg built-ins.""" + assert isinstance(resolve_evaluator_spec("skill-selection-accuracy"), SkillSelectionAccuracyEvaluator) + assert isinstance(resolve_evaluator_spec("skill-instruction-following"), SkillInstructionFollowingEvaluator) diff --git a/tests/strands_evals/cli/test_run.py b/tests/strands_evals/cli/test_run.py index eeaddaa4..9378f772 100644 --- a/tests/strands_evals/cli/test_run.py +++ b/tests/strands_evals/cli/test_run.py @@ -7,7 +7,10 @@ import pytest +from strands_evals.cli.commands.run import _display_expanded, _print_summary from strands_evals.cli.main import main +from strands_evals.types import NOT_APPLICABLE, EvaluationOutput +from strands_evals.types.evaluation_report import EvaluationReport def test_run_with_task_smoke(experiment_file: Path, capsys, tmp_path: Path): @@ -748,3 +751,80 @@ def test_run_custom_evaluator_threading(experiment_file: Path, tmp_path: Path): payload = json.loads(out_path.read_text()) assert payload["cases"][0]["evaluator"] == "AlwaysPasses" assert payload["test_passes"] == [True] + + +def _report_with_one_not_applicable_case() -> EvaluationReport: + """A two-case report where the second case had nothing to judge. + + Its 0.0 is a placeholder, not a verdict, so `calculate_overall_score` leaves it out and the + overall score is the first case's 1.0. + """ + judged = [EvaluationOutput(score=1.0, test_pass=True, reason="judged", label="Yes")] + nothing_to_judge = [EvaluationOutput(score=0.0, test_pass=True, reason="no skill invoked", label=NOT_APPLICABLE)] + scores = [1.0, 0.0] + detailed = [judged, nothing_to_judge] + return EvaluationReport( + overall_score=EvaluationReport.calculate_overall_score(scores, detailed), + scores=scores, + cases=[ + {"name": "used a skill", "evaluator": "SkillInstructionFollowingEvaluator"}, + {"name": "used none", "evaluator": "SkillInstructionFollowingEvaluator"}, + ], + test_passes=[True, True], + reasons=["judged", "no skill invoked"], + detailed_results=detailed, + ) + + +def test_summary_average_matches_overall_score(capsys): + """The per-evaluator average and `overall:` are computed from the same rows. + + Averaging the not-applicable case's placeholder 0.0 into the per-evaluator number printed + `(0.50)` next to `overall: 1.00` off the very same two rows. + """ + report = _report_with_one_not_applicable_case() + + _print_summary(report) + + summary = capsys.readouterr().err + assert "SkillInstructionFollowingEvaluator: 2/2 passed (1.00)" in summary + assert "overall: 1.00" in summary + + +def test_summary_reports_zero_when_every_case_is_not_applicable(capsys): + """No judged rows means no average to print, not a failing score dressed up as one.""" + outputs = [EvaluationOutput(score=0.0, test_pass=True, reason="no skill invoked", label=NOT_APPLICABLE)] + report = EvaluationReport( + overall_score=EvaluationReport.calculate_overall_score([0.0], [outputs]), + scores=[0.0], + cases=[{"name": "used none", "evaluator": "SkillInstructionFollowingEvaluator"}], + test_passes=[True], + reasons=["no skill invoked"], + detailed_results=[outputs], + ) + + _print_summary(report) + + summary = capsys.readouterr().err + assert "SkillInstructionFollowingEvaluator: 1/1 passed (0.00)" in summary + assert "overall: 0.00" in summary + + +def test_expanded_display_shows_na_instead_of_a_placeholder_zero(monkeypatch): + """The score column of an unjudged case reads "n/a", not the worst possible verdict.""" + from strands_evals.cli.commands import run as run_module + + captured: dict = {} + + class _FakeDisplay: + def __init__(self, items: dict, overall_score: float) -> None: + captured["items"] = items + + def run(self, static: bool = True) -> None: + pass + + monkeypatch.setattr(run_module, "CollapsibleTableReportDisplay", _FakeDisplay) + + _display_expanded(_report_with_one_not_applicable_case(), include_recommendations=False) + + assert [row["details"]["score"] for row in captured["items"].values()] == ["1.00", "n/a"] diff --git a/tests/strands_evals/evaluators/deterministic/test_skill_invoked.py b/tests/strands_evals/evaluators/deterministic/test_skill_invoked.py new file mode 100644 index 00000000..9e85408c --- /dev/null +++ b/tests/strands_evals/evaluators/deterministic/test_skill_invoked.py @@ -0,0 +1,150 @@ +import pytest + +from strands_evals.evaluators.deterministic import SkillInvoked +from strands_evals.types.evaluation import EvaluationData + + +def _case(invoked_skill: str | None): + messages = [] + if invoked_skill: + messages.append( + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "t", "name": "skills", "input": {"skill_name": invoked_skill}}}], + } + ) + messages.append( + {"role": "user", "content": [{"toolResult": {"toolUseId": "t", "content": [{"text": "# body"}]}}]} + ) + return EvaluationData(input="do pdf", actual_output="done", actual_trajectory=messages) + + +def test_skill_invoked_present(): + result = SkillInvoked(skill_name="pdf-processing").evaluate(_case("pdf-processing")) + assert len(result) == 1 + assert result[0].score == 1.0 + assert result[0].test_pass is True + + +def test_skill_invoked_absent(): + result = SkillInvoked(skill_name="pdf-processing").evaluate(_case("other-skill")) + assert result[0].score == 0.0 + assert result[0].test_pass is False + + +def test_skill_invoked_no_skill(): + result = SkillInvoked(skill_name="pdf-processing").evaluate(_case(None)) + assert result[0].score == 0.0 + assert result[0].test_pass is False + + +def test_refused_load_does_not_count_as_invoked(): + """The agent asked for the skill and the harness refused, so it was never used.""" + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "t", "name": "skills", "input": {"skill_name": "pdf-processing"}}}], + }, + { + "role": "user", + "content": [ + {"toolResult": {"toolUseId": "t", "status": "error", "content": [{"text": "skill not found"}]}} + ], + }, + ] + case = EvaluationData(input="do pdf", actual_output="done", actual_trajectory=messages) + + result = SkillInvoked(skill_name="pdf-processing").evaluate(case) + + assert result[0].score == 0.0 + assert result[0].test_pass is False + # Distinguished from never reaching for the skill, which needs a different fix, and carrying + # what the harness said so a misspelled name is distinguishable from a harness that mounted none. + assert result[0].reason == "skill 'pdf-processing' was requested but the load failed: skill not found" + + +def test_skill_invoked_no_trajectory(): + case = EvaluationData(input="x", actual_output="y", actual_trajectory=None) + result = SkillInvoked(skill_name="pdf-processing").evaluate(case) + assert result[0].score == 0.0 + assert "no trajectory" in result[0].reason + + +@pytest.mark.asyncio +async def test_skill_invoked_async(): + result = await SkillInvoked(skill_name="pdf-processing").evaluate_async(_case("pdf-processing")) + assert result[0].score == 1.0 + + +def test_a_refusal_then_a_success_still_counts_as_invoked(): + """The agent retried and got the skill, so the assertion that it was used holds.""" + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "a", "name": "skills", "input": {"skill_name": "pdf-processing"}}}], + }, + { + "role": "user", + "content": [{"toolResult": {"toolUseId": "a", "status": "error", "content": [{"text": "sandbox busy"}]}}], + }, + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "b", "name": "skills", "input": {"skill_name": "pdf-processing"}}}], + }, + {"role": "user", "content": [{"toolResult": {"toolUseId": "b", "content": [{"text": "# body"}]}}]}, + ] + case = EvaluationData(input="do pdf", actual_output="done", actual_trajectory=messages) + + result = SkillInvoked(skill_name="pdf-processing").evaluate(case) + + assert result[0].test_pass is True + assert result[0].reason == "skill 'pdf-processing' was invoked" + + +def test_repeated_refusals_report_how_many_attempts_were_made(): + """One refusal and five are different runs: the second is an agent stuck in a retry loop.""" + messages = [] + for index in range(3): + messages.append( + { + "role": "assistant", + "content": [ + { + "toolUse": { + "toolUseId": f"t{index}", + "name": "skills", + "input": {"skill_name": "pdf-processing"}, + } + } + ], + } + ) + messages.append( + { + "role": "user", + "content": [ + {"toolResult": {"toolUseId": f"t{index}", "status": "error", "content": [{"text": "not found"}]}} + ], + } + ) + case = EvaluationData(input="do pdf", actual_output="done", actual_trajectory=messages) + + result = SkillInvoked(skill_name="pdf-processing").evaluate(case) + + assert result[0].test_pass is False + assert result[0].reason == ("skill 'pdf-processing' was requested but the load failed (3 attempts): not found") + + +def test_a_request_with_no_recorded_outcome_is_named_apart_from_never_asking(): + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "t", "name": "skills", "input": {"skill_name": "pdf-processing"}}}], + } + ] + case = EvaluationData(input="do pdf", actual_output="done", actual_trajectory=messages) + + result = SkillInvoked(skill_name="pdf-processing").evaluate(case) + + assert result[0].test_pass is False + assert result[0].reason == "skill 'pdf-processing' was requested but the trajectory records no outcome" diff --git a/tests/strands_evals/evaluators/test_skill_instruction_following_evaluator.py b/tests/strands_evals/evaluators/test_skill_instruction_following_evaluator.py new file mode 100644 index 00000000..102e2dd9 --- /dev/null +++ b/tests/strands_evals/evaluators/test_skill_instruction_following_evaluator.py @@ -0,0 +1,405 @@ +from unittest.mock import Mock, patch + +import pytest + +from strands_evals.evaluators.skill_instruction_following_evaluator import ( + SkillFollowingRating, + SkillFollowingScore, + SkillInstructionFollowingEvaluator, + SkillStepRating, + _strip_frontmatter, +) +from strands_evals.extractors import extract_selected_skills +from strands_evals.types.evaluation import NOT_APPLICABLE, EvaluationData, EvaluationOutput + +_MODULE = "strands_evals.evaluators.skill_instruction_following_evaluator.Agent" + +SKILL_BODY = "# PDF Processing Skill\n1. Identify the PDF path.\n2. Extract text.\n3. Summarize." + + +def _ordinal_for(coverage: float) -> SkillFollowingScore: + """A reasonable ordinal for a coverage fraction, for building test ratings.""" + if coverage >= 0.95: + return SkillFollowingScore.FULLY_FOLLOWED + if coverage >= 0.75: + return SkillFollowingScore.MOSTLY_FOLLOWED + if coverage >= 0.5: + return SkillFollowingScore.PARTIALLY_FOLLOWED + if coverage >= 0.25: + return SkillFollowingScore.MINIMALLY_FOLLOWED + return SkillFollowingScore.NOT_FOLLOWED + + +def _rating( + *statuses: str, + reasoning: str = "step evidence", + score: SkillFollowingScore | None = None, +) -> SkillFollowingRating: + steps = [ + SkillStepRating(step=f"step {i}", status=status, evidence=f"evidence {i}") + for i, status in enumerate(statuses, start=1) + ] + coverage = sum({"covered": 1.0, "partial": 0.5, "skipped": 0.0}[s] for s in statuses) / len(statuses) + return SkillFollowingRating( + reasoning=reasoning, + steps=steps, + score=score if score is not None else _ordinal_for(coverage), + ) + + +def _case(invoked: list[tuple[str, str]]): + """invoked: list of (skill_name, body) -> Strands message list with toolUse/toolResult pairs.""" + messages = [] + for i, (name, body) in enumerate(invoked): + tid = f"t{i}" + messages.append( + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": tid, "name": "skills", "input": {"skill_name": name}}}], + } + ) + messages.append({"role": "user", "content": [{"toolResult": {"toolUseId": tid, "content": [{"text": body}]}}]}) + return EvaluationData(input="do pdf", actual_output="done", actual_trajectory=messages) + + +def test_strip_frontmatter(): + raw = "---\nname: pdf\ndescription: x\n---\n\n# Body\n1. step" + assert _strip_frontmatter(raw) == "# Body\n1. step" + # no frontmatter -> unchanged + assert _strip_frontmatter("# Body\n1. step") == "# Body\n1. step" + + +def test_init_defaults(): + ev = SkillInstructionFollowingEvaluator() + assert ev.version == "v0" + # aggregator overridden to drop N/A rows + assert ev.aggregator == ev._aggregate_dropping_na + + +def test_rating_derives_coverage_from_step_statuses(): + assert _rating("covered", "partial", "skipped").coverage == 0.5 + + +def test_rating_accepts_no_steps(): + """A skill body that prescribes nothing is a valid judgment, not a schema violation. + + `steps` is the structured-output schema, so rejecting an empty list sends the judge back to + re-emit the same answer until the retry loop exhausts the recursion limit. + """ + rating = SkillFollowingRating(reasoning="reference material only", steps=[], score=_ordinal_for(0.0)) + + assert rating.steps == [] + assert rating.coverage == 0.0 # vacuous, not a failure; callers check `steps` to tell them apart + + +def test_score_mapping_is_five_point_ordinal(): + ev = SkillInstructionFollowingEvaluator() + assert ev._score_mapping == { + SkillFollowingScore.FULLY_FOLLOWED: 1.0, + SkillFollowingScore.MOSTLY_FOLLOWED: 0.75, + SkillFollowingScore.PARTIALLY_FOLLOWED: 0.5, + SkillFollowingScore.MINIMALLY_FOLLOWED: 0.25, + SkillFollowingScore.NOT_FOLLOWED: 0.0, + } + + +@pytest.mark.parametrize("field", ["step", "evidence"]) +def test_step_rating_accepts_empty_text(field): + """No length floor on the judge's own prose. + + `SkillStepRating` is part of the structured-output schema, and a rejected value is not a + caught error: the judge is sent back to produce the same answer again, so a plausible output + (empty `evidence` for a step it found no evidence of) becomes an unbounded retry loop. + """ + values = {"step": "step", "status": "covered", "evidence": "evidence"} + values[field] = "" + + assert getattr(SkillStepRating(**values), field) == "" + + +def test_prompt_includes_trajectory_evidence(): + case = _case([("pdf-processing", SKILL_BODY)]) + case.actual_trajectory.append({"role": "assistant", "content": [{"text": "Extracted report.pdf"}]}) + + prompt = SkillInstructionFollowingEvaluator()._build_prompt( + extract_selected_skills(case.actual_trajectory)[0], + case, + ) + + assert "Agent trajectory" in prompt + assert "Extracted report.pdf" in prompt + assert "Agent's final response" in prompt + + +@patch(_MODULE) +def test_evaluate_one_per_invoked_skill(mock_agent_class): + mock_agent = Mock() + mock_result = Mock() + mock_result.structured_output = _rating("covered", reasoning="all steps covered") + mock_agent.return_value = mock_result + mock_agent_class.return_value = mock_agent + + result = SkillInstructionFollowingEvaluator().evaluate( + _case([("pdf-processing", SKILL_BODY), ("spreadsheet", "# S\n1. inspect")]) + ) + # One EvaluationOutput per invoked skill; both fully followed here. + assert len(result) == 2 + assert all(r.label == "Fully Followed" for r in result) + assert all(r.score == 1.0 and r.test_pass for r in result) + + +@pytest.mark.parametrize( + "ordinal,score,expected_pass", + [ + (SkillFollowingScore.FULLY_FOLLOWED, 1.0, True), + (SkillFollowingScore.MOSTLY_FOLLOWED, 0.75, True), + (SkillFollowingScore.PARTIALLY_FOLLOWED, 0.5, False), + (SkillFollowingScore.MINIMALLY_FOLLOWED, 0.25, False), + (SkillFollowingScore.NOT_FOLLOWED, 0.0, False), + ], +) +@patch(_MODULE) +def test_ordinal_score_and_threshold(mock_agent_class, ordinal, score, expected_pass): + mock_agent = Mock() + mock_result = Mock() + # A single per-step status is enough; the shipped score comes from the ordinal rating. + mock_result.structured_output = _rating("covered", score=ordinal) + mock_agent.return_value = mock_result + mock_agent_class.return_value = mock_agent + + result = SkillInstructionFollowingEvaluator().evaluate(_case([("pdf-processing", SKILL_BODY)])) + # Score is the five-point ordinal mapped to [0, 1]; test_pass requires Mostly Followed. + assert result[0].score == score + assert result[0].test_pass is expected_pass + assert result[0].label == ordinal.value + # Per-step statuses/evidence and the derived coverage are preserved in the reason string, + # not as an EvaluationOutput subclass field, so the base output schema stays unchanged. + assert "Steps:" in result[0].reason + assert "Coverage:" in result[0].reason + assert "evidence 1" in result[0].reason + + +def test_label_carries_ordinal_not_skill_name(): + """The shipped label is the ordinal rating value (like the other judges), not the skill name.""" + with patch(_MODULE) as mock_agent_class: + mock_agent = Mock() + mock_result = Mock() + mock_result.structured_output = _rating("covered", score=SkillFollowingScore.MOSTLY_FOLLOWED) + mock_agent.return_value = mock_result + mock_agent_class.return_value = mock_agent + result = SkillInstructionFollowingEvaluator().evaluate(_case([("pdf-processing", SKILL_BODY)])) + assert result[0].label == "Mostly Followed" + + +def test_no_skill_returns_not_applicable_row(): + ev = SkillInstructionFollowingEvaluator() + case = EvaluationData(input="x", actual_output="y", actual_trajectory=[]) + result = ev.evaluate(case) + assert len(result) == 1 + assert result[0].label == "not_applicable" + assert result[0].test_pass is True # no violation + + +@patch(_MODULE) +def test_skill_prescribing_nothing_is_not_applicable(mock_agent_class): + """A skill body with no prescribed steps has nothing to follow, so it is not scored. + + Scoring it either way would be arbitrary: 0.0 reads as a failure to adhere, 1.0 as vacuous + adherence, and both distort the mean. + """ + mock_agent = Mock() + mock_result = Mock() + mock_result.structured_output = SkillFollowingRating( + reasoning="the body is reference material, not instructions", + steps=[], + score=SkillFollowingScore.NOT_FOLLOWED, + ) + mock_agent.return_value = mock_result + mock_agent_class.return_value = mock_agent + + result = SkillInstructionFollowingEvaluator().evaluate(_case([("pdf-processing", "# Reference\nField notes.")])) + + assert len(result) == 1 + assert result[0].label == "not_applicable" + assert result[0].reason == "pdf-processing: no prescribed steps found in the skill body" + assert result[0].test_pass is True + + +def test_missing_trajectory_does_not_pass(): + """Absent data is not a run that had nothing to follow, so it must not report a pass.""" + ev = SkillInstructionFollowingEvaluator() + case = EvaluationData(input="x", actual_output="y", actual_trajectory=None) + + result = ev.evaluate(case) + + assert len(result) == 1 + assert result[0].label == "not_applicable" + assert result[0].test_pass is False + assert result[0].reason == "no trajectory provided" + # The aggregator must carry the failure rather than defaulting an all-N/A row to pass. + assert ev.aggregator(result)[1] is False + + +@pytest.mark.asyncio +async def test_missing_trajectory_does_not_pass_async(): + ev = SkillInstructionFollowingEvaluator() + case = EvaluationData(input="x", actual_output="y", actual_trajectory=None) + + result = await ev.evaluate_async(case) + + assert len(result) == 1 + assert result[0].test_pass is False + assert result[0].reason == "no trajectory provided" + + +@patch(_MODULE) +def test_invoked_skill_without_body_is_not_mislabeled_as_no_invocation(mock_agent_class): + messages = [ + { + "role": "assistant", + "content": [ + {"toolUse": {"toolUseId": "t", "name": "load_skill", "input": {"skill_name": "pdf-processing"}}} + ], + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "t", + "content": [{"text": '{"status":"loaded","path":".agents/pdf-processing"}'}], + } + } + ], + }, + ] + case = EvaluationData(input="x", actual_output="y", actual_trajectory=messages) + + result = SkillInstructionFollowingEvaluator().evaluate(case) + + assert len(result) == 1 + assert result[0].label == "not_applicable" + assert result[0].reason == "pdf-processing: skill body unavailable" + mock_agent_class.assert_not_called() + + +@patch(_MODULE) +def test_refused_load_reports_why_nothing_could_be_followed(mock_agent_class): + """The harness refused the load, so the agent never received any instructions. + + Reported separately from a missing body: both are not-applicable, but conflating them hides + a broken harness behind what reads as a trajectory-capture gap. + """ + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "t", "name": "skills", "input": {"skill_name": "pdf-processing"}}}], + }, + { + "role": "user", + "content": [ + {"toolResult": {"toolUseId": "t", "status": "error", "content": [{"text": "skill not found"}]}} + ], + }, + ] + case = EvaluationData(input="do pdf", actual_output="done", actual_trajectory=messages) + + result = SkillInstructionFollowingEvaluator().evaluate(case) + + assert len(result) == 1 + assert result[0].label == "not_applicable" + assert result[0].reason == ( + "pdf-processing: the harness refused the load (skill not found), so no instructions were received" + ) + mock_agent_class.assert_not_called() + + +@patch(_MODULE) +def test_duplicate_loads_trigger_one_judge_call(mock_agent_class): + mock_agent = Mock() + mock_result = Mock() + mock_result.structured_output = _rating("covered", reasoning="covered") + mock_agent.return_value = mock_result + mock_agent_class.return_value = mock_agent + case = _case([("pdf-processing", SKILL_BODY), ("pdf-processing", SKILL_BODY)]) + + result = SkillInstructionFollowingEvaluator().evaluate(case) + + assert len(result) == 1 + mock_agent.assert_called_once() + + +@patch(_MODULE) +def test_each_row_names_the_skill_it_scored(mock_agent_class): + """With several skills invoked, `label` holds the rating, so the reason must attribute the row.""" + mock_agent = Mock() + mock_result = Mock() + mock_result.structured_output = _rating("covered", reasoning="covered") + mock_agent.return_value = mock_result + mock_agent_class.return_value = mock_agent + case = _case([("pdf-processing", SKILL_BODY), ("redaction", SKILL_BODY)]) + + result = SkillInstructionFollowingEvaluator().evaluate(case) + + assert len(result) == 2 + assert result[0].reason.startswith("pdf-processing: ") + assert result[1].reason.startswith("redaction: ") + + +def test_aggregator_drops_not_applicable(): + ev = SkillInstructionFollowingEvaluator() + rows = [ + EvaluationOutput(score=1.0, test_pass=True, reason="covered", label="pdf-processing"), + EvaluationOutput(score=0.0, test_pass=True, reason="no skill invoked", label=NOT_APPLICABLE), + ] + avg, all_pass, _ = ev.aggregator(rows) + # the N/A row must not deflate the mean + assert avg == 1.0 + assert all_pass is True + + # all-N/A aggregates to a clean pass, not 0-deflated failure + only_na = [EvaluationOutput(score=0.0, test_pass=True, reason="no skill invoked", label=NOT_APPLICABLE)] + assert ev.aggregator(only_na) == (0.0, True, "no skill invoked") + + missing_body = [ + EvaluationOutput( + score=0.0, + test_pass=True, + reason="pdf-processing: skill body unavailable", + label=NOT_APPLICABLE, + ) + ] + assert ev.aggregator(missing_body) == (0.0, True, "pdf-processing: skill body unavailable") + + +@pytest.mark.asyncio +@patch(_MODULE) +async def test_evaluate_async(mock_agent_class): + mock_agent = Mock() + + async def mock_invoke_async(*args, **kwargs): + mock_result = Mock() + # coverage 0.95 -> the judge rates it MOSTLY_FOLLOWED (score 0.75) + mock_result.structured_output = _rating( + "covered", + "covered", + "covered", + "covered", + "covered", + "covered", + "covered", + "covered", + "covered", + "partial", + score=SkillFollowingScore.MOSTLY_FOLLOWED, + ) + return mock_result + + mock_agent.invoke_async = mock_invoke_async + mock_agent_class.return_value = mock_agent + + result = await SkillInstructionFollowingEvaluator().evaluate_async(_case([("pdf-processing", SKILL_BODY)])) + assert len(result) == 1 + # Shipped score is the ordinal mapping (0.75), not the raw coverage (0.95). + assert result[0].score == 0.75 + assert result[0].test_pass is True diff --git a/tests/strands_evals/evaluators/test_skill_selection_accuracy_evaluator.py b/tests/strands_evals/evaluators/test_skill_selection_accuracy_evaluator.py new file mode 100644 index 00000000..b23d13c5 --- /dev/null +++ b/tests/strands_evals/evaluators/test_skill_selection_accuracy_evaluator.py @@ -0,0 +1,352 @@ +from unittest.mock import Mock, patch + +import pytest + +from strands_evals.evaluators.skill_selection_accuracy_evaluator import ( + SkillSelectionAccuracyEvaluator, + SkillSelectionRating, + SkillSelectionScore, +) +from strands_evals.extractors import InvokedSkill +from strands_evals.types.evaluation import NOT_APPLICABLE, EvaluationData, EvaluationOutput + +_MODULE = "strands_evals.evaluators.skill_selection_accuracy_evaluator.Agent" + +AVAILABLE_BLOCK = """ +pdf-processingExtract text from PDFs. +spreadsheetAnalyze spreadsheets. +""" + + +def _case(invoked: list[str] | str | None): + names = [invoked] if isinstance(invoked, str) else (invoked or []) + messages = [{"role": "system", "content": AVAILABLE_BLOCK}] + for i, name in enumerate(names): + tid = f"t{i}" + messages.append( + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": tid, "name": "skills", "input": {"skill_name": name}}}], + } + ) + messages.append( + {"role": "user", "content": [{"toolResult": {"toolUseId": tid, "content": [{"text": "# Skill body"}]}}]} + ) + return EvaluationData(input="Extract text from report.pdf", actual_output="done", actual_trajectory=messages) + + +def test_init_defaults(): + ev = SkillSelectionAccuracyEvaluator() + assert ev.version == "v0" + assert ev.model is None + assert ev.system_prompt is not None + # unlike the tool evaluator, it does NOT slice via TraceExtractor + assert ev.evaluation_level is None + + +def test_aggregator_drops_not_applicable_rows(): + """A case with one judged skill and one unjudgeable row is not half right. + + The per-case aggregate has to drop the same rows `calculate_overall_score` drops, or the + case score reported in the table disagrees with the overall score computed from it. + """ + ev = SkillSelectionAccuracyEvaluator() + assert ev.aggregator == ev._aggregate_dropping_na + + rows = [ + EvaluationOutput(score=1.0, test_pass=True, reason="pdf-processing: fits", label="Yes"), + EvaluationOutput(score=0.0, test_pass=True, reason="no skills were available", label=NOT_APPLICABLE), + ] + avg, all_pass, _ = ev.aggregator(rows) + assert avg == 1.0 + assert all_pass is True + + +def test_prompt_focuses_on_one_invoked_skill(): + ev = SkillSelectionAccuracyEvaluator() + prompt = ev._build_prompt(_case("pdf-processing"), focus_skill=InvokedSkill("pdf-processing", "# body")) + assert "pdf-processing: Extract text from PDFs." in prompt # available list + assert "invoked the skill: pdf-processing" in prompt # focal decision + assert "Agent trajectory" in prompt + assert "report.pdf" in prompt + + +def test_prompt_says_an_unrecorded_catalog_was_not_recorded(): + """An empty list is a claim about the run; a missing one is a gap in the telemetry. + + Claude Code and the Claude Agent SDK never emit the offered set, so rendering it as an empty + collection lets the judge reason that the invoked skill did not exist and mark a correct pick + wrong. Naming the reason removes that inference. + """ + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "t0", "name": "skills", "input": {"skill_name": "pdf-processing"}}}], + }, + {"role": "user", "content": [{"toolResult": {"toolUseId": "t0", "content": [{"text": "# Skill body"}]}}]}, + ] + case = EvaluationData(input="Extract text from report.pdf", actual_output="done", actual_trajectory=messages) + + prompt = SkillSelectionAccuracyEvaluator()._build_prompt(case, focus_skill=InvokedSkill("pdf-processing", "# body")) + + assert "(not recorded by this harness)" in prompt + assert "[]" not in prompt + + +def test_prompt_says_none_when_the_harness_advertised_an_empty_catalog(): + """The other empty case, which must not read as missing telemetry. + + A harness that mounted no skills did record its offered set, and "none" is the honest + rendering. The Strands plugin emits the block with this exact wording. + """ + messages = [ + {"role": "system", "content": "\nNo skills are currently available.\n"}, + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "t0", "name": "skills", "input": {"skill_name": "pdf-processing"}}}], + }, + {"role": "user", "content": [{"toolResult": {"toolUseId": "t0", "content": [{"text": "# Skill body"}]}}]}, + ] + case = EvaluationData(input="Extract text from report.pdf", actual_output="done", actual_trajectory=messages) + + prompt = SkillSelectionAccuracyEvaluator()._build_prompt(case, focus_skill=InvokedSkill("pdf-processing", "# body")) + + assert "(none: this harness advertised no skills)" in prompt + assert "not recorded" not in prompt + + +def test_prompt_tells_the_judge_a_refused_load_is_not_a_wrong_choice(): + """Selection is about the choice, not the outcome. + + Left unsaid, the judge sees the harness error in the trajectory and scores a correct + selection as wrong for failing to load. + """ + ev = SkillSelectionAccuracyEvaluator() + + prompt = ev._build_prompt( + _case("pdf-processing"), + focus_skill=InvokedSkill("pdf-processing", None, status="failed"), + ) + + assert "invoked the skill: pdf-processing" in prompt + assert "harness refused the load" in prompt + assert "not whether it worked" in prompt + + +def test_prompt_carries_the_harness_refusal_message(): + """Which refusal it was still bears on the choice. + + A skill name the harness did not recognize is a worse pick than a right one the harness + could not mount, and only the refusal text says which happened. + """ + ev = SkillSelectionAccuracyEvaluator() + + prompt = ev._build_prompt( + _case("pdf-processing"), + focus_skill=InvokedSkill( + "pdf-procesing", None, status="failed", error="Skill 'pdf-procesing' not found. Available: pdf-processing" + ), + ) + + assert "The harness said: Skill 'pdf-procesing' not found. Available: pdf-processing" in prompt + + +def test_prompt_has_no_abstention_branch(): + """Selection judges invoked skills only, so no prompt path offers an abstention verdict.""" + ev = SkillSelectionAccuracyEvaluator() + prompt = ev._build_prompt(_case("pdf-processing"), focus_skill=InvokedSkill("pdf-processing", "# body")) + assert "abstained" not in prompt + assert "abstention" not in ev.system_prompt.casefold() + + +@pytest.mark.parametrize( + "score,expected_value,expected_pass", + [ + (SkillSelectionScore.YES, 1.0, True), + (SkillSelectionScore.NO, 0.0, False), + ], +) +@patch(_MODULE) +def test_evaluate_score_mapping_labels_by_rating(mock_agent_class, score, expected_value, expected_pass): + mock_agent = Mock() + mock_result = Mock() + mock_result.structured_output = SkillSelectionRating(reasoning="because", score=score) + mock_agent.return_value = mock_result + mock_agent_class.return_value = mock_agent + + result = SkillSelectionAccuracyEvaluator().evaluate(_case("pdf-processing")) + + # One output for the single invoked skill. `label` is the judge's rating, as in every other + # judge in the framework; which decision the row is about is named in `reason`. + assert len(result) == 1 + assert result[0].score == expected_value + assert result[0].test_pass is expected_pass + assert result[0].label == score.value + assert result[0].reason == "pdf-processing: because" + + +@patch(_MODULE) +def test_evaluate_loops_per_invoked_skill(mock_agent_class): + mock_agent = Mock() + mock_result = Mock() + mock_result.structured_output = SkillSelectionRating(reasoning="fits", score=SkillSelectionScore.YES) + mock_agent.return_value = mock_result + mock_agent_class.return_value = mock_agent + + result = SkillSelectionAccuracyEvaluator().evaluate(_case(["pdf-processing", "spreadsheet"])) + + # one EvaluationOutput per invoked skill, each naming its skill in `reason` + assert len(result) == 2 + assert {r.reason.split(":")[0] for r in result} == {"pdf-processing", "spreadsheet"} + assert mock_agent.call_count == 2 + # A fresh judge per skill: a reused Agent would carry the first verdict into the second + # prompt as conversation history and resend the trajectory on top of it. + assert mock_agent_class.call_count == 2 + + +@patch(_MODULE) +def test_missing_trajectory_is_not_scored(mock_agent_class): + """A None trajectory is absent data, so it fails rather than passing as not-applicable.""" + case = EvaluationData(input="do pdf", actual_output="done", actual_trajectory=None) + + result = SkillSelectionAccuracyEvaluator().evaluate(case) + + assert len(result) == 1 + assert result[0].label == "not_applicable" + assert result[0].score == 0.0 + assert result[0].test_pass is False + mock_agent_class.assert_not_called() # no judge call on missing data + + +@patch(_MODULE) +def test_no_invocation_without_a_catalog_is_not_scored(mock_agent_class): + """With nothing on offer and nothing invoked there was no selection decision to judge. + + A "Yes" would credit the agent for declining an offer it never received, and a "No" would + penalize it for the same. Not every trajectory carries a catalog, so this is common. + """ + case = EvaluationData( + input="Extract text from report.pdf", + actual_output="done", + actual_trajectory=[{"role": "user", "content": [{"text": "Extract text from report.pdf"}]}], + ) + + result = SkillSelectionAccuracyEvaluator().evaluate(case) + + assert len(result) == 1 + assert result[0].label == "not_applicable" + assert result[0].reason == "no skills were available to select from" + assert result[0].test_pass is True + mock_agent_class.assert_not_called() # nothing to judge, so no judge call + + +@pytest.mark.asyncio +@patch(_MODULE) +async def test_no_invocation_without_a_catalog_is_not_scored_async(mock_agent_class): + case = EvaluationData(input="do pdf", actual_output="done", actual_trajectory=[]) + + result = await SkillSelectionAccuracyEvaluator().evaluate_async(case) + + assert len(result) == 1 + assert result[0].label == "not_applicable" + mock_agent_class.assert_not_called() + + +@patch(_MODULE) +def test_invocation_with_no_catalog_is_still_judged(mock_agent_class): + """A skill that was actually invoked is a real decision, catalog or not.""" + mock_agent = Mock() + mock_result = Mock() + mock_result.structured_output = SkillSelectionRating(reasoning="fits", score=SkillSelectionScore.YES) + mock_agent.return_value = mock_result + mock_agent_class.return_value = mock_agent + case = _case("pdf-processing") + case.actual_trajectory = case.actual_trajectory[1:] # drop the system message + + result = SkillSelectionAccuracyEvaluator().evaluate(case) + + assert [r.reason for r in result] == ["pdf-processing: fits"] + assert mock_agent.call_count == 1 + + +@patch(_MODULE) +def test_refused_load_is_judged_as_a_selection_not_an_abstention(mock_agent_class): + """A skill the harness refused is still a selection the agent made. + + Dropping the row would leave the run looking like an abstention, so an agent that picked the + right skill and was refused would be scored on a decision it never took. + """ + mock_agent = Mock() + mock_result = Mock() + mock_result.structured_output = SkillSelectionRating(reasoning="right skill", score=SkillSelectionScore.YES) + mock_agent.return_value = mock_result + mock_agent_class.return_value = mock_agent + case = _case("pdf-processing") + case.actual_trajectory[-1]["content"][0]["toolResult"]["status"] = "error" + + result = SkillSelectionAccuracyEvaluator().evaluate(case) + + assert [r.reason for r in result] == ["pdf-processing: right skill"] + assert result[0].label == "Yes" + assert "harness refused the load" in mock_agent.call_args.args[0] + + +@patch(_MODULE) +def test_no_invocation_with_a_catalog_is_not_scored(mock_agent_class): + """Skills were on offer and none was taken: still not this evaluator's decision to judge. + + Whether declining was correct depends on the whole offered set, so it is a session-level + question. The reason distinguishes this from the nothing-on-offer case. + """ + result = SkillSelectionAccuracyEvaluator().evaluate(_case(None)) + + assert len(result) == 1 + assert result[0].label == "not_applicable" + assert result[0].score == 0.0 + assert result[0].test_pass is True + assert result[0].reason == "no skill invoked; whether declining was correct is not judged here" + mock_agent_class.assert_not_called() + + +@pytest.mark.asyncio +@patch(_MODULE) +async def test_evaluate_async_loops_per_skill(mock_agent_class): + mock_agent = Mock() + + async def mock_invoke_async(*args, **kwargs): + mock_result = Mock() + mock_result.structured_output = SkillSelectionRating(reasoning="ok", score=SkillSelectionScore.YES) + return mock_result + + mock_agent.invoke_async = mock_invoke_async + mock_agent_class.return_value = mock_agent + + result = await SkillSelectionAccuracyEvaluator().evaluate_async(_case(["pdf-processing", "spreadsheet"])) + assert len(result) == 2 + assert {r.reason.split(":")[0] for r in result} == {"pdf-processing", "spreadsheet"} + assert all(r.score == 1.0 for r in result) + assert mock_agent_class.call_count == 2 # a fresh judge per skill, same as the sync path + + +@patch(_MODULE) +def test_each_judge_sees_exactly_one_prompt(mock_agent_class): + """Contract: no judge is asked twice, so no verdict leaks into the next skill's prompt.""" + agents = [] + + def new_agent(*_args, **_kwargs): + agent = Mock() + result = Mock() + result.structured_output = SkillSelectionRating(reasoning="fits", score=SkillSelectionScore.YES) + agent.return_value = result + agents.append(agent) + return agent + + mock_agent_class.side_effect = new_agent + + SkillSelectionAccuracyEvaluator().evaluate(_case(["pdf-processing", "spreadsheet"])) + + assert len(agents) == 2 + assert [a.call_count for a in agents] == [1, 1] + prompts = [a.call_args.args[0] for a in agents] + assert "invoked the skill: pdf-processing" in prompts[0] + assert "invoked the skill: spreadsheet" in prompts[1] diff --git a/tests/strands_evals/evaluators/test_trajectory_prompt_template.py b/tests/strands_evals/evaluators/test_trajectory_prompt_template.py new file mode 100644 index 00000000..f193c987 --- /dev/null +++ b/tests/strands_evals/evaluators/test_trajectory_prompt_template.py @@ -0,0 +1,23 @@ +"""Unit tests for trajectory serialization in judge prompts.""" + +from strands_evals.evaluators.prompt_templates.trajectory_prompt_template import serialize_trajectory + + +def test_serialize_trajectory_truncates_oversized_runs(): + """Real runs can exceed any judge context window, so the middle is dropped.""" + huge = [{"role": "user", "content": [{"text": "x" * 900_000}]}] + + serialized = serialize_trajectory(huge) + + assert len(serialized) < 900_000 + assert "characters omitted" in serialized + + +def test_serialize_trajectory_leaves_normal_runs_intact(): + small = [{"role": "user", "content": [{"text": "do pdf"}]}] + + assert "omitted" not in serialize_trajectory(small) + + +def test_serialize_trajectory_reports_a_missing_trajectory(): + assert serialize_trajectory(None) == "(no trajectory)" diff --git a/tests/strands_evals/extractors/__init__.py b/tests/strands_evals/extractors/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/strands_evals/extractors/fixtures/capture_skill_fixtures.py b/tests/strands_evals/extractors/fixtures/capture_skill_fixtures.py new file mode 100644 index 00000000..c260775b --- /dev/null +++ b/tests/strands_evals/extractors/fixtures/capture_skill_fixtures.py @@ -0,0 +1,278 @@ +"""Regenerate the captured skill fixtures from the harnesses themselves. + +Every JSON file next to this script came out of a real harness rather than out of a guess at its +wire format, and this is the script that got it. Run it to refresh a fixture after an SDK upgrade, +or to check that the recorded shape is still the shape the harness emits: + + python tests/strands_evals/extractors/fixtures/capture_skill_fixtures.py --list + python tests/strands_evals/extractors/fixtures/capture_skill_fixtures.py strands adk + +It is not part of the test run. The fixtures are checked in, and the tests read those, so the suite +needs neither these SDKs nor network access. What the script buys is a way to prove the fixture is +still faithful and to regenerate it when a harness changes. + +Provenance, per fixture: + +- `strands_agent_skills.json` -- captured here. Runs a real `strands.Agent` with the real + `AgentSkills` plugin (strands-agents 1.44.0) against a scripted model, so the tool spec, the + system-prompt injection, the result envelope and the refusal strings are all the SDK's own. The + model is scripted only to decide which calls happen, which is what makes the run deterministic + and offline. +- `google_adk_load_skill.json` -- captured here. Drives the real `LoadSkillTool` from + `google.adk.tools.skill_toolset` (google-adk 2.4.0) for a load, a misspelled name and a missing + argument. +- `claude_code_skill_tool.json` -- transcribed from a real Claude Code run + (`claude -p "Use the pdf-processing skill on report.pdf" --model haiku`, CLI transcript at + `~/.claude/projects//.jsonl`), trimmed to the skill-load window. Not captured here + because it needs the CLI and a model call. +- `codex_exec_json.json` -- transcribed from a real `codex exec --json` run (codex-cli 0.144.4), + trimmed to the skill-load window. +- `codex_session_rollout.json` -- the same run's session rollout + (`~/.codex/sessions//rollout-*.jsonl`), which records Responses API items rather than the + `item.completed` events the `--json` stream emits. Both shapes are real and the extractor reads + both, so both are fixtures. + +Gemini CLI, OpenAI Agents and OpenHands have no captured fixture. Their shapes in +`skill_fixtures.py` are hand-written from the documented format, and are marked as such there. +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +from pathlib import Path +from typing import Any + +FIXTURES = Path(__file__).resolve().parent +SKILL_MD = """--- +name: pdf-processing +description: Use this skill when the task requires reading or extracting text from PDF files. +--- + +# PDF Processing Skill + +1. Identify the PDF file path. +2. Extract the text. +3. Summarize it. +""" +SPREADSHEET_MD = """--- +name: spreadsheet-analysis +description: Analyze, edit, or generate spreadsheets. +--- + +# Spreadsheet Analysis +1. Inspect. +""" + + +# ---- Strands ---------------------------------------------------------------- + + +class _ScriptedModel: + """A `Model` that replays a fixed list of turns, so the capture is deterministic and offline.""" + + def __init__(self, turns: list[list[dict[str, Any]]]) -> None: + self._turns = list(turns) + self.system_prompts: list[Any] = [] + + def update_config(self, **model_config: Any) -> None: + pass + + def get_config(self) -> Any: + return {} + + def structured_output(self, output_model: Any, prompt: Any, system_prompt: Any = None, **kwargs: Any) -> Any: + raise NotImplementedError + + @property + def stateful(self) -> bool: + return False + + @property + def context_window_limit(self) -> int | None: + return None + + async def stream(self, messages: Any, tool_specs: Any = None, system_prompt: Any = None, **kwargs: Any) -> Any: + self.system_prompts.append(kwargs.get("system_prompt_content") or system_prompt) + blocks = self._turns.pop(0) if self._turns else [{"text": "Done."}] + yield {"messageStart": {"role": "assistant"}} + for index, block in enumerate(blocks): + if "text" in block: + yield {"contentBlockStart": {"start": {}, "contentBlockIndex": index}} + yield {"contentBlockDelta": {"delta": {"text": block["text"]}, "contentBlockIndex": index}} + else: + use = block["toolUse"] + yield { + "contentBlockStart": { + "start": {"toolUse": {"toolUseId": use["toolUseId"], "name": use["name"]}}, + "contentBlockIndex": index, + } + } + yield { + "contentBlockDelta": { + "delta": {"toolUse": {"input": json.dumps(use["input"])}}, + "contentBlockIndex": index, + } + } + yield {"contentBlockStop": {"contentBlockIndex": index}} + yield {"messageStop": {"stopReason": "tool_use" if any("toolUse" in b for b in blocks) else "end_turn"}} + yield {"metadata": {"usage": {"inputTokens": 0, "outputTokens": 0, "totalTokens": 0}, "metrics": {}}} + + +def _skill_call(tool_use_id: str, skill_name: str) -> dict[str, Any]: + return {"toolUse": {"toolUseId": tool_use_id, "name": "skills", "input": {"skill_name": skill_name}}} + + +async def capture_strands(skills_dir: Path) -> dict[str, Any]: + """Run the real AgentSkills plugin through the loads, refusals and repeats worth recording.""" + from strands import Agent + from strands.vended_plugins.skills import AgentSkills + + async def run(turns: list[list[dict[str, Any]]], prompt: str) -> dict[str, Any]: + model = _ScriptedModel(turns) + agent = Agent( + model=model, # type: ignore[arg-type] + plugins=[AgentSkills(skills=str(skills_dir))], + system_prompt="You are a helpful agent.", + ) + await agent.invoke_async(prompt) + prompts = [prompt_content for prompt_content in model.system_prompts if prompt_content] + return { + "system_prompt": prompts[0] if prompts else None, + # Only the wire fields: the SDK also attaches per-message usage and metrics, which say + # nothing about the shape a skill load takes. + "messages": [ + {key: value for key, value in m.items() if key in ("role", "content")} for m in agent.messages + ], + } + + pdf = "Extract text from report.pdf" + return { + "loaded": await run( + [ + [{"text": "I'll use the pdf-processing skill."}, _skill_call("tu-1", "pdf-processing")], + [{"text": "Extracted and summarized."}], + ], + pdf, + ), + # A one-letter typo. The plugin returns an ordinary string, which `@tool` marks + # status="success", so the refusal is only visible in the text. + "typo_refusal": await run( + [ + [{"text": "Loading the skill."}, _skill_call("tu-1", "pdf-procesing")], + [{"text": "Sorry, I could not load it."}], + ], + pdf, + ), + "empty_name_refusal": await run([[_skill_call("tu-1", "")], [{"text": "Sorry."}]], pdf), + "retry_after_refusal": await run( + [ + [_skill_call("tu-1", "pdf-procesing")], + [{"text": "Retrying with the right name."}, _skill_call("tu-2", "pdf-processing")], + [{"text": "Done."}], + ], + pdf, + ), + "two_skills": await run( + [ + [_skill_call("tu-1", "pdf-processing")], + [_skill_call("tu-2", "spreadsheet-analysis")], + [{"text": "Done."}], + ], + "Extract the PDF then build a sheet", + ), + "repeated_load": await run( + [[_skill_call("tu-1", "pdf-processing")], [_skill_call("tu-2", "pdf-processing")], [{"text": "Done."}]], + pdf, + ), + } + + +# ---- Google ADK ------------------------------------------------------------- + + +async def capture_adk() -> dict[str, Any]: + """Drive the real `LoadSkillTool` and wrap each payload in the Content shape ADK records.""" + from unittest.mock import MagicMock + + from google.adk.skills.models import Frontmatter, Skill + from google.adk.tools.skill_toolset import SkillToolset + + skill = Skill( + frontmatter=Frontmatter( + name="pdf-processing", + description="Use this skill when the task requires reading or extracting text from PDF files.", + ), + instructions=SKILL_MD.split("---", 2)[2].strip(), + ) + tools = {tool.name: tool for tool in await SkillToolset(skills=[skill]).get_tools()} + load = tools["load_skill"] + + def context() -> Any: + ctx = MagicMock() + ctx.state = {} + ctx.agent_name = "pdf-agent" + ctx.invocation_id = "inv-1" + return ctx + + async def call(**args: Any) -> dict[str, Any]: + response = await load.run_async(args=args, tool_context=context()) + name = args.get("skill_name", "") + return { + "contents": [ + { + "role": "model", + "parts": [{"function_call": {"id": "adk-1", "args": {"skill_name": name}, "name": "load_skill"}}], + }, + { + "role": "user", + "parts": [{"function_response": {"id": "adk-1", "name": "load_skill", "response": response}}], + }, + ] + } + + return { + "loaded": await call(skill_name="pdf-processing"), + "not_found": await call(skill_name="pdf-procesing"), + "missing_arg": await call(), + } + + +# ---- Entry point ------------------------------------------------------------ + +CAPTURES = { + "strands": ("strands_agent_skills.json", "strands-agents, real AgentSkills plugin"), + "adk": ("google_adk_load_skill.json", "google-adk, real LoadSkillTool"), +} + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument( + "which", nargs="*", default=[], choices=[*sorted(CAPTURES), []], help="which fixtures to regenerate" + ) + parser.add_argument("--list", action="store_true", help="list what this script can capture and exit") + args = parser.parse_args() + + if args.list or not args.which: + for key, (filename, description) in sorted(CAPTURES.items()): + print(f"{key:10} {filename:32} {description}") # noqa: T201 + return + + for key in args.which: + filename, _ = CAPTURES[key] + if key == "strands": + skills_dir = Path("/tmp/agent_skills_capture/skills") + for name, body in (("pdf-processing", SKILL_MD), ("spreadsheet-analysis", SPREADSHEET_MD)): + (skills_dir / name).mkdir(parents=True, exist_ok=True) + (skills_dir / name / "SKILL.md").write_text(body) + captured = asyncio.run(capture_strands(skills_dir)) + else: + captured = asyncio.run(capture_adk()) + (FIXTURES / filename).write_text(json.dumps(captured, indent=2) + "\n") + print(f"wrote {filename}") # noqa: T201 + + +if __name__ == "__main__": + main() diff --git a/tests/strands_evals/extractors/fixtures/claude_code_skill_tool.json b/tests/strands_evals/extractors/fixtures/claude_code_skill_tool.json new file mode 100644 index 00000000..12e759a3 --- /dev/null +++ b/tests/strands_evals/extractors/fixtures/claude_code_skill_tool.json @@ -0,0 +1,53 @@ +{ + "messages": [ + { + "type": "user", + "message": { + "role": "user", + "content": "Use the pdf-processing skill on report.pdf" + } + }, + { + "type": "assistant", + "message": { + "role": "assistant", + "content": [ + { + "type": "tool_use", + "id": "toolu_bdrk_017tApeVjBHWpYV5SVrPQcoD", + "name": "Skill", + "input": { + "skill": "pdf-processing", + "args": "report.pdf" + } + } + ] + } + }, + { + "type": "user", + "message": { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": "toolu_bdrk_017tApeVjBHWpYV5SVrPQcoD", + "content": "Launching skill: pdf-processing" + } + ] + } + }, + { + "type": "user", + "message": { + "role": "user", + "content": [ + { + "type": "text", + "text": "Base directory for this skill: /tmp/agent_skills_capture/claude/.claude/skills/pdf-processing\n\n# PDF Processing Skill\n\n1. Identify the PDF file path.\n2. Extract the text.\n3. Summarize it.\n\n\nARGUMENTS: report.pdf" + } + ] + } + } + ] +} \ No newline at end of file diff --git a/tests/strands_evals/extractors/fixtures/codex_exec_json.json b/tests/strands_evals/extractors/fixtures/codex_exec_json.json new file mode 100644 index 00000000..43dd3814 --- /dev/null +++ b/tests/strands_evals/extractors/fixtures/codex_exec_json.json @@ -0,0 +1,42 @@ +{ + "events": [ + { + "type": "item.completed", + "item": { + "id": "item_0", + "type": "agent_message", + "text": "I\u2019m using the `pdf-processing` skill to inspect `report.pdf` and determine the appropriate extraction workflow." + } + }, + { + "type": "item.started", + "item": { + "id": "item_1", + "type": "command_execution", + "command": "/bin/bash -lc 'cat /tmp/agent_skills_capture/codex/.agents/skills/pdf-processing/SKILL.md'", + "aggregated_output": "", + "exit_code": null, + "status": "in_progress" + } + }, + { + "type": "item.completed", + "item": { + "id": "item_1", + "type": "command_execution", + "command": "/bin/bash -lc 'cat /tmp/agent_skills_capture/codex/.agents/skills/pdf-processing/SKILL.md'", + "aggregated_output": "---\nname: pdf-processing\ndescription: Use this skill when the task requires reading or extracting text from PDF files.\n---\n\n# PDF Processing Skill\n\n1. Identify the PDF file path.\n2. Extract the text.\n3. Summarize it.\n", + "exit_code": 0, + "status": "completed" + } + }, + { + "type": "item.completed", + "item": { + "id": "item_2", + "type": "agent_message", + "text": "The skill specifies text extraction followed by summarization. I\u2019m locating the PDF and checking which local extractor is available." + } + } + ] +} \ No newline at end of file diff --git a/tests/strands_evals/extractors/fixtures/codex_session_rollout.json b/tests/strands_evals/extractors/fixtures/codex_session_rollout.json new file mode 100644 index 00000000..076d2af8 --- /dev/null +++ b/tests/strands_evals/extractors/fixtures/codex_session_rollout.json @@ -0,0 +1,32 @@ +{ + "items": [ + { + "type": "function_call", + "id": "fc_7001304f89bd5b439f1dd55a7f21f819", + "name": "exec_command", + "arguments": "{\"cmd\":\"cat /tmp/agent_skills_capture/codex/.agents/skills/pdf-processing/SKILL.md\",\"workdir\":\"/tmp/agent_skills_capture/codex\",\"yield_time_ms\":10000,\"max_output_tokens\":20000}", + "call_id": "call_dc797c9cf6045e2eb5bad0c1e3e1c16b", + "internal_chat_message_metadata_passthrough": { + "turn_id": "019fafbf-3fb1-7d80-93bb-bfb2563751bd" + } + }, + { + "type": "function_call_output", + "call_id": "call_dc797c9cf6045e2eb5bad0c1e3e1c16b", + "output": "Chunk ID: c9e488\nWall time: 0.0000 seconds\nProcess exited with code 0\nOriginal token count: 55\nOutput:\n---\nname: pdf-processing\ndescription: Use this skill when the task requires reading or extracting text from PDF files.\n---\n\n# PDF Processing Skill\n\n1. Identify the PDF file path.\n2. Extract the text.\n3. Summarize it.\n", + "internal_chat_message_metadata_passthrough": { + "turn_id": "019fafbf-3fb1-7d80-93bb-bfb2563751bd" + } + }, + { + "type": "function_call", + "id": "fc_3be2ee059d145bc2a02c1d0420e22eab", + "name": "exec_command", + "arguments": "{\"cmd\":\"rg --files -g 'report.pdf' -g '*.pdf'\",\"workdir\":\"/tmp/agent_skills_capture/codex\",\"yield_time_ms\":10000,\"max_output_tokens\":2000}", + "call_id": "call_5f170a1e76b55341b4f7cc62aee577df", + "internal_chat_message_metadata_passthrough": { + "turn_id": "019fafbf-3fb1-7d80-93bb-bfb2563751bd" + } + } + ] +} \ No newline at end of file diff --git a/tests/strands_evals/extractors/fixtures/google_adk_load_skill.json b/tests/strands_evals/extractors/fixtures/google_adk_load_skill.json new file mode 100644 index 00000000..e471b755 --- /dev/null +++ b/tests/strands_evals/extractors/fixtures/google_adk_load_skill.json @@ -0,0 +1,109 @@ +{ + "loaded": { + "contents": [ + { + "role": "model", + "parts": [ + { + "function_call": { + "id": "adk-1", + "args": { + "skill_name": "pdf-processing" + }, + "name": "load_skill" + } + } + ] + }, + { + "role": "user", + "parts": [ + { + "function_response": { + "id": "adk-1", + "name": "load_skill", + "response": { + "skill_name": "pdf-processing", + "instructions": "# PDF Processing Skill\n\n1. Identify the PDF file path.\n2. Extract the text.\n3. Summarize it.", + "frontmatter": { + "name": "pdf-processing", + "description": "Use this skill when the task requires reading or extracting text from PDF files.", + "license": null, + "compatibility": null, + "allowed_tools": null, + "metadata": {} + } + } + } + } + ] + } + ] + }, + "not_found": { + "contents": [ + { + "role": "model", + "parts": [ + { + "function_call": { + "id": "adk-1", + "args": { + "skill_name": "pdf-procesing" + }, + "name": "load_skill" + } + } + ] + }, + { + "role": "user", + "parts": [ + { + "function_response": { + "id": "adk-1", + "name": "load_skill", + "response": { + "error": "Skill 'pdf-procesing' not found.", + "error_code": "SKILL_NOT_FOUND" + } + } + } + ] + } + ] + }, + "missing_arg": { + "contents": [ + { + "role": "model", + "parts": [ + { + "function_call": { + "id": "adk-1", + "args": { + "skill_name": "" + }, + "name": "load_skill" + } + } + ] + }, + { + "role": "user", + "parts": [ + { + "function_response": { + "id": "adk-1", + "name": "load_skill", + "response": { + "error": "Argument 'skill_name' is required.", + "error_code": "INVALID_ARGUMENTS" + } + } + } + ] + } + ] + } +} diff --git a/tests/strands_evals/extractors/fixtures/strands_agent_skills.json b/tests/strands_evals/extractors/fixtures/strands_agent_skills.json new file mode 100644 index 00000000..63605610 --- /dev/null +++ b/tests/strands_evals/extractors/fixtures/strands_agent_skills.json @@ -0,0 +1,449 @@ +{ + "loaded": { + "system_prompt": [ + { + "text": "You are a helpful agent." + }, + { + "text": "\n\npdf-processing\nUse this skill when the task requires reading or extracting text from PDF files.\n/tmp/agent_skills_capture/skills/pdf-processing/SKILL.md\n\n\nspreadsheet-analysis\nAnalyze, edit, or generate spreadsheets.\n/tmp/agent_skills_capture/skills/spreadsheet-analysis/SKILL.md\n\n" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "text": "Extract text from report.pdf" + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "I'll use the pdf-processing skill." + }, + { + "toolUse": { + "toolUseId": "tu-1", + "name": "skills", + "input": { + "skill_name": "pdf-processing" + } + } + } + ] + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "tu-1", + "status": "success", + "content": [ + { + "text": "# PDF Processing Skill\n\n1. Identify the PDF file path.\n2. Extract the text.\n3. Summarize it.\n\n---\nLocation: /tmp/agent_skills_capture/skills/pdf-processing/SKILL.md" + } + ] + } + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "Extracted and summarized." + } + ] + } + ] + }, + "typo_refusal": { + "system_prompt": [ + { + "text": "You are a helpful agent." + }, + { + "text": "\n\npdf-processing\nUse this skill when the task requires reading or extracting text from PDF files.\n/tmp/agent_skills_capture/skills/pdf-processing/SKILL.md\n\n\nspreadsheet-analysis\nAnalyze, edit, or generate spreadsheets.\n/tmp/agent_skills_capture/skills/spreadsheet-analysis/SKILL.md\n\n" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "text": "Extract text from report.pdf" + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "Loading the skill." + }, + { + "toolUse": { + "toolUseId": "tu-1", + "name": "skills", + "input": { + "skill_name": "pdf-procesing" + } + } + } + ] + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "tu-1", + "status": "success", + "content": [ + { + "text": "Skill 'pdf-procesing' not found. Available skills: pdf-processing, spreadsheet-analysis" + } + ] + } + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "Sorry, I could not load it." + } + ] + } + ] + }, + "empty_name_refusal": { + "system_prompt": [ + { + "text": "You are a helpful agent." + }, + { + "text": "\n\npdf-processing\nUse this skill when the task requires reading or extracting text from PDF files.\n/tmp/agent_skills_capture/skills/pdf-processing/SKILL.md\n\n\nspreadsheet-analysis\nAnalyze, edit, or generate spreadsheets.\n/tmp/agent_skills_capture/skills/spreadsheet-analysis/SKILL.md\n\n" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "text": "Extract text from report.pdf" + } + ] + }, + { + "role": "assistant", + "content": [ + { + "toolUse": { + "toolUseId": "tu-1", + "name": "skills", + "input": { + "skill_name": "" + } + } + } + ] + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "tu-1", + "status": "success", + "content": [ + { + "text": "Error: skill_name is required. Available skills: pdf-processing, spreadsheet-analysis" + } + ] + } + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "Sorry." + } + ] + } + ] + }, + "retry_after_refusal": { + "system_prompt": [ + { + "text": "You are a helpful agent." + }, + { + "text": "\n\npdf-processing\nUse this skill when the task requires reading or extracting text from PDF files.\n/tmp/agent_skills_capture/skills/pdf-processing/SKILL.md\n\n\nspreadsheet-analysis\nAnalyze, edit, or generate spreadsheets.\n/tmp/agent_skills_capture/skills/spreadsheet-analysis/SKILL.md\n\n" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "text": "Extract text from report.pdf" + } + ] + }, + { + "role": "assistant", + "content": [ + { + "toolUse": { + "toolUseId": "tu-1", + "name": "skills", + "input": { + "skill_name": "pdf-procesing" + } + } + } + ] + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "tu-1", + "status": "success", + "content": [ + { + "text": "Skill 'pdf-procesing' not found. Available skills: pdf-processing, spreadsheet-analysis" + } + ] + } + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "Retrying with the right name." + }, + { + "toolUse": { + "toolUseId": "tu-2", + "name": "skills", + "input": { + "skill_name": "pdf-processing" + } + } + } + ] + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "tu-2", + "status": "success", + "content": [ + { + "text": "# PDF Processing Skill\n\n1. Identify the PDF file path.\n2. Extract the text.\n3. Summarize it.\n\n---\nLocation: /tmp/agent_skills_capture/skills/pdf-processing/SKILL.md" + } + ] + } + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "Done." + } + ] + } + ] + }, + "two_skills": { + "system_prompt": [ + { + "text": "You are a helpful agent." + }, + { + "text": "\n\npdf-processing\nUse this skill when the task requires reading or extracting text from PDF files.\n/tmp/agent_skills_capture/skills/pdf-processing/SKILL.md\n\n\nspreadsheet-analysis\nAnalyze, edit, or generate spreadsheets.\n/tmp/agent_skills_capture/skills/spreadsheet-analysis/SKILL.md\n\n" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "text": "Extract the PDF then build a sheet" + } + ] + }, + { + "role": "assistant", + "content": [ + { + "toolUse": { + "toolUseId": "tu-1", + "name": "skills", + "input": { + "skill_name": "pdf-processing" + } + } + } + ] + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "tu-1", + "status": "success", + "content": [ + { + "text": "# PDF Processing Skill\n\n1. Identify the PDF file path.\n2. Extract the text.\n3. Summarize it.\n\n---\nLocation: /tmp/agent_skills_capture/skills/pdf-processing/SKILL.md" + } + ] + } + } + ] + }, + { + "role": "assistant", + "content": [ + { + "toolUse": { + "toolUseId": "tu-2", + "name": "skills", + "input": { + "skill_name": "spreadsheet-analysis" + } + } + } + ] + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "tu-2", + "status": "success", + "content": [ + { + "text": "# Spreadsheet Analysis\n1. Inspect.\n\n---\nLocation: /tmp/agent_skills_capture/skills/spreadsheet-analysis/SKILL.md" + } + ] + } + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "Done." + } + ] + } + ] + }, + "repeated_load": { + "system_prompt": [ + { + "text": "You are a helpful agent." + }, + { + "text": "\n\npdf-processing\nUse this skill when the task requires reading or extracting text from PDF files.\n/tmp/agent_skills_capture/skills/pdf-processing/SKILL.md\n\n\nspreadsheet-analysis\nAnalyze, edit, or generate spreadsheets.\n/tmp/agent_skills_capture/skills/spreadsheet-analysis/SKILL.md\n\n" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "text": "Extract text from report.pdf" + } + ] + }, + { + "role": "assistant", + "content": [ + { + "toolUse": { + "toolUseId": "tu-1", + "name": "skills", + "input": { + "skill_name": "pdf-processing" + } + } + } + ] + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "tu-1", + "status": "success", + "content": [ + { + "text": "# PDF Processing Skill\n\n1. Identify the PDF file path.\n2. Extract the text.\n3. Summarize it.\n\n---\nLocation: /tmp/agent_skills_capture/skills/pdf-processing/SKILL.md" + } + ] + } + } + ] + }, + { + "role": "assistant", + "content": [ + { + "toolUse": { + "toolUseId": "tu-2", + "name": "skills", + "input": { + "skill_name": "pdf-processing" + } + } + } + ] + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "tu-2", + "status": "success", + "content": [ + { + "text": "# PDF Processing Skill\n\n1. Identify the PDF file path.\n2. Extract the text.\n3. Summarize it.\n\n---\nLocation: /tmp/agent_skills_capture/skills/pdf-processing/SKILL.md" + } + ] + } + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "Done." + } + ] + } + ] + } +} diff --git a/tests/strands_evals/extractors/skill_fixtures.py b/tests/strands_evals/extractors/skill_fixtures.py new file mode 100644 index 00000000..cddfa21e --- /dev/null +++ b/tests/strands_evals/extractors/skill_fixtures.py @@ -0,0 +1,212 @@ +"""Golden fixtures for the skill parsing helpers. + +Cross-harness raw-message-list shapes plus near-miss cases. Session-object +fixtures are built in the test module (they need real trace types); these are +the harness-native raw shapes the helpers must also handle. +""" + +PDF_DESCRIPTION = "Use this skill when the task requires reading or extracting text from PDF files." + +# The available-skills block a harness injects into the system prompt. +AVAILABLE_BLOCK = f"""You are a helpful agent. + + + +pdf-processing +{PDF_DESCRIPTION} +/skills/pdf-processing/SKILL.md + + +spreadsheet-analysis +Analyze, edit, or generate spreadsheets. +/skills/spreadsheet-analysis/SKILL.md + + +""" + +SKILL_BODY = "# PDF Processing Skill\n\n1. Identify the PDF file path.\n2. Extract the text.\n3. Summarize it." + + +def _tool_use(tool_use_id, name, inp): + return {"role": "assistant", "content": [{"toolUse": {"toolUseId": tool_use_id, "name": name, "input": inp}}]} + + +def _tool_result(tool_use_id, text): + return {"role": "user", "content": [{"toolResult": {"toolUseId": tool_use_id, "content": [{"text": text}]}}]} + + +# Strands native in-memory message shape: reserved `skills` tool, arg `skill_name`, +# body returned in the following user message's toolResult. +STRANDS_MESSAGES = [ + {"role": "system", "content": AVAILABLE_BLOCK}, + {"role": "user", "content": [{"text": "Extract text from report.pdf"}]}, + { + "role": "assistant", + "content": [ + {"text": "I'll use the pdf-processing skill."}, + {"toolUse": {"toolUseId": "tu-1", "name": "skills", "input": {"skill_name": "pdf-processing"}}}, + ], + }, + _tool_result("tu-1", SKILL_BODY), + {"role": "assistant", "content": [{"text": "Done."}]}, +] + +# Claude Code shape: reserved `Skill` tool, arg `skill`. (Body arrives in a +# following user message rather than the launch acknowledgement.) +CLAUDE_CODE_MESSAGES = [ + {"role": "system", "content": AVAILABLE_BLOCK}, + { + "type": "assistant", + "message": { + "role": "assistant", + "content": [ + { + "type": "tool_use", + "id": "cc-1", + "name": "Skill", + "input": {"skill": "pdf-processing", "args": "report.pdf"}, + } + ], + }, + }, + { + "type": "user", + "message": { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": "cc-1", + "content": "Launching skill: pdf-processing", + } + ], + }, + }, + { + "role": "user", + "content": (f"Base directory for this skill: /skills/pdf-processing\n\n{SKILL_BODY}"), + }, +] + +# Gemini CLI shape: reserved `activate_skill` tool, arg `name`. +GEMINI_MESSAGES = [ + {"role": "system", "content": AVAILABLE_BLOCK}, + _tool_use("g-1", "activate_skill", {"name": "pdf-processing"}), + _tool_result("g-1", f"{SKILL_BODY}"), +] + +CODEX_AVAILABLE_PROMPT = """# Skills +### Available skills +- pdf-processing: Use this skill for PDFs. (file: /skills/pdf-processing/SKILL.md) +### How to use skills +- Read the source before applying a skill. +""" + +CODEX_MESSAGES = [ + {"role": "system", "content": CODEX_AVAILABLE_PROMPT}, + { + "type": "item.completed", + "item": { + "type": "command_execution", + "command": "/bin/bash -lc \"sed -n '1,220p' /skills/pdf-processing/SKILL.md\"", + "status": "completed", + "exit_code": 0, + "aggregated_output": SKILL_BODY, + }, + }, +] + +OPENAI_AGENTS_MESSAGES = [ + {"role": "system", "content": CODEX_AVAILABLE_PROMPT}, + _tool_use("oa-1", "load_skill", {"skill_name": "pdf-processing"}), + _tool_result( + "oa-1", + '{"status": "loaded", "skill_name": "pdf-processing", "path": ".agents/pdf-processing"}', + ), + _tool_use("oa-2", "read_file", {"path": ".agents/pdf-processing/SKILL.md"}), + _tool_result("oa-2", SKILL_BODY), +] + +FAILED_LOAD_MESSAGES = [ + _tool_use("fail-1", "skills", {"skill_name": "pdf-processing"}), + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "fail-1", + "status": "error", + "content": [{"text": "skill not found"}], + } + } + ], + }, +] + +BODY_MISSING_MESSAGES = [ + _tool_use("missing-1", "load_skill", {"skill_name": "pdf-processing"}), + _tool_result( + "missing-1", + '{"status": "loaded", "skill_name": "pdf-processing", "path": ".agents/pdf-processing"}', + ), +] + +DUPLICATE_LOAD_MESSAGES = [ + _tool_use("dup-1", "skills", {"skill_name": "pdf-processing"}), + _tool_result("dup-1", SKILL_BODY), + _tool_use("dup-2", "skills", {"skill_name": "pdf-processing"}), + _tool_result("dup-2", SKILL_BODY), +] + +GEMINI_STREAM_MESSAGES = [ + {"tool_name": "activate_skill", "parameters": {"name": "pdf-processing"}}, + { + "type": "tool_result", + "status": "success", + "llmContent": f"{SKILL_BODY}", + }, +] + +GOOGLE_ADK_MESSAGES = [ + {"name": "myapp_load_skill", "args": {"skill_name": "pdf-processing"}, "id": "adk-1"}, + { + "type": "tool_response", + "id": "adk-1", + "response": {"skill_name": "pdf-processing", "instructions": SKILL_BODY}, + }, +] + +OPENHANDS_MESSAGES = [ + {"kind": "InvokeSkillAction", "name": "pdf-processing"}, + { + "kind": "InvokeSkillObservation", + "skill_name": "pdf-processing", + "is_error": False, + "content": [{"type": "text", "text": SKILL_BODY}], + }, +] + +# Near-miss: a skill NAME mentioned in prose / a non-skill tool call. Neither is an invocation. +NEAR_MISS_MESSAGES = [ + {"role": "system", "content": AVAILABLE_BLOCK}, + { + "role": "assistant", + "content": [{"text": "I could use the pdf-processing skill, but let me just read the file."}], + }, + _tool_use("nm-1", "file_read", {"path": "/skills/pdf-processing/SKILL.md"}), +] + +# No available block, no invocation. +EMPTY_MESSAGES = [ + {"role": "user", "content": [{"text": "hello"}]}, + {"role": "assistant", "content": [{"text": "hi"}]}, +] + +# Multiple invocations in one run (two distinct skills). +MULTI_INVOKE_MESSAGES = [ + {"role": "system", "content": AVAILABLE_BLOCK}, + _tool_use("m-1", "skills", {"skill_name": "pdf-processing"}), + _tool_result("m-1", SKILL_BODY), + _tool_use("m-2", "skills", {"skill_name": "spreadsheet-analysis"}), + _tool_result("m-2", "# Spreadsheet Analysis\n1. Inspect."), +] diff --git a/tests/strands_evals/extractors/test_skills.py b/tests/strands_evals/extractors/test_skills.py new file mode 100644 index 00000000..dc4466f1 --- /dev/null +++ b/tests/strands_evals/extractors/test_skills.py @@ -0,0 +1,1405 @@ +"""Unit tests for the skill parsing helpers (parse_available_skills, extract_selected_skills).""" + +import json +from datetime import datetime +from pathlib import Path + +import pytest + +from strands_evals.extractors import ( + AvailableSkill, + InvokedSkill, + advertised_a_catalog, + extract_selected_skills, + extract_skill_load_events, + parse_available_skills, +) +from strands_evals.types.trace import ( + AgentInvocationSpan, + AssistantMessage, + Session, + SpanInfo, + TextContent, + ToolCall, + ToolCallContent, + ToolExecutionSpan, + ToolResult, + ToolResultContent, + Trace, + UserMessage, +) + +from . import skill_fixtures as fx + + +def _loaded(name: str, body: str | None) -> InvokedSkill: + """A skill the harness loaded successfully, for comparing whole extraction results.""" + return InvokedSkill(name, body) + + +def _failed(name: str, error: str | None = None) -> InvokedSkill: + """A skill the agent asked for and the harness refused, with what the harness said.""" + return InvokedSkill(name, None, status="failed", error=error) + + +# ---- raw message-list path -------------------------------------------------- + + +def test_available_skills_from_strands_list(): + skills = parse_available_skills(fx.STRANDS_MESSAGES) + assert skills == [ + AvailableSkill("pdf-processing", fx.PDF_DESCRIPTION), + AvailableSkill("spreadsheet-analysis", "Analyze, edit, or generate spreadsheets."), + ] + + +def test_available_skills_unescapes_xml_entities(): + prompt = ( + "research&review" + "Compare A < B & report." + ) + + assert parse_available_skills(prompt) == [ + AvailableSkill("research&review", "Compare A < B & report."), + ] + + +def test_selected_skills_from_strands_list_with_body(): + invoked = extract_selected_skills(fx.STRANDS_MESSAGES) + assert len(invoked) == 1 + assert invoked[0].name == "pdf-processing" + assert invoked[0].body is not None and "PDF Processing Skill" in invoked[0].body + + +@pytest.mark.parametrize( + "messages,expected_name,expect_body_substr", + [ + (fx.STRANDS_MESSAGES, "pdf-processing", "PDF Processing Skill"), + (fx.CLAUDE_CODE_MESSAGES, "pdf-processing", "PDF Processing Skill"), + (fx.CODEX_MESSAGES, "pdf-processing", "PDF Processing Skill"), + (fx.OPENAI_AGENTS_MESSAGES, "pdf-processing", "PDF Processing Skill"), + (fx.GEMINI_MESSAGES, "pdf-processing", ""), + (fx.GEMINI_STREAM_MESSAGES, "pdf-processing", ""), + (fx.GOOGLE_ADK_MESSAGES, "pdf-processing", "PDF Processing Skill"), + (fx.OPENHANDS_MESSAGES, "pdf-processing", "PDF Processing Skill"), + ], +) +def test_selected_skills_cross_harness(messages, expected_name, expect_body_substr): + invoked = extract_selected_skills(messages) + assert len(invoked) == 1 + assert invoked[0].name == expected_name + assert invoked[0].body is not None and expect_body_substr in invoked[0].body + + +def test_near_miss_is_not_an_invocation(): + # A skill name mentioned in prose is not an invocation, and here the file_read + # of the SKILL.md path is rejected specifically because its result carried NO + # skill body. A SKILL.md read whose result DOES return a body is a valid + # filesystem-skill load (Codex / OpenAI Agents); see + # test_session_skill_file_read_with_body. Do not remove the file-read branch. + assert extract_selected_skills(fx.NEAR_MISS_MESSAGES) == [] + # The available block is still recoverable from the same messages. + assert [s.name for s in parse_available_skills(fx.NEAR_MISS_MESSAGES)] == [ + "pdf-processing", + "spreadsheet-analysis", + ] + + +def test_empty_messages(): + assert parse_available_skills(fx.EMPTY_MESSAGES) == [] + assert extract_selected_skills(fx.EMPTY_MESSAGES) == [] + + +def test_a_harness_that_advertised_nothing_is_not_a_harness_that_recorded_nothing(): + """`parse_available_skills` returns [] for two different runs, and callers must tell them apart. + + A harness that mounted no skills said so; the Strands plugin emits the block with + "No skills are currently available." A harness that never records the offered set said + nothing at all, which is the Claude Code and Claude Agent SDK case. Showing a judge an empty + set for the second one invites it to conclude the invoked skill did not exist. + """ + advertised_none = [ + {"role": "system", "content": "\nNo skills are currently available.\n"} + ] + + assert parse_available_skills(advertised_none) == [] + assert parse_available_skills(fx.EMPTY_MESSAGES) == [] + # Same parse result, different runs. + assert advertised_a_catalog(advertised_none) is True + assert advertised_a_catalog(fx.EMPTY_MESSAGES) is False + + +def test_a_populated_catalog_counts_as_advertised(): + assert advertised_a_catalog(fx.STRANDS_MESSAGES) is True + assert advertised_a_catalog(None) is False + + +def test_multiple_invocations_in_order(): + invoked = extract_selected_skills(fx.MULTI_INVOKE_MESSAGES) + assert [i.name for i in invoked] == ["pdf-processing", "spreadsheet-analysis"] + assert all(i.body for i in invoked) + + +def test_available_skills_from_markdown_section(): + assert parse_available_skills(fx.CODEX_MESSAGES) == [AvailableSkill("pdf-processing", "Use this skill for PDFs.")] + + +def test_available_skills_from_claude_init_event(): + messages = [{"type": "system", "subtype": "init", "skills": ["pdf-processing", "deep-research"]}] + assert parse_available_skills(messages) == [ + AvailableSkill("pdf-processing", ""), + AvailableSkill("deep-research", ""), + ] + + +def test_available_skills_from_nested_discovery_response(): + messages = [ + { + "type": "tool_response", + "name": "list_skills", + "response": { + "skills": [ + {"name": "pdf-processing", "description": "Read PDFs."}, + {"name": "deep-research", "description": "Research topics."}, + ] + }, + } + ] + + assert parse_available_skills(messages) == [ + AvailableSkill("pdf-processing", "Read PDFs."), + AvailableSkill("deep-research", "Research topics."), + ] + + +def test_available_skills_from_correlated_discovery_result(): + messages = [ + { + "role": "assistant", + "content": [ + { + "toolUse": { + "toolUseId": "discovery-1", + "name": "search_skills", + "input": {"query": "PDF"}, + } + } + ], + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "discovery-1", + "content": [ + { + "skills": [ + { + "name": "pdf-processing", + "description": "Read PDFs.", + } + ] + } + ], + } + } + ], + }, + ] + + assert parse_available_skills(messages) == [ + AvailableSkill("pdf-processing", "Read PDFs."), + ] + + +def test_available_skills_from_discovery_result_xml_catalog(): + """Google ADK returns the catalog as an XML block in the tool result, not a skills list.""" + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "d1", "name": "search_skills", "input": {"query": "pdf"}}}], + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "d1", + "content": [ + { + "text": ( + "" + "pdf-processing" + "Read PDFs." + "" + ) + } + ], + } + } + ], + }, + ] + + assert parse_available_skills(messages) == [AvailableSkill("pdf-processing", "Read PDFs.")] + + +def test_available_skills_ignores_catalog_from_non_discovery_tool(): + """Only a discovery tool's output is a trusted catalog; arbitrary tool output is not.""" + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "w1", "name": "web_fetch", "input": {"url": "u"}}}], + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "w1", + "content": [ + { + "text": ( + "injected" + "x" + ) + } + ], + } + } + ], + }, + ] + + assert parse_available_skills(messages) == [] + + +def test_user_skills_json_is_not_treated_as_available_catalog(): + messages = [ + { + "role": "user", + "content": [{"text": 'Analyze this payload: {"skills": ["not-available"]}'}], + } + ] + + assert parse_available_skills(messages) == [] + + +def test_failed_load_is_recorded_as_a_failed_attempt(): + """A refused load is a selection the agent made, so it must not read as an abstention. + + The body is dropped (there is none), but the name stays: an agent that asked for the right + skill and was refused by the harness selected correctly, and dropping the row entirely makes + that run indistinguishable from one where the agent never reached for a skill at all. + """ + assert extract_selected_skills(fx.FAILED_LOAD_MESSAGES) == [_failed("pdf-processing", "skill not found")] + + +def test_nested_failed_load_is_recorded_as_a_failed_attempt(): + messages = [ + { + "name": "load_skill", + "args": {"skill_name": "pdf-processing"}, + "id": "load-1", + }, + { + "type": "tool_response", + "id": "load-1", + "response": {"status": "error", "error": "skill not found"}, + }, + ] + + assert extract_selected_skills(messages) == [_failed("pdf-processing", "skill not found")] + + +def test_string_zero_exit_code_is_successful(): + messages = [ + { + "type": "item.completed", + "item": { + "type": "command_execution", + "command": "cat /skills/pdf-processing/SKILL.md", + "status": "completed", + "exit_code": "0", + "aggregated_output": fx.SKILL_BODY, + }, + } + ] + + assert extract_selected_skills(messages) == [_loaded("pdf-processing", fx.SKILL_BODY)] + + +def test_successful_load_without_body_preserves_invocation(): + assert extract_selected_skills(fx.BODY_MISSING_MESSAGES) == [_loaded("pdf-processing", None)] + + +def test_duplicate_loads_are_coalesced(): + invoked = extract_selected_skills(fx.DUPLICATE_LOAD_MESSAGES) + assert invoked == [_loaded("pdf-processing", fx.SKILL_BODY)] + + +def test_selected_skills_from_typed_messages(): + messages = [ + AssistantMessage( + content=[ + TextContent(text="Loading the PDF skill"), + ToolCallContent( + name="skills", + arguments={"skill_name": "pdf-processing"}, + tool_call_id="typed-1", + ), + ] + ), + UserMessage( + content=[ + ToolResultContent( + content=fx.SKILL_BODY, + tool_call_id="typed-1", + ) + ] + ), + ] + + invoked = extract_selected_skills(messages) + + assert invoked == [_loaded("pdf-processing", fx.SKILL_BODY)] + + +def test_unsupported_trajectory_type_returns_empty(): + assert parse_available_skills(None) == [] + assert extract_selected_skills(None) == [] + assert parse_available_skills("not a trajectory") == [] + + +def test_parse_available_from_bare_system_prompt_string(): + # Some session mappers store the system prompt separately from the message list; + # parse_available_skills accepts the bare prompt string too. + skills = parse_available_skills(fx.AVAILABLE_BLOCK) + assert [s.name for s in skills] == ["pdf-processing", "spreadsheet-analysis"] + + +# ---- Session path ----------------------------------------------------------- + + +def _span_info() -> SpanInfo: + return SpanInfo(session_id="s", start_time=datetime(2026, 7, 14), end_time=datetime(2026, 7, 14)) + + +def _session(spans) -> Session: + return Session(session_id="s", traces=[Trace(trace_id="t", session_id="s", spans=spans)]) + + +def test_session_available_from_system_prompt(): + agent_span = AgentInvocationSpan( + span_info=_span_info(), + user_prompt="do pdf", + agent_response="done", + available_tools=[], + system_prompt=fx.AVAILABLE_BLOCK, + ) + skills = parse_available_skills(_session([agent_span])) + assert [s.name for s in skills] == ["pdf-processing", "spreadsheet-analysis"] + + +def test_session_selected_from_tool_execution_span(): + tool_span = ToolExecutionSpan( + span_info=_span_info(), + tool_call=ToolCall(name="skills", arguments={"skill_name": "pdf-processing"}, tool_call_id="tu-1"), + tool_result=ToolResult(content=fx.SKILL_BODY), + ) + invoked = extract_selected_skills(_session([tool_span])) + assert len(invoked) == 1 + assert invoked[0].name == "pdf-processing" + assert "PDF Processing Skill" in invoked[0].body + + +def test_session_non_skill_tool_ignored(): + tool_span = ToolExecutionSpan( + span_info=_span_info(), + tool_call=ToolCall(name="calculator", arguments={"expression": "2+2"}, tool_call_id="c-1"), + tool_result=ToolResult(content="4"), + ) + assert extract_selected_skills(_session([tool_span])) == [] + + +def test_session_failed_skill_load_is_recorded_as_a_failed_attempt(): + tool_span = ToolExecutionSpan( + span_info=_span_info(), + tool_call=ToolCall(name="skills", arguments={"skill_name": "pdf-processing"}, tool_call_id="tu-1"), + tool_result=ToolResult(content="skill not found", error="error"), + ) + assert extract_selected_skills(_session([tool_span])) == [_failed("pdf-processing", "error")] + + +def test_session_failed_read_of_a_skill_file_is_not_an_invocation(): + """A read of a `SKILL.md` that errored recovered no name and no body, so there is nothing + to report: unlike a reserved skill tool, the path read carries no declared skill name.""" + tool_span = ToolExecutionSpan( + span_info=_span_info(), + tool_call=ToolCall(name="read_file", arguments={"path": "/skills/pdf/SKILL.md"}, tool_call_id="r-1"), + tool_result=ToolResult(content="No such file", error="error"), + ) + assert extract_selected_skills(_session([tool_span])) == [] + + +def test_session_skill_file_read_with_body(): + tool_span = ToolExecutionSpan( + span_info=_span_info(), + tool_call=ToolCall( + name="read_file", + arguments={"path": "/skills/pdf-processing/SKILL.md"}, + tool_call_id="read-1", + ), + tool_result=ToolResult(content=fx.SKILL_BODY), + ) + assert extract_selected_skills(_session([tool_span])) == [_loaded("pdf-processing", fx.SKILL_BODY)] + + +def test_session_skill_file_read_uses_frontmatter_name(): + body = "---\nname: canonical-skill\ndescription: Test skill.\n---\n# Steps\n1. Test." + tool_span = ToolExecutionSpan( + span_info=_span_info(), + tool_call=ToolCall( + name="read_file", + arguments={"path": "/skills/directory-alias/SKILL.md"}, + tool_call_id="read-1", + ), + tool_result=ToolResult(content=body), + ) + + assert extract_selected_skills(_session([tool_span])) == [_loaded("canonical-skill", body)] + + +def test_opaque_load_and_alias_path_read_are_coalesced(): + body = "---\nname: canonical-skill\ndescription: Test skill.\n---\n# Steps\n1. Test." + messages = [ + { + "role": "assistant", + "content": [ + { + "toolUse": { + "toolUseId": "load-1", + "name": "load_skill", + "input": {"skill_name": "canonical-skill"}, + } + } + ], + }, + { + "role": "user", + "content": [{"toolResult": {"toolUseId": "load-1", "content": [{"text": "Loaded skill"}]}}], + }, + { + "role": "assistant", + "content": [ + { + "toolUse": { + "toolUseId": "read-1", + "name": "read_file", + "input": {"path": "/skills/directory-alias/SKILL.md"}, + } + } + ], + }, + { + "role": "user", + "content": [{"toolResult": {"toolUseId": "read-1", "content": [{"text": body}]}}], + }, + ] + + assert extract_selected_skills(messages) == [_loaded("canonical-skill", body)] + + +def test_command_execution_skill_read_uses_frontmatter_name(): + body = "---\nname: canonical-skill\ndescription: Test skill.\n---\n# Steps\n1. Test." + messages = [ + { + "type": "item.completed", + "item": { + "type": "command_execution", + "command": "cat /skills/directory-alias/SKILL.md", + "status": "completed", + "exit_code": 0, + "aggregated_output": body, + }, + } + ] + + assert extract_selected_skills(messages) == [_loaded("canonical-skill", body)] + + +def _shell_command(command: str, output: str = "col1,col2\n1,2", exit_code: int = 0): + return [ + { + "type": "item.completed", + "item": { + "type": "command_execution", + "command": command, + "status": "completed", + "exit_code": exit_code, + "aggregated_output": output, + }, + } + ] + + +@pytest.mark.parametrize( + "command", + [ + "cat draft.md > /skills/my-new-skill/SKILL.md", # writes a skill, does not load one + "echo '# Steps' >> /skills/my-new-skill/SKILL.md", + "sed -i 's/a/b/' /skills/pdf-processing/SKILL.md", # edits in place + "sed --in-place 's/a/b/' /skills/pdf-processing/SKILL.md", + "echo '# Steps' | tee /skills/my-new-skill/SKILL.md", + "cat data.csv; ls -l /skills/pdf-processing/SKILL.md", # verb and path, different commands + "grep -n Extract /skills/pdf-processing/SKILL.md", # not a read of the whole file + ], +) +def test_shell_command_that_does_not_read_a_skill_is_not_an_invocation(command): + """The read verb has to own the path, not merely appear somewhere in the same line. + + Searching for a verb and a path independently makes writes and unrelated work look like + skill loads, and the phantom body is whatever the command happened to print. + """ + assert extract_selected_skills(_shell_command(command)) == [] + + +@pytest.mark.parametrize( + "command", + [ + "cat /skills/pdf-processing/SKILL.md", + "sed -n '1,220p' /skills/pdf-processing/SKILL.md", + "/bin/bash -lc \"sed -n '1,220p' /skills/pdf-processing/SKILL.md\"", # harness wrapper + "sudo cat /skills/pdf-processing/SKILL.md", + "cat /skills/pdf-processing/SKILL.md | head -20", # paged + "cd /tmp && cat /skills/pdf-processing/SKILL.md", # read in a later segment + "cat draft.md > /tmp/out.md; cat /skills/pdf-processing/SKILL.md", # write then read + ], +) +def test_shell_read_of_a_skill_is_an_invocation(command): + body = "# PDF Processing\n1. Identify the path.\n2. Extract." + + assert extract_selected_skills(_shell_command(command, output=body)) == [_loaded("pdf-processing", body)] + + +@pytest.mark.parametrize( + "command,expected", + [ + ("cat data.csv > /skills/my-new-skill/SKILL.md", []), + ("cat /skills/pdf-processing/SKILL.md", [_loaded("pdf-processing", "# PDF Processing\n1. Extract.")]), + ], +) +def test_shell_tool_read_uses_the_same_rule_as_command_execution(command, expected): + """A `bash` tool call and a Codex `command_execution` event are the same shell command.""" + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "b1", "name": "bash", "input": {"command": command}}}], + }, + { + "role": "user", + "content": [{"toolResult": {"toolUseId": "b1", "content": [{"text": "# PDF Processing\n1. Extract."}]}}], + }, + ] + + assert extract_selected_skills(messages) == expected + + +def test_sigpipe_exit_code_does_not_discard_a_read_body(): + """`cat SKILL.md | head -20` exits 141 once head closes the pipe, having printed the body.""" + body = "# PDF Processing\n1. Identify the path." + + invoked = extract_selected_skills( + _shell_command("cat /skills/pdf-processing/SKILL.md | head -20", output=body, exit_code=141) + ) + + assert invoked == [_loaded("pdf-processing", body)] + + +def test_failing_read_is_still_discarded(): + """Only SIGPIPE is tolerated; a read that actually failed carries no body.""" + assert extract_selected_skills(_shell_command("cat /skills/pdf/SKILL.md", output="No such file", exit_code=1)) == [] + + +def test_malformed_frontmatter_body_falls_back_to_path_name(): + # A SKILL.md whose frontmatter is not parseable YAML must not abort the whole + # extraction; the name degrades to the directory alias and the body is kept. + body = "---\nname: [unclosed\ndescription: broken\n---\n# Steps\n1. Test." + messages = [ + { + "type": "item.completed", + "item": { + "type": "command_execution", + "command": "cat /skills/directory-alias/SKILL.md", + "status": "completed", + "exit_code": 0, + "aggregated_output": body, + }, + } + ] + + assert extract_selected_skills(messages) == [_loaded("directory-alias", body)] + + +def test_unkeyed_results_are_not_reused_across_skill_calls(): + messages = [ + {"tool_name": "activate_skill", "parameters": {"name": "first"}}, + {"type": "tool_result", "status": "success", "llmContent": "first body"}, + {"tool_name": "activate_skill", "parameters": {"name": "second"}}, + {"type": "tool_result", "status": "success", "llmContent": "second body"}, + ] + + assert extract_selected_skills(messages) == [ + _loaded("first", "first body"), + _loaded("second", "second body"), + ] + + +def test_session_available_absent_when_no_system_prompt(): + # Mapped sessions may drop the system prompt; then the block is not recoverable + # from the Session (would fall back to a raw message list in practice). + agent_span = AgentInvocationSpan( + span_info=_span_info(), + user_prompt="do pdf", + agent_response="done", + available_tools=[], + system_prompt=None, + ) + assert parse_available_skills(_session([agent_span])) == [] + + +def test_google_adk_function_call_shape(): + """Google ADK emits Gemini content parts and nests its payload under response/result.""" + messages = [ + { + "role": "model", + "content": [{"functionCall": {"id": "c1", "name": "list_skills", "args": {}}}], + }, + { + "role": "user", + "content": [ + { + "functionResponse": { + "id": "c1", + "name": "list_skills", + "response": { + "result": ( + "pdf-processing" + "Read PDFs." + ) + }, + } + } + ], + }, + { + "role": "model", + "content": [{"functionCall": {"id": "c2", "name": "load_skill", "args": {"skill_name": "pdf-processing"}}}], + }, + { + "role": "user", + "content": [ + { + "functionResponse": { + "id": "c2", + "name": "load_skill", + "response": {"skill_name": "pdf-processing", "instructions": "## Phase 1\nRun pdfinfo."}, + } + } + ], + }, + ] + + assert parse_available_skills(messages) == [AvailableSkill("pdf-processing", "Read PDFs.")] + invoked = extract_selected_skills(messages) + assert [s.name for s in invoked] == ["pdf-processing"] + assert invoked[0].body == "## Phase 1\nRun pdfinfo." + + +def test_load_acknowledgement_is_not_treated_as_a_body(): + """Gemini CLI's displayed output is a status line; scoring it as instructions would be wrong.""" + messages = [ + {"tool_name": "activate_skill", "parameters": {"name": "pdf-processing"}, "id": "g1"}, + { + "type": "tool_result", + "id": "g1", + "output": "Skill activated. Resources loaded from pdf-processing/", + }, + ] + + invoked = extract_selected_skills(messages) + + assert [s.name for s in invoked] == ["pdf-processing"] + assert invoked[0].body is None + + +# The three strings the Strands AgentSkills plugin returns instead of a skill body. They arrive +# marked successful, because `@tool` reports any plain string return as `status="success"`. +_AGENT_SKILLS_NON_BODIES = [ + "Skill 'pdf-processing' not found. Available skills: spreadsheet-analysis, docx-editing", + "Error: skill_name is required. Available skills: pdf-processing, spreadsheet-analysis", + "Skill 'pdf-processing' activated (no instructions available).", +] + + +@pytest.mark.parametrize("result_text", _AGENT_SKILLS_NON_BODIES) +def test_agent_skills_status_string_is_not_a_body(result_text): + """A refused or empty load carries no instructions, so the judge must not be handed one.""" + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "t1", "name": "skills", "input": {"skill_name": "pdf-processing"}}}], + }, + {"role": "user", "content": [{"toolResult": {"toolUseId": "t1", "content": [{"text": result_text}]}}]}, + ] + + invoked = extract_selected_skills(messages) + + assert [s.name for s in invoked] == ["pdf-processing"] + assert invoked[0].body is None + + +@pytest.mark.parametrize("result_text", _AGENT_SKILLS_NON_BODIES) +def test_agent_skills_status_string_is_not_a_body_on_the_session_path(result_text): + """Same on the Session path: the plugin's string lands in `content` with `error` unset.""" + tool_span = ToolExecutionSpan( + span_info=_span_info(), + tool_call=ToolCall(name="skills", arguments={"skill_name": "pdf-processing"}, tool_call_id="tu-1"), + tool_result=ToolResult(content=result_text), + ) + + invoked = extract_selected_skills(_session([tool_span])) + + assert [s.name for s in invoked] == ["pdf-processing"] + assert invoked[0].body is None + + +# The two plugin strings above that mean no skill was loaded. The third +# ("activated (no instructions available)") is a real load of a skill with no body, so it stays +# `loaded`: the agent did receive the skill, there was just nothing prescriptive in it. +_AGENT_SKILLS_REFUSALS = _AGENT_SKILLS_NON_BODIES[:2] + + +@pytest.mark.parametrize("result_text", _AGENT_SKILLS_REFUSALS) +def test_agent_skills_refusal_is_recorded_as_a_failed_load(result_text): + """A mistyped lookup key is a refused load, not a load whose body went uncaptured. + + `@tool` marks the plugin's plain-string return `status="success"`, so the text is the only + signal. Without reading it, requesting `pdf-procesing` for a registered `pdf-processing` + reports as a successful invocation and the adherence judge blames the agent for not + following instructions it never received. + """ + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "t1", "name": "skills", "input": {"skill_name": "pdf-procesing"}}}], + }, + {"role": "user", "content": [{"toolResult": {"toolUseId": "t1", "content": [{"text": result_text}]}}]}, + ] + + invoked = extract_selected_skills(messages) + + assert invoked == [_failed("pdf-procesing", result_text)] + + +@pytest.mark.parametrize("result_text", _AGENT_SKILLS_REFUSALS) +def test_agent_skills_refusal_is_recorded_as_a_failed_load_on_the_session_path(result_text): + tool_span = ToolExecutionSpan( + span_info=_span_info(), + tool_call=ToolCall(name="skills", arguments={"skill_name": "pdf-procesing"}, tool_call_id="tu-1"), + tool_result=ToolResult(content=result_text), + ) + + invoked = extract_selected_skills(_session([tool_span])) + + assert invoked == [_failed("pdf-procesing", result_text)] + + +def test_refusal_message_distinguishes_a_bad_name_from_an_empty_mount(): + """Both runs fail the same way; only the harness's message says what to fix. + + A misspelled skill name is the agent's mistake, an empty catalog is the harness's. Without + the message both read as "the load failed" and whoever reads the result cannot tell which. + """ + + def refused(text: str) -> list[dict]: + return [ + { + "role": "assistant", + "content": [ + {"toolUse": {"toolUseId": "t1", "name": "skills", "input": {"skill_name": "pdf-procesing"}}} + ], + }, + {"role": "user", "content": [{"toolResult": {"toolUseId": "t1", "content": [{"text": text}]}}]}, + ] + + typo = "Skill 'pdf-procesing' not found. Available skills: pdf-processing" + empty = "Skill 'pdf-procesing' not found. Available skills: (none)" + + assert extract_selected_skills(refused(typo))[0].error == typo + assert extract_selected_skills(refused(empty))[0].error == empty + + +def test_a_loaded_skill_carries_no_refusal_message(): + assert extract_selected_skills(fx.STRANDS_MESSAGES)[0].error is None + + +def test_a_skill_that_activated_with_no_instructions_still_counts_as_loaded(): + """An empty skill is not a refusal: the agent got what it asked for, body and all.""" + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "t1", "name": "skills", "input": {"skill_name": "pdf-processing"}}}], + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "t1", + "content": [{"text": "Skill 'pdf-processing' activated (no instructions available)."}], + } + } + ], + }, + ] + + assert extract_selected_skills(messages) == [_loaded("pdf-processing", None)] + + +def test_body_mentioning_a_missing_file_is_kept(): + """The load-error filter matches a whole status line, not the words wherever they appear.""" + body = "# PDF Processing\n\n1. If the skill file is not found, stop.\n2. Extract the text." + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "t1", "name": "skills", "input": {"skill_name": "pdf-processing"}}}], + }, + {"role": "user", "content": [{"toolResult": {"toolUseId": "t1", "content": [{"text": body}]}}]}, + ] + + assert extract_selected_skills(messages)[0].body == body + + +def test_skill_read_twice_keeps_the_fullest_body(): + """Repeated reads of one skill collapse, keeping the read that carried the whole file.""" + body = "---\nname: chart-builder\ndescription: Charts.\n---\n\n## Phase 1\nBuild the chart.\n" + messages = [ + { + "type": "command_execution", + "command": "sed -n '1,3p' /skills/chart_builder/SKILL.md", + "aggregated_output": "---\nname: chart-builder\ndescription: Charts.\n", + }, + { + "type": "command_execution", + "command": "cat /skills/chart-builder/SKILL.md", + "aggregated_output": body, + }, + ] + + invoked = extract_selected_skills(messages) + + assert len(invoked) == 1 + assert invoked[0].name == "chart-builder" + assert "## Phase 1" in (invoked[0].body or "") + + +def test_body_prefixed_by_an_acknowledgement_is_kept(): + """A status line ahead of the instructions must not discard the instructions with it.""" + body = "# PDF Processing\n\n1. Identify the PDF path.\n2. Extract the text." + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "t1", "name": "skills", "input": {"skill_name": "pdf-processing"}}}], + }, + { + "role": "user", + "content": [{"toolResult": {"toolUseId": "t1", "content": [{"text": f"Skill activated.\n\n{body}"}]}}], + }, + ] + + invoked = extract_selected_skills(messages) + + assert [s.name for s in invoked] == ["pdf-processing"] + assert "1. Identify the PDF path." in (invoked[0].body or "") + + +def test_skill_names_differing_only_by_a_dot_stay_separate(): + """`.` is legal in a skill name, so `data.clean` and `data-clean` are two skills.""" + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "a", "name": "skills", "input": {"skill_name": "data.clean"}}}], + }, + {"role": "user", "content": [{"toolResult": {"toolUseId": "a", "content": [{"text": "# Dotted\n1. a"}]}}]}, + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "b", "name": "skills", "input": {"skill_name": "data-clean"}}}], + }, + { + "role": "user", + "content": [{"toolResult": {"toolUseId": "b", "content": [{"text": "# Hyphenated\n1. b\n2. c"}]}}], + }, + ] + + assert [s.name for s in extract_selected_skills(messages)] == ["data.clean", "data-clean"] + + +def test_a_longer_read_of_the_same_skill_only_wins_if_it_contains_what_was_kept(): + """The containment rule, on the shape that actually exercises it: two real read verbs. + + A paged read recovers part of a skill, and a later read of the same skill returns unrelated + but longer output. Length alone would let the second displace the first, so the judge would be + handed stray stdout as the skill's instructions. The kept body has to be a subset of the + challenger for it to win. + """ + partial = "---\nname: pdf-processing\n---\n# Real\n1. first step" + messages = [ + { + "type": "command_execution", + "command": "sed -n '1,5p' /skills/pdf-processing/SKILL.md", + "exit_code": 0, + "aggregated_output": partial, + }, + { + # A read verb, the same skill path, and longer output that does NOT contain `partial`. + "type": "command_execution", + "command": "cat /skills/pdf-processing/SKILL.md", + "exit_code": 0, + "aggregated_output": "unrelated stdout that happens to be much longer " * 4, + }, + ] + + invoked = extract_selected_skills(messages) + + assert [s.name for s in invoked] == ["pdf-processing"] + assert invoked[0].body == partial + + +def test_longer_unrelated_output_does_not_displace_a_recovered_body(): + """Only a superset of what was already recovered wins, so stray stdout cannot take over.""" + real_body = "---\nname: pdf-processing\n---\n# Real\n1. step" + messages = [ + { + "type": "command_execution", + "command": "cat /skills/pdf-processing/SKILL.md", + "exit_code": 0, + "aggregated_output": real_body, + }, + { + "type": "command_execution", + "command": "cat report.csv; ls -l /skills/pdf-processing/SKILL.md", + "exit_code": 0, + "aggregated_output": "col1,col2\n" + "x,y\n" * 40, + }, + ] + + invoked = extract_selected_skills(messages) + + assert len(invoked) == 1 + assert invoked[0].body == real_body + + +def _claude_launch(call_id: str, skill: str) -> list[dict]: + """A Claude Code `Skill` call and the launch acknowledgement that carries no body.""" + return [ + { + "type": "assistant", + "message": { + "role": "assistant", + "content": [{"type": "tool_use", "id": call_id, "name": "Skill", "input": {"skill": skill}}], + }, + }, + { + "type": "user", + "message": { + "role": "user", + "content": [{"type": "tool_result", "tool_use_id": call_id, "content": f"Launching skill: {skill}"}], + }, + }, + ] + + +def _claude_injected_body(skill: str, body: str) -> dict: + return {"role": "user", "content": f"Base directory for this skill: /skills/{skill}\n\n{body}"} + + +def test_parallel_claude_skill_calls_each_get_their_own_body(): + """Claude Code can launch several skills in one turn; each must get its own instructions. + + Taking the first injected body after the call index gives every skill the first skill's + steps, and the adherence judge then scores the agent against instructions it never received. + """ + messages = [ + *_claude_launch("cc-1", "pdf-processing"), + *_claude_launch("cc-2", "spreadsheet-analysis"), + _claude_injected_body("spreadsheet-analysis", "# Spreadsheet\n1. Open the sheet."), + _claude_injected_body("pdf-processing", "# PDF\n1. Extract the text."), + ] + + invoked = extract_selected_skills(messages) + + assert [s.name for s in invoked] == ["pdf-processing", "spreadsheet-analysis"] + assert "1. Extract the text." in (invoked[0].body or "") + assert "1. Open the sheet." in (invoked[1].body or "") + + +def test_single_claude_body_is_used_even_when_the_directory_is_an_alias(): + """One launch and one injected body pair up, since the directory can differ from the name.""" + messages = [ + *_claude_launch("cc-1", "pdf-processing"), + _claude_injected_body("directory-alias", "# PDF\n1. Extract the text."), + ] + + invoked = extract_selected_skills(messages) + + assert [s.name for s in invoked] == ["pdf-processing"] + assert "1. Extract the text." in (invoked[0].body or "") + + +def _load_attempt(call_id: str, skill: str, result: dict) -> list[dict]: + return [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": call_id, "name": "load_skill", "input": {"skill_name": skill}}}], + }, + {"role": "user", "content": [{"toolResult": {"toolUseId": call_id, **result}}]}, + ] + + +_REFUSED = {"status": "error", "content": [{"text": "skill not found"}]} +_LOADED_OPAQUE = {"content": [{"text": '{"status": "loaded", "path": ".agents/pdf-processing"}'}]} + + +def test_retry_after_a_refused_load_is_reported_as_loaded(): + """One success anywhere in the run means the agent got the skill. + + The retry here returns no body, so a merge that only compares body length would leave the + first attempt's refusal in place and report a skill the agent did receive as failed. + """ + messages = [ + *_load_attempt("1", "pdf-processing", _REFUSED), + *_load_attempt("2", "pdf-processing", _LOADED_OPAQUE), + ] + + assert extract_selected_skills(messages) == [_loaded("pdf-processing", None)] + + +def test_skill_refused_on_every_attempt_stays_failed(): + messages = [ + *_load_attempt("1", "pdf-processing", _REFUSED), + *_load_attempt("2", "pdf-processing", _REFUSED), + ] + + assert extract_selected_skills(messages) == [_failed("pdf-processing", "skill not found")] + + +def test_a_later_refusal_does_not_discard_a_recovered_body(): + """The agent already had the instructions; a failed re-load does not take them away.""" + body = "# PDF\n1. Extract the text." + messages = [ + *_load_attempt("1", "pdf-processing", {"content": [{"text": body}]}), + *_load_attempt("2", "pdf-processing", _REFUSED), + ] + + assert extract_selected_skills(messages) == [_loaded("pdf-processing", body)] + + +def test_unkeyed_result_of_an_unrelated_tool_is_not_taken_as_the_skill_body(): + """An unkeyed result only pairs with the call it follows, not the next unclaimed one. + + Here the skill call's own result is missing and a later tool's is not; pairing across the + intervening call attributes that tool's output to the skill as its instructions. + """ + messages = [ + {"tool_name": "activate_skill", "parameters": {"name": "pdf-processing"}}, + {"tool_name": "get_weather", "parameters": {"city": "Paris"}}, + {"type": "tool_result", "status": "success", "llmContent": "Weather in Paris: 21C"}, + ] + + assert extract_selected_skills(messages) == [_loaded("pdf-processing", None)] + + +def test_a_malformed_match_does_not_hide_a_second_reading_of_the_same_call(): + """Shapes overlap, so one block can be a broken call in one shape and a whole one in another. + + A harness that wraps a `toolUse` and also tags the block `type: "tool_use"` carries the call + twice. If the wrapper is truncated, stopping at the first recognizer that matched throws away + the flat fields that did survive, and a real skill load reads as no load at all. + """ + messages = [ + { + "toolUse": {"toolUseId": "t1", "name": "skills"}, # no `input`: unusable + "type": "tool_use", + "id": "t1", + "name": "skills", + "input": {"skill_name": "pdf-processing"}, # the same call, intact + }, + { + "role": "user", + "content": [{"type": "tool_result", "tool_use_id": "t1", "content": fx.SKILL_BODY}], + }, + ] + + assert extract_selected_skills(messages) == [_loaded("pdf-processing", fx.SKILL_BODY)] + + +@pytest.mark.parametrize( + "block", + [ + pytest.param({"toolUse": {"toolUseId": "t1", "name": "skills"}}, id="bedrock-wrapper"), + pytest.param({"tool_name": "skills", "parameters": None}, id="gemini-stream"), + pytest.param({"content_type": "tool_use", "name": "skills"}, id="typed-message"), + ], +) +def test_a_sibling_name_and_args_pair_is_not_read_as_the_broken_call(block): + """Recovering a second reading must not become inventing a different call. + + `{"name", "args"}` is the loosest shape recognized, and a block that declares a harness is not + it: these are one malformed call, not a valid bare one. The sibling pair here names a real skill + tool and a skill the agent never asked for, so reading it would not merely lose the broken call + but report the wrong skill as loaded. Reporting no call is the honest answer, so a block that + declares a harness is offered only to the recognizers that read tagged shapes. + """ + messages = [{**block, "name": "skills", "args": {"skill_name": "never-requested"}}] + + assert extract_selected_skills(messages) == [] + + +# ---- The load-event layer ---------------------------------------------------- +# +# `extract_selected_skills` reports one row per skill, which is what the judges want. These +# assert the distinctions that folding necessarily loses, and that they survive one layer down. + + +def test_events_keep_repeated_loads_that_the_summary_folds(): + events = extract_skill_load_events(fx.DUPLICATE_LOAD_MESSAGES) + summary = extract_selected_skills(fx.DUPLICATE_LOAD_MESSAGES) + + assert [e.name for e in events] == ["pdf-processing", "pdf-processing"] + assert len(summary) == 1 + # Distinct calls, so an evaluator counting reloads has something to count. + assert len({e.call_id for e in events}) == 2 + assert [e.position for e in events] == sorted(e.position for e in events) + + +def test_a_call_whose_outcome_never_appears_is_attempted_not_loaded(): + """A trajectory that stops before the result is not a load that succeeded silently. + + The summary can only say the agent asked for the skill, so it reports it as invoked with no + body, which is the same row a successful load with an uncaptured body produces. The event + keeps the two apart. + """ + messages = [ + { + "role": "assistant", + "content": [{"toolUse": {"toolUseId": "no-result", "name": "skills", "input": {"skill_name": "xlsx"}}}], + } + ] + + (event,) = extract_skill_load_events(messages) + assert (event.name, event.status, event.body) == ("xlsx", "attempted", None) + + (loaded,) = extract_skill_load_events(fx.BODY_MISSING_MESSAGES) + assert loaded.status == "loaded" + assert loaded.body is None + + +def test_a_refusal_and_its_retry_are_two_events_but_one_row(): + body = "# PDF\n\n1. Read it." + messages = [ + *_load_attempt("1", "pdf-processing", _REFUSED), + *_load_attempt("2", "pdf-processing", {"content": [{"text": body}]}), + ] + + assert [(e.status, e.error) for e in extract_skill_load_events(messages)] == [ + ("failed", "skill not found"), + ("loaded", None), + ] + assert extract_selected_skills(messages) == [_loaded("pdf-processing", body)] + + +def test_events_from_a_session_carry_position_and_the_call_id(): + spans = [ + ToolExecutionSpan( + span_info=_span_info(), + tool_call=ToolCall(name="calculator", arguments={"expression": "2+2"}, tool_call_id="c-1"), + tool_result=ToolResult(content="4"), + ), + ToolExecutionSpan( + span_info=_span_info(), + tool_call=ToolCall(name="skills", arguments={"skill_name": "pdf-processing"}, tool_call_id="tu-1"), + tool_result=ToolResult(content=fx.SKILL_BODY, tool_call_id="tu-1"), + ), + ] + + (event,) = extract_skill_load_events(_session(spans)) + assert (event.name, event.status, event.call_id) == ("pdf-processing", "loaded", "tu-1") + # Position counts tool spans, so it locates the load among them rather than among skill loads. + assert event.position == 2 + + +def test_a_load_is_attributed_to_the_agent_that_made_it_when_the_trace_says_so(): + """Which sub-agent loaded a skill, for traces whose mapper records it. + + The trace types carry no agent identity, so this reads `metadata`. A trace without one + reports None rather than guessing. + """ + spans = [ + ToolExecutionSpan( + span_info=_span_info(), + metadata={"agent_name": "researcher"}, + tool_call=ToolCall(name="skills", arguments={"skill_name": "pdf-processing"}, tool_call_id="a"), + tool_result=ToolResult(content=fx.SKILL_BODY), + ), + ToolExecutionSpan( + span_info=_span_info(), + tool_call=ToolCall(name="skills", arguments={"skill_name": "spreadsheet-analysis"}, tool_call_id="b"), + tool_result=ToolResult(content="# Spreadsheets\n\n1. Open it."), + ), + ] + + assert [(e.name, e.agent_id) for e in extract_skill_load_events(_session(spans))] == [ + ("pdf-processing", "researcher"), + ("spreadsheet-analysis", None), + ] + + +def test_no_trajectory_yields_no_events(): + assert extract_skill_load_events(None) == [] + assert extract_skill_load_events([]) == [] + + +# ---- Captured harness fixtures ----------------------------------------------- +# +# The shapes above are hand-written from each harness's documented format. These read what the +# harnesses actually emitted, so a wire format that drifts from the hand-written version fails here +# rather than in a user's run. See `fixtures/capture_skill_fixtures.py` for how each was captured +# and which SDK version produced it. + + +def _captured(name: str): + """Load a captured fixture by filename stem.""" + path = Path(__file__).resolve().parent / "fixtures" / f"{name}.json" + return json.loads(path.read_text()) + + +def _system_prompt_text(prompt) -> str: + """The system prompt as text, whether the harness recorded a string or content blocks.""" + if isinstance(prompt, list): + return "\n".join(block.get("text", "") for block in prompt) + return prompt or "" + + +@pytest.mark.parametrize( + "case, expected", + [ + ("loaded", [("pdf-processing", "loaded")]), + # The typo case: the plugin returns a plain string, which `@tool` marks status="success", + # so only the text says the load failed. + ("typo_refusal", [("pdf-procesing", "failed")]), + ("retry_after_refusal", [("pdf-procesing", "failed"), ("pdf-processing", "loaded")]), + ("two_skills", [("pdf-processing", "loaded"), ("spreadsheet-analysis", "loaded")]), + ("repeated_load", [("pdf-processing", "loaded"), ("pdf-processing", "loaded")]), + ], +) +def test_the_real_agent_skills_plugin_is_read_as_captured(case, expected): + run = _captured("strands_agent_skills")[case] + events = extract_skill_load_events(run["messages"]) + assert [(event.name, event.status) for event in events] == expected + + +def test_the_real_agent_skills_refusal_text_is_carried_not_mistaken_for_a_body(): + """jjbuck's case, against the plugin's own output rather than a transcription of it.""" + run = _captured("strands_agent_skills")["typo_refusal"] + + (event,) = extract_skill_load_events(run["messages"]) + assert event.body is None + assert event.error == ("Skill 'pdf-procesing' not found. Available skills: pdf-processing, spreadsheet-analysis") + assert extract_selected_skills(run["messages"]) == [_failed("pdf-procesing", event.error)] + + +def test_the_real_agent_skills_catalog_injection_parses(): + run = _captured("strands_agent_skills")["loaded"] + skills = parse_available_skills(_system_prompt_text(run["system_prompt"])) + assert [skill.name for skill in skills] == ["pdf-processing", "spreadsheet-analysis"] + assert skills[0].description.startswith("Use this skill when the task requires") + + +def test_a_repeated_load_is_two_captured_events_and_one_row(): + run = _captured("strands_agent_skills")["repeated_load"] + assert len(extract_skill_load_events(run["messages"])) == 2 + assert len(extract_selected_skills(run["messages"])) == 1 + + +def test_the_real_claude_code_skill_tool_is_read_as_captured(): + """Claude Code acknowledges the launch and injects the body as a separate user message.""" + messages = _captured("claude_code_skill_tool")["messages"] + + (event,) = extract_skill_load_events(messages) + assert (event.name, event.status) == ("pdf-processing", "loaded") + # The acknowledgement ("Launching skill: pdf-processing") is not the body. + assert event.body is not None + assert "Identify the PDF file path." in event.body + + +def test_the_real_codex_exec_stream_is_read_as_captured(): + """`codex exec --json`: a skill load is a shell read, wrapped in an `item.completed` event.""" + events = _captured("codex_exec_json")["events"] + + (event,) = extract_skill_load_events(events) + assert (event.name, event.status) == ("pdf-processing", "loaded") + assert event.body is not None + assert "Identify the PDF file path." in event.body + + +def test_the_real_codex_session_rollout_is_read_as_captured(): + """The same run recorded as Responses API items, with Codex's output preamble stripped.""" + items = _captured("codex_session_rollout")["items"] + + (event,) = extract_skill_load_events(items) + assert (event.name, event.status) == ("pdf-processing", "loaded") + assert event.body is not None + # The preamble Codex prints ahead of the output is not part of the skill body. + assert event.body.startswith("---\nname: pdf-processing") + assert "Wall time" not in event.body + + +@pytest.mark.parametrize( + "case, expected", + [ + ("loaded", [("pdf-processing", "loaded")]), + ("not_found", [("pdf-procesing", "failed")]), + ], +) +def test_the_real_google_adk_load_skill_is_read_as_captured(case, expected): + contents = _captured("google_adk_load_skill")[case]["contents"] + events = extract_skill_load_events(contents) + assert [(event.name, event.status) for event in events] == expected + + +def test_the_real_google_adk_refusal_carries_its_own_message(): + contents = _captured("google_adk_load_skill")["not_found"]["contents"] + (event,) = extract_skill_load_events(contents) + assert event.body is None + assert event.error == "Skill 'pdf-procesing' not found." + + +@pytest.mark.parametrize( + "fixture, case", + [ + ("strands_agent_skills", "empty_name_refusal"), + ("google_adk_load_skill", "missing_arg"), + ], +) +def test_a_call_with_no_skill_name_yields_no_event(fixture, case): + """Both harnesses refuse these for real, and neither refusal names a skill to report. + + Deliberate: an event would have to invent a skill named "", which would then be scored as a + wrong selection. See `_skill_name_from_args`. + """ + run = _captured(fixture)[case] + messages = run.get("messages") or run["contents"] + assert extract_skill_load_events(messages) == [] + assert extract_selected_skills(messages) == [] diff --git a/tests/strands_evals/mappers/test_openinference_session_mapper.py b/tests/strands_evals/mappers/test_openinference_session_mapper.py index d008bf39..e8597f5e 100644 --- a/tests/strands_evals/mappers/test_openinference_session_mapper.py +++ b/tests/strands_evals/mappers/test_openinference_session_mapper.py @@ -596,6 +596,37 @@ def test_system_prompt_only_returns_no_span(self): # No user content → no InferenceSpan produced assert session.traces == [] + def test_system_prompt_backfilled_to_agent_span_independent_of_span_order(self): + available_block = ( + "pdf-processing" + "Read PDFs." + ) + llm_attrs = { + "openinference.span.kind": "LLM", + "llm.input_messages.0.message.role": "system", + "llm.input_messages.0.message.content": available_block, + "llm.input_messages.1.message.role": "user", + "llm.input_messages.1.message.content": "Read report.pdf", + "llm.output_messages.0.message.role": "assistant", + "llm.output_messages.0.message.content": "Done", + } + llm_span = make_span( + trace_id="prompt-trace", + span_id="llm", + attributes=llm_attrs, + ) + agent_span = make_chain_span( + trace_id="prompt-trace", + span_id="agent", + user_query="Read report.pdf", + agent_response="Done", + ) + + session = self.mapper.map_to_session([agent_span, llm_span], "sess-1") + agent = next(span for span in session.traces[0].spans if isinstance(span, AgentInvocationSpan)) + + assert agent.system_prompt == available_block + # ========================================================================= # Python Repr Parsing Tests diff --git a/tests/strands_evals/mappers/test_strands_in_memory_mapper.py b/tests/strands_evals/mappers/test_strands_in_memory_mapper.py index c44e23c9..d2d4a9d4 100644 --- a/tests/strands_evals/mappers/test_strands_in_memory_mapper.py +++ b/tests/strands_evals/mappers/test_strands_in_memory_mapper.py @@ -4,6 +4,7 @@ from opentelemetry.sdk.trace import ReadableSpan, TracerProvider from opentelemetry.trace import SpanContext, SpanKind, TraceFlags +from strands_evals.extractors import parse_available_skills from strands_evals.mappers import GenAIConventionVersion, StrandsInMemorySessionMapper from strands_evals.types.trace import AgentInvocationSpan, InferenceSpan, ToolExecutionSpan @@ -77,6 +78,121 @@ def test_agent_span(provider): assert agent.available_tools[0].name == "calc" +def test_system_prompt_is_read_from_a_span_attribute_too(provider): + """The other wire shape: `gen_ai.system_instructions` as a span attribute, not an event. + + The SDK's tracer writes it as an attribute when Langfuse or `gen_ai_span_attributes_only` is + active, and unconditionally for bidi sessions. Only the event branch was covered, so the + attribute branch could be deleted with every mapper test still green, and the skill catalog + would silently vanish for those runs. + """ + available_block = ( + "pdf-processing" + "Read PDFs." + ) + chat_span = make_span( + provider, + 0xAAA, + 0xBBB, + 0xAAA1, + "chat", + { + "gen_ai.operation.name": "chat", + "gen_ai.provider.name": "strands-agents", + "gen_ai.system_instructions": json.dumps([{"type": "text", "content": available_block}]), + }, + lambda span: span.add_event("gen_ai.user.message", {"content": '[{"text": "read it"}]'}), + ) + agent_span = make_span( + provider, + 0xAAA, + 0xCCC, + None, + "invoke_agent", + {"gen_ai.operation.name": "invoke_agent", "gen_ai.provider.name": "strands-agents"}, + lambda span: span.add_event("gen_ai.user.message", {"content": '[{"text": "read it"}]'}), + ) + + session = StrandsInMemorySessionMapper().map_to_session([chat_span, agent_span], "sid") + agent = next(span for span in session.traces[0].spans if isinstance(span, AgentInvocationSpan)) + + assert agent.system_prompt == available_block + assert [skill.name for skill in parse_available_skills(session)] == ["pdf-processing"] + + +@pytest.mark.parametrize("latest", [False, True]) +def test_agent_span_receives_system_prompt_from_chat_span(provider, latest): + available_block = ( + "pdf-processing" + "Read PDFs." + ) + convention_attrs = {"gen_ai.provider.name": "strands-agents"} if latest else {"gen_ai.system": "strands-agents"} + + if latest: + + def chat_events(span): + span.add_event( + "gen_ai.client.inference.operation.details", + { + "gen_ai.system_instructions": json.dumps([{"type": "text", "content": available_block}]), + "gen_ai.input.messages": json.dumps( + [{"role": "user", "parts": [{"type": "text", "content": "read it"}]}] + ), + "gen_ai.output.messages": json.dumps( + [{"role": "assistant", "parts": [{"type": "text", "content": "done"}]}] + ), + }, + ) + + def agent_events(span): + span.add_event( + "gen_ai.client.inference.operation.details", + { + "gen_ai.input.messages": json.dumps( + [{"role": "user", "parts": [{"type": "text", "content": "read it"}]}] + ), + "gen_ai.output.messages": json.dumps( + [{"role": "assistant", "parts": [{"type": "text", "content": "done"}]}] + ), + }, + ) + else: + + def chat_events(span): + span.add_event("gen_ai.system.message", {"content": json.dumps([{"text": available_block}])}) + span.add_event("gen_ai.user.message", {"content": '[{"text": "read it"}]'}) + span.add_event("gen_ai.choice", {"message": '[{"text": "done"}]'}) + + def agent_events(span): + span.add_event("gen_ai.user.message", {"content": '[{"text": "read it"}]'}) + span.add_event("gen_ai.choice", {"message": "done"}) + + chat_span = make_span( + provider, + 0xAAA, + 0xBBB, + 0xAAA1, + "chat", + {"gen_ai.operation.name": "chat", **convention_attrs}, + chat_events, + ) + agent_span = make_span( + provider, + 0xAAA, + 0xCCC, + None, + "invoke_agent", + {"gen_ai.operation.name": "invoke_agent", **convention_attrs}, + agent_events, + ) + + session = StrandsInMemorySessionMapper().map_to_session([chat_span, agent_span], "sid") + agent = next(span for span in session.traces[0].spans if isinstance(span, AgentInvocationSpan)) + + assert agent.system_prompt == available_block + assert [skill.name for skill in parse_available_skills(session)] == ["pdf-processing"] + + def test_tool_span(provider): span = make_span( provider, diff --git a/tests/strands_evals/test_experiment.py b/tests/strands_evals/test_experiment.py index b8be325e..10a9c370 100644 --- a/tests/strands_evals/test_experiment.py +++ b/tests/strands_evals/test_experiment.py @@ -17,14 +17,19 @@ Evaluator, InteractionsEvaluator, OutputEvaluator, + SkillInstructionFollowingEvaluator, + SkillInvoked, + SkillSelectionAccuracyEvaluator, StartsWith, ToolCalled, TrajectoryEvaluator, ) from strands_evals.evaluators.evaluator import DEFAULT_BEDROCK_MODEL_ID -from strands_evals.experiment import is_throttling_error +from strands_evals.evaluators.skill_selection_accuracy_evaluator import SkillSelectionScore +from strands_evals.experiment import _get_label_from_score, is_throttling_error from strands_evals.providers.trace_provider import TraceProvider from strands_evals.types import EvaluationData, EvaluationOutput +from strands_evals.types.evaluation import NOT_APPLICABLE from strands_evals.types.trace import ( AgentInvocationSpan, Session, @@ -1691,6 +1696,50 @@ def test_deterministic_evaluator_from_dict_round_trip(): assert original_report.test_passes == restored_report.test_passes +def test_skill_evaluator_from_dict_round_trip(): + """The skill evaluators must be loadable from an experiment file, like every other built-in. + + `from_dict` resolves `evaluator_type` against a fixed registry, so an evaluator missing from it + raises "Cannot find ..." and the experiment file cannot be run at all. + """ + experiment = Experiment( + cases=[Case(name="pdf", input="Extract text from report.pdf")], + evaluators=[ + SkillSelectionAccuracyEvaluator(), + SkillInstructionFollowingEvaluator(), + SkillInvoked(skill_name="pdf-processing"), + ], + ) + + restored = Experiment.from_dict(experiment.to_dict()) + + assert [e.get_type_name() for e in restored.evaluators] == [ + "SkillSelectionAccuracyEvaluator", + "SkillInstructionFollowingEvaluator", + "SkillInvoked", + ] + assert restored.evaluators[2].skill_name == "pdf-processing" + + +def test_all_not_applicable_case_is_not_labeled_with_a_verdict(): + """A case with nothing to judge must not report the score mapping's worst label. + + Its aggregate score is the 0.0 placeholder, which reverse-maps to the mapping's zero-scored + label ("No" for selection). That would publish a failing verdict, on the span and in the + CloudWatch record, for a run the judge never rated, while `test_pass` on the same rows is + True. Passing the rows in lets the label say "not applicable" instead. + """ + evaluator = SkillSelectionAccuracyEvaluator() + not_applicable = [EvaluationOutput(score=0.0, test_pass=True, reason="nothing to judge", label=NOT_APPLICABLE)] + judged_no = [EvaluationOutput(score=0.0, test_pass=False, reason="wrong pick", label="No")] + + assert _get_label_from_score(evaluator, 0.0, not_applicable) == NOT_APPLICABLE + # A real zero-scored verdict still maps to the mapping's label, and so does every existing + # caller that passes no rows at all. + assert _get_label_from_score(evaluator, 0.0, judged_no) == str(SkillSelectionScore.NO) + assert _get_label_from_score(evaluator, 0.0) == str(SkillSelectionScore.NO) + + def test_deterministic_evaluator_error_isolation(): """Test that a failing deterministic evaluator doesn't crash other evaluators.""" cases = [ diff --git a/tests/strands_evals/types/test_evaluation_report.py b/tests/strands_evals/types/test_evaluation_report.py index fac44c39..83900739 100644 --- a/tests/strands_evals/types/test_evaluation_report.py +++ b/tests/strands_evals/types/test_evaluation_report.py @@ -4,12 +4,127 @@ import pytest +from strands_evals.types.evaluation import NOT_APPLICABLE, EvaluationOutput from strands_evals.types.evaluation_report import EvaluationReport +class TestNotApplicableLabel: + """The shared not-applicable marker every reader of a score has to agree on.""" + + def test_not_applicable_is_recognized_by_the_property(self): + assert EvaluationOutput(score=0.0, test_pass=True, label=NOT_APPLICABLE).not_applicable is True + + def test_a_judged_row_is_applicable(self): + assert EvaluationOutput(score=1.0, test_pass=True, label="Yes").not_applicable is False + + def test_an_unlabeled_row_is_applicable(self): + """Most evaluators leave `label` unset, and their scores are still verdicts.""" + assert EvaluationOutput(score=0.5, test_pass=True).not_applicable is False + + def test_is_applicable_needs_only_one_judged_row(self): + mixed = [ + EvaluationOutput(score=1.0, test_pass=True, label="Yes"), + EvaluationOutput(score=0.0, test_pass=True, label=NOT_APPLICABLE), + ] + assert EvaluationReport.is_applicable(mixed) is True + assert EvaluationReport.is_applicable(mixed[1:]) is False + + def test_a_case_with_no_rows_at_all_is_left_in(self): + """An evaluator that produced nothing is not the same as one that judged nothing. + + "Every row is not-applicable" is vacuously true of no rows, so the reading that drops an + all-N/A case would drop this one too. It must not: an evaluator that returned no rows + failed to judge rather than declining to, `_default_aggregator` scores that `test_pass` + False, and dropping the case would take that failure out of every average. + """ + assert EvaluationReport.is_applicable([]) is True + + def test_a_case_with_no_rows_still_counts_toward_the_mean(self): + """The consequence of the line above, at the level the number is actually read. + + A case that produced nothing scores 0.0 and that 0.0 is real, so a corpus of one such case + and one perfect case averages 0.5. Were the empty case dropped the corpus would report + 1.0, a clean sweep, with the failure invisible. + """ + judged = [EvaluationOutput(score=1.0, test_pass=True, label="Yes")] + + assert EvaluationReport.calculate_overall_score([0.0, 1.0], [[], judged]) == 0.5 + # Contrast: a case that declined to judge is dropped, so the mean is the judged case alone. + declined = [EvaluationOutput(score=0.0, test_pass=True, label=NOT_APPLICABLE)] + assert EvaluationReport.calculate_overall_score([0.0, 1.0], [declined, judged]) == 1.0 + + def test_a_case_that_failed_to_judge_is_not_droppable(self): + """The shape both evaluators emit for a missing trajectory, which is not a clean pass. + + `test_pass` is what separates declining to judge from failing to. Dropping a + not-applicable row without reading it would take a real failure out of the mean and report + a run that never produced a verdict as a clean sweep. `actual_trajectory` defaults to None, + so this is reachable whenever one case fails to capture a trajectory. + """ + judged = [EvaluationOutput(score=1.0, test_pass=True, label="Yes")] + failed = [EvaluationOutput(score=0.0, test_pass=False, label=NOT_APPLICABLE, reason="no trajectory provided")] + + assert EvaluationReport.is_applicable(failed) is True + assert EvaluationReport.calculate_overall_score([0.0, 1.0], [failed, judged]) == 0.5 + + class TestEvaluationReportFlatten: """Tests for the flatten() classmethod.""" + def test_overall_score_excludes_not_applicable_rows(self): + applicable = [EvaluationOutput(score=1.0, test_pass=True, label="skill-a")] + not_applicable = [ + EvaluationOutput( + score=0.0, + test_pass=True, + label=NOT_APPLICABLE, + reason="no skill invoked", + ) + ] + + assert ( + EvaluationReport.calculate_overall_score( + [1.0, 0.0], + [applicable, not_applicable], + ) + == 1.0 + ) + + def test_overall_score_matches_plain_mean_when_no_row_is_not_applicable(self): + """Backward compatibility: with no not-applicable rows this is the plain mean. + + Only the skill evaluators emit label=NOT_APPLICABLE, so this pins that the + aggregation is unchanged for every pre-existing evaluator. + """ + scores = [1.0, 0.0, 0.5] + detailed_results = [ + [EvaluationOutput(score=1.0, test_pass=True, label="appropriate")], + [EvaluationOutput(score=0.0, test_pass=False, label="inappropriate")], + [EvaluationOutput(score=0.5, test_pass=True)], + ] + + assert EvaluationReport.calculate_overall_score(scores, detailed_results) == sum(scores) / len(scores) + + def test_overall_score_counts_rows_with_any_applicable_output(self): + """A row is dropped only if every output in it is not-applicable.""" + mixed = [ + EvaluationOutput(score=1.0, test_pass=True, label="skill-a"), + EvaluationOutput(score=0.0, test_pass=True, label=NOT_APPLICABLE), + ] + + assert EvaluationReport.calculate_overall_score([1.0], [mixed]) == 1.0 + + def test_overall_score_is_zero_when_every_row_is_not_applicable(self): + not_applicable = [EvaluationOutput(score=0.0, test_pass=True, label=NOT_APPLICABLE)] + + assert ( + EvaluationReport.calculate_overall_score( + [0.0, 0.0], + [not_applicable, not_applicable], + ) + == 0.0 + ) + def test_flatten_empty_list(self): flattened = EvaluationReport.flatten([]) assert flattened.overall_score == 0.0 @@ -73,6 +188,35 @@ def test_flatten_multiple_reports(self): "Equals", ] + def test_flatten_excludes_not_applicable_rows_from_overall_score(self): + applicable = EvaluationReport( + overall_score=1.0, + scores=[1.0], + cases=[{"name": "used"}], + test_passes=[True], + detailed_results=[[EvaluationOutput(score=1.0, test_pass=True, label="skill-a")]], + ) + not_applicable = EvaluationReport( + overall_score=0.0, + scores=[0.0], + cases=[{"name": "unused"}], + test_passes=[True], + detailed_results=[ + [ + EvaluationOutput( + score=0.0, + test_pass=True, + label=NOT_APPLICABLE, + ) + ] + ], + ) + + flattened = EvaluationReport.flatten([applicable, not_applicable]) + + assert flattened.scores == [1.0, 0.0] + assert flattened.overall_score == 1.0 + def test_flatten_preserves_case_data(self): report = EvaluationReport( overall_score=0.5, @@ -135,8 +279,6 @@ def test_flatten_handles_mismatched_lengths(self): assert flattened.reasons[1] == "" def test_flatten_preserves_detailed_results(self): - from strands_evals.types.evaluation import EvaluationOutput - detailed = [EvaluationOutput(score=0.5, test_pass=True, reason="detail")] report = EvaluationReport( overall_score=0.5,