feat: add skill-level evaluators for skill-equipped agents

AGENTS.md

@@ -39,6 +39,7 @@ strands-evals depends on **`strands-agents`** (the Python SDK) for all LLM inter
   - `strands.tools.decorator` (`DecoratedFunctionTool`, `FunctionToolMetadata`) and the `@tool` decorator from `strands`
   - `strands.types.content.Message`
   - `strands.types.exceptions` (`EventLoopException`, `ModelThrottledException`)
+  - `strands.Skill` (public, exported in `strands.__all__` from `strands.vended_plugins.skills.skill`), used by the skill extractors to read `SKILL.md` frontmatter
 - **Not used as SDK imports (do not add to the list without verifying a real import):**
   - `strands.types.traces` — this repo defines its own `strands_evals.types.trace` for session/span modeling; the SDK's trace types are not imported.
   - `strands.telemetry` — the evals repo has its own `strands_evals.telemetry` module. The only reference to `strands.telemetry` is the literal string `"strands.telemetry.tracer"` used as an OpenTelemetry scope name in `mappers/constants.py`.

README.md

@@ -38,6 +38,7 @@ Strands Evaluation is a powerful framework for evaluating AI agents and LLM appl
 
 - **Multiple Evaluation Types**: Output evaluation, trajectory analysis, tool usage assessment, and interaction evaluation
 - **Multimodal Evaluation**: MLLM-as-a-Judge evaluators for image-to-text tasks with built-in rubrics
+- **Skill Evaluation**: Assess which skills a skill-equipped agent selected and whether it followed their instructions
 - **Dynamic Simulators**: Multi-turn conversation simulation with realistic user behavior, goal-oriented interactions, and LLM-powered tool simulation with shared state
 - **LLM-as-a-Judge**: Built-in evaluators using language models for sophisticated assessment with structured scoring
 - **Trace-based Evaluation**: Analyze agent behavior through OpenTelemetry execution traces
@@ -517,6 +518,35 @@ tool_parameter_evaluator = ToolParameterAccuracyEvaluator(
 )
 ```
 
+### Skill Selection and Instruction Following
+
+A skill is an instruction file (usually `SKILL.md`) that the harness offers to the agent at
+runtime; the agent decides which, if any, to load. Evaluate both halves of that behavior,
+which skill the agent picked and whether it then followed the skill's steps:
+
+```python
+from strands_evals.evaluators import (
+    SkillInstructionFollowingEvaluator,
+    SkillInvoked,
+    SkillSelectionAccuracyEvaluator,
+)
+
+# Was each invoked skill an appropriate pick? Binary, one result per invoked skill.
+# A run that invoked nothing has no selection to judge and yields a not-applicable row.
+selection_evaluator = SkillSelectionAccuracyEvaluator()
+
+# Did the agent follow the invoked skill's steps? Five-level rating, one result per
+# invoked skill, grounded in per-step covered/partial/skipped evidence.
+following_evaluator = SkillInstructionFollowingEvaluator()
+
+# Deterministic presence check, no model.
+invoked_check = SkillInvoked(skill_name="pdf-processing")
+```
+
+Both judges read the trajectory only, so any harness that emits one works: pass a `Session`
+or a raw message list as `actual_trajectory`. The skill signals are recovered by
+`parse_available_skills` and `extract_selected_skills` from `strands_evals.extractors`.
+
 ### Multimodal Evaluation (MLLM-as-a-Judge)
 
 Evaluate multimodal agent responses involving images and text using MLLM-as-a-Judge:

SKILL.md

@@ -131,11 +131,12 @@ Pick the evaluator by scope:
 | Scope | Evaluators |
 | --- | --- |
 | Tool-level | `ToolSelectionAccuracyEvaluator`, `ToolParameterAccuracyEvaluator` |
+| Skill-level (per invoked skill) | `SkillSelectionAccuracyEvaluator`, `SkillInstructionFollowingEvaluator`, `SkillInvoked` (deterministic) |
 | Trace-level (last turn) | `CorrectnessEvaluator`, `HelpfulnessEvaluator`, `FaithfulnessEvaluator`, `CoherenceEvaluator`, `ConcisenessEvaluator`, `ResponseRelevanceEvaluator`, `HarmfulnessEvaluator`, `RefusalEvaluator`, `StereotypingEvaluator`, `InstructionFollowingEvaluator` |
 | Session-level (full conversation) | `GoalSuccessRateEvaluator` |
 | Multi-agent interactions and handoffs | `InteractionsEvaluator` (output-based) |
 
-Helpfulness uses a seven-level scale, 0.0 Not helpful to 1.0 Above and beyond. Correctness uses a three-level rubric in basic mode, or binary CORRECT/INCORRECT in reference mode when `expected_assertion` is set on the case. Conciseness uses three levels. Coherence uses five levels. Harmfulness, Refusal, Stereotyping, InstructionFollowing are binary.
+Helpfulness uses a seven-level scale, 0.0 Not helpful to 1.0 Above and beyond. Correctness uses a three-level rubric in basic mode, or binary CORRECT/INCORRECT in reference mode when `expected_assertion` is set on the case. Conciseness uses three levels. Coherence uses five levels. Harmfulness, Refusal, Stereotyping, InstructionFollowing are binary. SkillInstructionFollowing uses a five-level scale, 0.0 Not Followed to 1.0 Fully Followed, and returns one result per invoked skill. SkillSelectionAccuracy is binary and also returns one result per invoked skill.
 
 For traces from external systems pick the matching mapper:
 - `CloudWatchSessionMapper` paired with `CloudWatchProvider` and `CloudWatchLogsParser`

pyproject.toml

@@ -32,6 +32,7 @@ dependencies = [
     "opentelemetry-instrumentation-threading>=0.51b0,<1.00b0",
     "boto3>=1.26.0",
     "tenacity>=8.0.0,<10.0.0",
+    "pyyaml>=6.0.0,<7.0.0",
 ]
 
 [tool.hatch.build.targets.wheel]

src/strands_evals/cli/_entrypoint.py

@@ -29,6 +29,8 @@
     InstructionFollowingEvaluator,
     RefusalEvaluator,
     ResponseRelevanceEvaluator,
+    SkillInstructionFollowingEvaluator,
+    SkillSelectionAccuracyEvaluator,
     StereotypingEvaluator,
     ToolParameterAccuracyEvaluator,
     ToolSelectionAccuracyEvaluator,
@@ -53,6 +55,10 @@
     "instruction-following": InstructionFollowingEvaluator,
     "refusal": RefusalEvaluator,
     "response-relevance": ResponseRelevanceEvaluator,
+    # `skill-invoked` is absent for the same reason as `tool-called`: `SkillInvoked`
+    # requires a `skill_name`, so it belongs in an experiment file.
+    "skill-instruction-following": SkillInstructionFollowingEvaluator,
+    "skill-selection-accuracy": SkillSelectionAccuracyEvaluator,
     "stereotyping": StereotypingEvaluator,
     "tool-parameter-accuracy": ToolParameterAccuracyEvaluator,
     "tool-selection-accuracy": ToolSelectionAccuracyEvaluator,

src/strands_evals/cli/commands/run.py

@@ -222,13 +222,21 @@ def _print_summary(report: EvaluationReport) -> None:
     `run_evaluations_async` returns a single flattened report whose `cases`
     rows are tagged with an `evaluator` key; regroup by that tag so the
     summary still reads `<evaluator>: P/T passed (avg)` per evaluator.
+
+    Each average is computed the same way as `report.overall_score`, dropping cases that had
+    nothing to judge. Averaging their placeholder 0.0 instead would print a lower per-evaluator
+    number than the overall score computed from the very same rows.
     """
     by_eval: dict[str, list[int]] = {}
     by_eval_scores: dict[str, list[float]] = {}
     for i, case in enumerate(report.cases):
         name = case.get("evaluator", "unknown")
         by_eval.setdefault(name, []).append(int(report.test_passes[i]))
-        by_eval_scores.setdefault(name, []).append(report.scores[i])
+        outputs = report.detailed_results[i] if i < len(report.detailed_results) else []
+        if EvaluationReport.is_applicable(outputs):
+            by_eval_scores.setdefault(name, []).append(report.scores[i])
+        else:
+            by_eval_scores.setdefault(name, [])
 
     parts: list[str] = ["strands-evals run"]
     for name, passes in by_eval.items():
@@ -271,7 +279,10 @@ def _display_expanded(
         details: dict[str, Any] = {"name": case.get("name", f"Test {i + 1}")}
         if "evaluator" in case:
             details["evaluator"] = case["evaluator"]
-        details["score"] = f"{report.scores[i]:.2f}"
+        outputs = report.detailed_results[i] if i < len(report.detailed_results) else []
+        # A case with nothing to judge is excluded from every average, so printing its
+        # placeholder 0.00 in the score column would read as the worst possible verdict.
+        details["score"] = f"{report.scores[i]:.2f}" if EvaluationReport.is_applicable(outputs) else "n/a"
         details["test_pass"] = report.test_passes[i] if i < len(report.test_passes) else False
         details["reason"] = report.reasons[i] if i < len(report.reasons) else ""
         details["input"] = EvaluationReport.format_input_for_display(case.get("input"))

src/strands_evals/evaluators/__init__.py

@@ -1,7 +1,7 @@
 from .coherence_evaluator import CoherenceEvaluator
 from .conciseness_evaluator import ConcisenessEvaluator
 from .correctness_evaluator import CorrectnessEvaluator
-from .deterministic import Contains, Equals, StartsWith, StateEquals, ToolCalled
+from .deterministic import Contains, Equals, SkillInvoked, StartsWith, StateEquals, ToolCalled
 from .evaluator import Evaluator
 from .faithfulness_evaluator import FaithfulnessEvaluator
 from .goal_success_rate_evaluator import GoalSuccessRateEvaluator
@@ -17,12 +17,17 @@
 from .output_evaluator import OutputEvaluator
 from .refusal_evaluator import RefusalEvaluator
 from .response_relevance_evaluator import ResponseRelevanceEvaluator
+from .skill_instruction_following_evaluator import SkillInstructionFollowingEvaluator
+from .skill_selection_accuracy_evaluator import SkillSelectionAccuracyEvaluator
 from .stereotyping_evaluator import StereotypingEvaluator
 from .tool_parameter_accuracy_evaluator import ToolParameterAccuracyEvaluator
 from .tool_selection_accuracy_evaluator import ToolSelectionAccuracyEvaluator
 from .trajectory_evaluator import TrajectoryEvaluator
 
 __all__ = [
+    "SkillSelectionAccuracyEvaluator",
+    "SkillInstructionFollowingEvaluator",
+    "SkillInvoked",
     "Evaluator",
     "OutputEvaluator",
     "MultimodalOutputEvaluator",

src/strands_evals/evaluators/deterministic/__init__.py

@@ -1,8 +1,10 @@
 from .environment_state import StateEquals
 from .output import Contains, Equals, StartsWith
+from .skill_invoked import SkillInvoked
 from .trajectory import ToolCalled
 
 __all__ = [
+    "SkillInvoked",
     "Contains",
     "Equals",
     "StartsWith",

src/strands_evals/evaluators/deterministic/skill_invoked.py

@@ -0,0 +1,50 @@
+from ...extractors.skills import extract_skill_load_events
+from ...types.evaluation import EvaluationData, EvaluationOutput, InputT, OutputT
+from ..evaluator import Evaluator
+
+
+class SkillInvoked(Evaluator[InputT, OutputT]):
+    """Checks if a specific skill was invoked in the trajectory."""
+
+    def __init__(self, skill_name: str, name: str | None = None):
+        super().__init__(name=name)
+        self.skill_name = skill_name
+
+    def evaluate(self, evaluation_case: EvaluationData[InputT, OutputT]) -> list[EvaluationOutput]:
+        trajectory = evaluation_case.actual_trajectory
+        if trajectory is None:
+            return [EvaluationOutput(score=0.0, test_pass=False, reason="no trajectory provided")]
+
+        # Read the individual attempts rather than the per-skill summary, so this check applies its
+        # own definition of "invoked": a refused load does not count, because the agent never
+        # received the skill and an assertion that it was used is false. The summary folds a
+        # refusal and a later success into one loaded row, which is right for judging the choice
+        # and wrong for asserting the skill was in play.
+        attempts = [e for e in extract_skill_load_events(trajectory) if e.name == self.skill_name]
+        found = any(e.status == "loaded" for e in attempts)
+        refusal = next((e for e in attempts if e.status == "failed"), None)
+        if found:
+            reason = f"skill '{self.skill_name}' was invoked"
+        elif refusal is not None:
+            # Report what the harness said: a check that fails on a misspelled skill name and one
+            # that fails because nothing was mounted call for different fixes.
+            detail = f": {refusal.error}" if refusal.error else ""
+            attempted = f" ({len(attempts)} attempts)" if len(attempts) > 1 else ""
+            reason = f"skill '{self.skill_name}' was requested but the load failed{attempted}{detail}"
+        elif attempts:
+            # Every attempt was made and none has a recorded outcome, so the trajectory does not
+            # say whether the skill was received. Reported as not invoked, since the check asserts
+            # use and use was not observed, but named apart from never asking.
+            reason = f"skill '{self.skill_name}' was requested but the trajectory records no outcome"
+        else:
+            reason = f"skill '{self.skill_name}' was not invoked"
+        return [
+            EvaluationOutput(
+                score=1.0 if found else 0.0,
+                test_pass=found,
+                reason=reason,
+            )
+        ]
+
+    async def evaluate_async(self, evaluation_case: EvaluationData[InputT, OutputT]) -> list[EvaluationOutput]:
+        return self.evaluate(evaluation_case)

src/strands_evals/evaluators/evaluator.py

@@ -88,6 +88,26 @@ def _default_aggregator(outputs: list[EvaluationOutput]) -> tuple[float, bool, s
         combined_reason = " | ".join(o.reason for o in outputs if o.reason)
         return avg_score, all_pass, combined_reason
 
+    @staticmethod
+    def _aggregate_dropping_na(outputs: list[EvaluationOutput]) -> tuple[float, bool, str]:
+        """Average only the rows that carry a verdict.
+
+        For evaluators that emit one row per decision and can find some of those decisions
+        unjudgeable, the not-applicable rows score 0.0 as a placeholder. Averaging that in would
+        report a case with one perfectly judged decision and one unjudgeable one as half right.
+        Set `self.aggregator` to this in `__init__` to opt in.
+        """
+        scored = [o for o in outputs if not o.not_applicable]
+        if not scored:
+            reason = " | ".join(o.reason for o in outputs if o.reason) or "not applicable"
+            # Carry the rows' own verdicts: "nothing to judge" passes, but absent data fails.
+            all_pass = all(o.test_pass for o in outputs) if outputs else True
+            return (0.0, all_pass, reason)
+        avg = sum(o.score for o in scored) / len(scored)
+        all_pass = all(o.test_pass for o in scored)
+        reason = " | ".join(o.reason for o in scored if o.reason)
+        return avg, all_pass, reason
+
     def evaluate(self, evaluation_case: EvaluationData[InputT, OutputT]) -> list[EvaluationOutput]:
         """
         Evaluate the performance of the task on the given test cases.

src/strands_evals/evaluators/prompt_templates/skill_instruction_following/__init__.py

@@ -0,0 +1,11 @@
+from . import skill_instruction_following_v0
+
+VERSIONS = {
+    "v0": skill_instruction_following_v0,
+}
+
+DEFAULT_VERSION = "v0"
+
+
+def get_template(version: str = DEFAULT_VERSION):
+    return VERSIONS[version]

src/strands_evals/evaluators/prompt_templates/skill_instruction_following/skill_instruction_following_v0.py

@@ -0,0 +1,39 @@
+SYSTEM_PROMPT = """You are an objective judge evaluating whether an AI agent followed the \
+instructions of a skill it loaded.
+
+A skill is an instruction file (SKILL.md) with prescribed steps. You are given:
+- the SKILL.md body (the steps the agent was supposed to follow),
+- the agent's run (its actions, tool calls, and outputs).
+
+## Evaluation Task
+Identify the prescribed steps in the SKILL.md body, then, for EACH step, judge from the agent's
+run whether the step was:
+- "covered": the run clearly shows the agent carried out the step,
+- "partial": the run shows incomplete or ambiguous adherence,
+- "skipped": the run shows no evidence the step was carried out.
+
+After the per-step judgments, give an overall five-point rating of how fully the agent
+followed the skill's steps:
+- "Fully Followed": essentially every step carried out; no meaningful gaps.
+- "Mostly Followed": the great majority of steps carried out; only minor gaps.
+- "Partially Followed": a mix of carried-out and skipped steps.
+- "Minimally Followed": most steps skipped; only a few carried out.
+- "Not Followed": the skill's steps were essentially ignored.
+
+The overall rating must be consistent with your per-step statuses: it should track the share
+of steps that were covered (a run with nearly all steps covered is "Fully Followed"; one with
+nearly all skipped is "Not Followed").
+
+## Guidelines
+- Judge only against the steps this skill prescribes, not generic task quality.
+- Ground each per-step judgment in specific evidence from the run.
+- A plan, unexecuted code snippet, or claim that a step will be done is not evidence
+  that an executable step was carried out. Require an action or result in the trajectory.
+- If the skill has no clearly enumerable steps, treat its core instructions as the steps.
+- If the skill body prescribes nothing at all (it is reference material rather than instructions),
+  return an empty step list rather than inventing steps.
+
+## Output Format
+Provide, for each step, its status (covered / partial / skipped) with a short evidence note;
+brief overall reasoning; and the overall five-point rating.
+"""

src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/__init__.py

@@ -0,0 +1,11 @@
+from . import skill_selection_accuracy_v0
+
+VERSIONS = {
+    "v0": skill_selection_accuracy_v0,
+}
+
+DEFAULT_VERSION = "v0"
+
+
+def get_template(version: str = DEFAULT_VERSION):
+    return VERSIONS[version]

src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/skill_selection_accuracy_v0.py

@@ -0,0 +1,28 @@
+SYSTEM_PROMPT = """You are an objective judge evaluating whether an AI agent made an \
+appropriate skill-selection decision for a task.
+
+A skill is a reusable instruction file the agent may load to help with a task. At runtime the
+agent is shown a list of available skills (each with a name and description) and decides which,
+if any, to load. You are given:
+- the task the agent was asked to do,
+- the list of available skills (name + description),
+- one skill the agent invoked, which is the decision under evaluation,
+- the agent's run.
+
+## Evaluation Question
+Judge only the one skill named as the decision under evaluation.
+- "Yes" if that skill's description fits the task (it was a reasonable skill to load).
+- "No" if that skill does not fit the task (an inappropriate pick).
+- On a task that needs several skills, invoking any one skill that genuinely fits is
+  appropriate on its own; judge this skill on its own merits, not on whether the agent also
+  loaded the other skills it needed.
+
+## Guidelines
+- Judge the selection decision, not how well the agent then executed the skill.
+- Base the decision on the skill descriptions and the task, not on the outcome.
+
+## Output Format
+First give brief step-by-step reasoning, then a single verdict:
+- "Yes" if the decision was appropriate,
+- "No" if it was not.
+"""

src/strands_evals/evaluators/prompt_templates/trajectory_prompt_template.py

@@ -0,0 +1,41 @@
+"""Rendering a trajectory into judge-prompt text."""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+from pydantic import BaseModel
+
+from ...types.trace import Session
+
+# Cap on serialized trajectory size in judge prompts, ~150k tokens at 4 chars/token.
+MAX_TRAJECTORY_CHARS = 600_000
+
+
+def serialize_trajectory(trajectory: Session | list[Any] | None, max_chars: int = MAX_TRAJECTORY_CHARS) -> str:
+    """Serialize a trajectory into stable JSON, for use in judge prompts.
+
+    The middle of a long run is dropped: a real trajectory can reach millions of tokens (one read
+    of a large artifact is enough), which overflows any judge context window. The head and tail are
+    kept because skills are loaded early and the outcome lands late.
+
+    Args:
+        trajectory: A `Session` or a raw message list, or None.
+        max_chars: Size ceiling for the returned text. Pass 0 to keep the whole trajectory.
+
+    Returns:
+        str: The serialized trajectory, "(no trajectory)" when None, with an inline note naming the
+        character count where the middle was omitted.
+    """
+    if trajectory is None:
+        return "(no trajectory)"
+    if isinstance(trajectory, Session):
+        value: Any = trajectory.model_dump(mode="json")
+    else:
+        value = [item.model_dump(mode="json") if isinstance(item, BaseModel) else item for item in trajectory]
+    text = json.dumps(value, indent=2, default=str)
+    if max_chars <= 0 or len(text) <= max_chars:
+        return text
+    keep = max_chars // 2
+    return f"{text[:keep]}\n\n... [{len(text) - 2 * keep} characters omitted] ...\n\n{text[-keep:]}"