feat: add evaluator metadata method and types

[max-rattray-aws]

Summary

Adds a metadata() instance method to the Evaluator base class, allowing evaluators to declare what they check, how they work, and their tier in the evaluation hierarchy.

This enables downstream systems to:

• Aggregate results by tier (guardrail failures override pass/fail, diagnostics are informational)
• Render informative reports showing what each evaluator measures
• Route models based on evaluation method category

Usage

from strands_evals.evaluators import Contains, FaithfulnessEvaluator
from strands_evals.types import validate_metadata

# Deterministic evaluator
evaluator = Contains(value="hello", case_sensitive=False)
meta = evaluator.metadata()
# {'checks': 'Whether actual_output contains a required substring',
#  'method': {'category': 'deterministic_string', 'summary': 'Case-insensitive substring search on actual_output.'},
#  'threshold': 'substring present',
#  'tier': 'quality'}

# LLM-judge evaluator
faith = FaithfulnessEvaluator()
meta = faith.metadata()
# {'checks': 'Whether the agent\'s response is grounded in the conversation history',
#  'method': {'category': 'llm_judge_output', ...},
#  'threshold': 'score >= 0.50',
#  'tier': 'guardrail'}

# Validate metadata at configuration time
validate_metadata(meta, faith.get_name())  # Raises ValueError if invalid

What's new

• src/strands_evals/types/evaluator_metadata.py - New module with EvaluatorMetadata, MethodInfo, MethodCategory, Tier types and validate_metadata() function
• metadata() method on base Evaluator class (returns None by default)
• Metadata implementations on 11 built-in evaluators:
  • Deterministic: Contains, Equals, StartsWith, ToolCalled, StateEquals
  • LLM judges: FaithfulnessEvaluator, HarmfulnessEvaluator, CorrectnessEvaluator, ToolSelectionAccuracyEvaluator, ToolParameterAccuracyEvaluator, GoalSuccessRateEvaluator

What's tested

• 43 tests covering:
  • Type construction and field access
  • validate_metadata() with all valid/invalid input combinations
  • Base class returns None, subclass override works
  • Each built-in evaluator's metadata passes validation
  • Instance-dependent metadata (case_sensitive, tool_name, state_name)
  • Import accessibility from expected module paths

Related to #350

[github-actions]

Issue: VALID_METHOD_CATEGORIES (and VALID_TIERS below) duplicate the exact string values already declared in the MethodCategory and Tier Literals. Any future addition has to be made in two places, and they can silently drift.

Suggestion: Derive the runtime sets from the Literals so there's a single source of truth:

from typing_extensions import get_args
VALID_METHOD_CATEGORIES: set[str] = set(get_args(MethodCategory))
VALID_TIERS: set[str] = set(get_args(Tier))

The test_valid_method_categories_match_literal / test_valid_tiers_match_literal tests catch drift but only after it happens — single-sourcing prevents it entirely.

[github-actions]

Resolved in 74ae442 — VALID_METHOD_CATEGORIES and VALID_TIERS are now derived via set(get_args(MethodCategory)) / set(get_args(Tier)), single-sourcing them from the Literals. 👍

[github-actions]

Issue: The per-evaluator metadata tests assert individual fields one at a time (meta["checks"], meta["method"]["category"], meta["tier"], ...). Since each evaluator's metadata is fully deterministic (or fully determined by constructor args), per-field assertions silently miss regressions in fields that aren't checked — e.g. a changed method.summary, a wrong threshold string, or an unexpectedly added key.

Suggestion: Assert the whole dict in a single equality check, e.g.:

assert meta == {
    "checks": "Whether actual_output exactly equals an expected value",
    "method": {"category": "deterministic_string", "summary": "Exact equality comparison between actual_output and expected value."},
    "threshold": "exact match",
    "tier": "quality",
}

This applies to Equals, Contains, StartsWith, ToolCalled, StateEquals, and the LLM-judge tests. For the case-sensitivity variants, you can still build the expected dict from case_sensitive and compare the full shape. Keep validate_metadata(meta, ...) as an additional check.

[github-actions]

Resolved in 74ae442 — the per-evaluator tests now assert the full metadata dict in a single equality check (including the case_sensitive variants) while keeping validate_metadata(meta, ...) as an additional check, exactly as suggested. Thanks!

[github-actions]

Issue: validate_metadata is typed to accept EvaluatorMetadata, but the documented producer — Evaluator.metadata() — returns EvaluatorMetadata | None. If a caller passes the result of an evaluator that hasn't declared metadata (returns None), this raises an opaque AttributeError on metadata.keys() rather than a clear message.

Suggestion: Either guard for None early with an explicit error (e.g. "Evaluator '<name>' did not declare metadata"), or document that callers must null-check before calling. A small guard keeps the failure mode consistent with the other clear ValueErrors in this function.

[github-actions]

Thanks for addressing this — the current code now types the parameter as EvaluatorMetadata | None and returns early when None, with the docstring updated to match. Resolves the concern. 👍

[github-actions]

Issue: metadata() is implemented on 11 built-in evaluators, but the package ships ~25 (e.g. Coherence, Conciseness, Helpfulness, InstructionFollowing, ResponseRelevance, Refusal, Stereotyping, the multimodal_* family, and Trajectory). Those all inherit the base None. Downstream systems that "aggregate results by tier" and "render reports" — the stated motivation — will therefore get inconsistent coverage.

Suggestion: This is a reasonable incremental first step and doesn't need to block the PR. Please open a tracking issue for adding metadata to the remaining evaluators so the intent (and which evaluators are guardrail vs quality vs diagnostic) is captured. Non-blocking.

[github-actions]

Assessment: Comment (approve after minor changes)

Clean, well-scoped addition. The type definitions are clear, validate_metadata covers the important invalid cases, and the threshold strings I spot-checked (FaithfulnessEvaluator score >= 0.50, GoalSuccessRateEvaluator score == 1.0) accurately match the actual test_pass logic. No blocking issues.

Review themes

• Maintainability (DRY): Enumerated values are declared twice — as Literals and as parallel runtime sets. Derive the sets from the Literals via get_args() to single-source them.
• Testing: Metadata tests assert field-by-field; since the dicts are deterministic, full-dict equality assertions would catch regressions and unexpected fields that per-field checks miss.
• Robustness: validate_metadata doesn't handle the None that metadata() can return, producing an opaque error.
• Coverage (non-blocking): Metadata is on 11 of ~25 evaluators; suggest a tracking issue for the rest.

Nice work keeping the metadata declarations consistent with real evaluator behavior — that's the part most likely to rot, and it's accurate here.

[github-actions]

Issue (scope / acceptance criteria): validate_metadata is exported but never called anywhere in the framework, and Evaluator.metadata() is never consumed by any framework code — grep for validate_metadata and .metadata() shows no callers outside tests. Issue #350 explicitly lists as part of the proposed solution:

Validation happens at experiment construction time. If metadata() returns a non-None value, the framework checks required keys are present and values are in the allowed sets.

...and frames the tier field as driving aggregation semantics (guardrail failures override the verdict). As it stands the PR delivers the declarations but none of the wiring — construction-time validation, tier-based aggregation, and report rendering are all absent.

Suggestion: This is fine as an incremental first PR, but please make the scope explicit so reviewers align: either (a) wire validate_metadata into Experiment construction now (a small addition — iterate over evaluators and validate each non-None metadata), or (b) note in the PR description that validation/aggregation wiring is a deliberate follow-up and open the tracking issue. Right now the validator is dead code from the framework's perspective, which is easy to miss and regress. Non-blocking.

[github-actions]

Assessment: Approve ✅

Re-reviewed after 74ae442 ("wire validate_metadata into Experiment.run_evaluations"). All substantive feedback from the prior round is addressed:

Resolution status

• DRY ✅ — VALID_METHOD_CATEGORIES / VALID_TIERS now derived from the Literals via get_args(); no more parallel maintenance.
• Testing ✅ — per-evaluator tests now assert the full metadata dict in a single equality check (incl. case_sensitive variants), with validate_metadata kept as an additional guard.
• Robustness ✅ — validate_metadata accepts EvaluatorMetadata | None and no-ops on None.
• Wiring / acceptance criteria ([FEATURE] Evaluator metadata taxonomy (tier, method, description) #350) ✅ — Experiment.run_evaluations now calls _validate_evaluator_metadata(), validating every declaring evaluator at run time. This closes the "dead validator" gap and fulfills the construction-time validation from the issue.

Verified locally: test_evaluator_metadata.py 47 passed, test_experiment.py 111 passed.

One non-blocking follow-up still stands: metadata is on 11 of ~25 evaluators, and tier-based aggregation / report rendering remain future work — worth a tracking issue. Nice, responsive iteration on this. 🚀