Skip to content

fix: CorrectnessEvaluator honors expected_output for reference-mode grading - #356

Open
AmirF194 wants to merge 1 commit into
strands-agents:mainfrom
AmirF194:fix/334-correctness-evaluator-expected-output
Open

fix: CorrectnessEvaluator honors expected_output for reference-mode grading#356
AmirF194 wants to merge 1 commit into
strands-agents:mainfrom
AmirF194:fix/334-correctness-evaluator-expected-output

Conversation

@AmirF194

@AmirF194 AmirF194 commented Aug 8, 2026

Copy link
Copy Markdown

Description

CorrectnessEvaluator only enters reference-mode grading when expected_assertion is set on the evaluation case. expected_output is a separate, documented field on EvaluationData ("the expected response given the input"), but nothing in correctness_evaluator.py ever reads it: _has_reference() checks expected_assertion alone, and _format_reference_prompt() interpolates only expected_assertion into the judge prompt. A case that sets expected_output without also setting expected_assertion silently falls through to _evaluate_basic()'s trajectory-only grading, which can score a response CORRECT even when it contradicts the expected output, with no warning to the caller.

Fix: _has_reference() now treats either field as a reference, and _format_reference_prompt() prefers expected_assertion when both are set (unchanged behavior for existing callers) and falls back to expected_output when only that is present.

Related Issues

Fixes #334

Documentation PR

None; docstring updated in place, no separate docs page describes this behavior.

Type of Change

Bug fix

Testing

  • tests/strands_evals/evaluators/test_correctness_evaluator.py::test_has_reference_true_with_expected_output_only, test_format_reference_prompt_uses_expected_output_when_no_assertion, and test_evaluate_with_expected_output_only_uses_reference_mode fail on main and pass with this change.

  • Full suite passes: hatch test tests --cover (1760 passed).

  • hatch run test-lint (ruff check + mypy) is clean.

  • Not checked: behavior against a live model call in reference mode; the new tests mock the judge model the same way the existing reference-mode tests do.

  • I ran hatch run prepare (ran its constituent checks separately: hatch run test-lint and hatch test tests --cover, not the formatter step, to avoid reformatting unrelated pre-existing files)

Checklist

  • I have read the CONTRIBUTING document
  • I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
  • My change is focused and reasonably small; I have split unrelated work into separate PRs
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…rading

A case that sets only expected_output (documented as "the expected response
given the input") silently fell back to _evaluate_basic()'s trajectory
grading, since _has_reference() gated reference mode on expected_assertion
alone. That path can score a response CORRECT even when it contradicts the
expected output, with no warning.

_has_reference() now treats either field as a reference; _format_reference_prompt()
prefers expected_assertion when both are set (unchanged behavior) and falls
back to expected_output otherwise.

Fixes strands-agents#334
@AmirF194
AmirF194 requested a review from a team as a code owner August 8, 2026 12:32
@AmirF194
AmirF194 requested a review from mkmeral August 8, 2026 12:32
@github-actions github-actions Bot added bug Something isn't working area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics labels Aug 8, 2026
@AmirF194

Copy link
Copy Markdown
Author

Checking in after a week, no rush. Two of the checks are held pending an internal approval gate on your side (integration tests for external PRs), everything else that has run is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] CorrectnessEvaluator ignores Case.expected_output and silently falls back to trajectory grading

1 participant