Skip to content

adk eval --num_runs can pass hallucinations_v1 even when every run left it unscored #8230

Description

@EdwardGlockner

🔴 Required Information

Describe the Bug:
hallucinations_v1 returns NOT_EVALUATED (score None) for a run when the judge can't label one of the responses. That came in with 59b0dd6, so an unlabelled response can't quietly drop out of the average.

With --num_runs > 1 that gets lost again. _get_metric_run_result still passes the run's per-invocation scores to mean_over_invocations, which skips the None and averages the rest. So across runs the metric is scored only on the responses the judge managed to label, and it can pass.

I noticed this while reading the aggregator, then checked it with the real evaluator and a stubbed judge (script below).

This is only on main for now (59b0dd6 isn't in 2.14.1). Judge failures do happen: in a 3-run adk eval of my own agent on main, one run logged Sentence validator returned 0 blocks for 1 sentences; this response will not be scored. That case has one turn per run, so it came out fine. You need a multi-turn case with one unscored turn to hit this.

Steps to Reproduce:

  1. main at c74df83, uv sync --all-extras.
  2. Run the script below. No network needed: the judge is stubbed to label the first response and fail on the second.
import asyncio
from unittest import mock

from google.adk.evaluation._eval_case_result_multi_run_aggregator import aggregate_eval_case_results
from google.adk.evaluation.eval_case import Invocation
from google.adk.evaluation.eval_metrics import EvalMetric
from google.adk.evaluation.eval_metrics import EvalMetricResult
from google.adk.evaluation.eval_metrics import EvalMetricResultPerInvocation
from google.adk.evaluation.eval_metrics import HallucinationsCriterion
from google.adk.evaluation.eval_metrics import JudgeModelOptions
from google.adk.evaluation.eval_result import EvalCaseResult
from google.genai import types

with mock.patch("google.adk.evaluation.hallucinations_v1.LLMRegistry"):
  from google.adk.evaluation.hallucinations_v1 import HallucinationsV1Evaluator

  metric = EvalMetric(
      metric_name="hallucinations_v1",
      threshold=0.5,
      criterion=HallucinationsCriterion(
          threshold=0.5,
          judge_model_options=JudgeModelOptions(judge_model="gemini-2.5-flash"),
      ),
  )
  evaluator = HallucinationsV1Evaluator(metric)


async def judge(nl_response, context):
  if nl_response.startswith("Paris"):
    return 1.0, "[]"
  return None, "unparsable judge output"


evaluator._evaluate_nl_response = judge


def invocation(text):
  return Invocation(
      user_content=types.Content(parts=[types.Part(text="q")]),
      final_response=types.Content(parts=[types.Part(text=text)]),
  )


async def run(index):
  invocations = [
      invocation("Paris is the capital of France."),
      invocation("The Eiffel Tower is 900 m tall."),
  ]
  result = await evaluator.evaluate_invocations(invocations)
  # Built the way LocalEvalService builds a run's result.
  return EvalCaseResult(
      eval_set_id="set",
      eval_id="case",
      final_eval_status=result.overall_eval_status,
      overall_eval_metric_results=[
          EvalMetricResult(
              score=result.overall_score,
              eval_status=result.overall_eval_status,
              **metric.model_dump(),
          )
      ],
      eval_metric_result_per_invocation=[
          EvalMetricResultPerInvocation(
              actual_invocation=actual,
              eval_metric_results=[
                  EvalMetricResult(
                      score=r.score,
                      eval_status=r.eval_status,
                      **metric.model_dump(),
                  )
              ],
          )
          for actual, r in zip(invocations, result.per_invocation_results)
      ],
      session_id=f"s{index}",
      repetition_index=index,
  )


async def main():
  runs = [await run(i) for i in range(3)]
  for n in (1, 3):
    case = aggregate_eval_case_results(runs[:n])[0]
    m = case.overall_eval_metric_results[0]
    print(f"{n} run(s): score={m.score} {m.eval_status.name}, case {case.final_eval_status.name}")


asyncio.run(main())

Expected Behavior:
Three runs that are each unscored should be unscored overall, same as one run:

1 run(s): score=None NOT_EVALUATED, case NOT_EVALUATED
3 run(s): score=None NOT_EVALUATED, case NOT_EVALUATED

Observed Behavior:

1 run(s): score=None NOT_EVALUATED, case NOT_EVALUATED
3 run(s): score=1.0 PASSED, case PASSED

The judge never checked the second response (which is false), and the case passes.

Environment Details:

  • ADK Library Version: main @ c74df83
  • Desktop OS: Windows 10
  • Python Version: 3.11.9

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: N/A (stubbed judge)

🟡 Optional Information

Regression:
Not in a release yet, see above. #8122 (retrying unscored judge calls) makes this rarer, but once the retries run out the aggregator still does the same thing.

Proposed fix:
In _get_metric_run_result, skip a run's invocation scores when its overall result is NOT_EVALUATED with no score. mean_of_runs and min_over_runs already leave that run out (its overall score is None), so all three score methods would agree. Scored runs still count, so one unscored run among scored ones is just left out. If you'd rather have any unscored run make the whole metric unscored, that's an easy change too.

I have this with a regression test in test_eval_case_result_multi_run_aggregator.py, and the eval unit tests pass. Happy to open a PR if that sounds right.

Activity

  1. somuai commented on Nov 18, 2026

    @somuai

    I investigated this failure mode and opened PR #8238 with a minimal fix and unit regression test.

    Root Cause Confirmation

    In _get_metric_run_result (src/google/adk/evaluation/_eval_case_result_multi_run_aggregator.py), invocation_scores was unconditionally collected from eval_metric_result_per_invocation even when the run's overall metric result was NOT_EVALUATED with score=None. Because mean_over_invocations filters out None turn scores, it ended up computing the average over only the turns that were successfully labeled, falsely producing a PASSED verdict with a non-null score across multiple runs.

    Fix & Verification

    In _get_metric_run_result:

    if (
        overall is not None
        and overall.eval_status == EvalStatus.NOT_EVALUATED
        and overall.score is None
    ):
      invocation_scores = []
    • Before fix:
      • 1 run: score=None, NOT_EVALUATED, case NOT_EVALUATED
      • 3 runs: score=1.0, PASSED, case PASSED (falsely passed)
    • After fix:
      • 1 run: score=None, NOT_EVALUATED, case NOT_EVALUATED
      • 3 runs: score=None, NOT_EVALUATED, case NOT_EVALUATED (correctly preserved)
      • Mixed set (1 unscored + 2 scored runs at 0.8 and 0.9): score=0.85, PASSED, case PASSED (unscored run excluded from the mean, matching the behavior of mean_of_runs and min_over_runs).

    All 88 unit tests in test_eval_case_result_multi_run_aggregator.py and all 51 tests in test_multi_run_aggregation_methods.py pass cleanly.

  2. surajksharma07 commented on Nov 19, 2026

    @surajksharma07
    Collaborator

    @EdwardGlockner Reproduced on main with your script: 1 run stays NOT_EVALUATED, but 3 runs come back 1.0 PASSED.

    As a workaround, clearing invocation_scores in _get_metric_run_result when the run’s overall result is NOT_EVALUATED with score None fixed it for me. 3 unscored runs now give None / NOT_EVALUATED, and in a mixed set the unscored run is just left out.

    Could you try that on your real multi-turn eval with --num_runs 3 and check it holds up, mixed runs included? Please test it thoroughly before raising anything.

    #8238 already makes this same change, so it might be easier to review that one than open a second PR. Let us know how it goes.

  3. EdwardGlockner commented on Nov 19, 2026

    @EdwardGlockner
    Author

    Thanks. I ran my repro against #8238: with num_runs=3 it now gives score=None /
    NOT_EVALUATED (main gives 1.0 / PASSED), and num_runs=1 is unchanged. My script uses
    the real evaluator with a stubbed judge, not a live eval. Agreed, let's go with #8238.
    I added an end-to-end test for it: somuai#1

  4. surajksharma07 commented on Nov 20, 2026

    @surajksharma07
    Collaborator

    Thanks @EdwardGlockner for checking #8238 and for the end-to-end test. One update: 59b0dd6 is in 2.15.0 now, so this isn’t main-only anymore. 2.15.0 users with multi-turn hallucinations_v1 cases and --num_runs > 1 can hit it.

    Merged #8238 (7496eaa) onto today’s main (c81876a). It’s 26 commits behind but merges cleanly. Your repro gives 1.0 PASSED on main and None / NOT_EVALUATED with the PR. In the mixed case the unscored run is left out (0.85 rather than the 0.9 main gives). The new aggregator test fails on main and passes with the PR, and the eval unit tests pass (1086). Your e2e test passes too, along with the rest of test_local_eval_service.py (56 passed). Only mean_over_invocations reads invocation_scores, so mean_of_runs and min_over_runs aren’t touched.

    @i-yliu could you review #8238 and approve the 2 pending workflows? Two things to check: whether dropping an unscored run in a mixed set (rather than failing the metric) is the behaviour you want, and that 7496eaa keeps Edward as co-author when it goes through Copybara.

  5. somuai commented on Nov 20, 2026

    @somuai

    @EdwardGlockner Thanks! I have fast-forward merged your commit from somuai#1 directly onto #8238, preserving your authorship intact. The PR branch has been updated with the end-to-end test.

  6. surajksharma07 commented on Nov 20, 2026

    @surajksharma07
    Collaborator

    Thanks @somuai. Re-checked the branch and the head is still 7496eaa, the same commit I tested earlier on top of c81876a, so those results still hold. Nothing in src/google/adk/evaluation has changed on main since 2.15.0, so there’s nothing new to rebase over.

    One note on credit: Copybara usually lands a PR as a single commit, so Edward’s separate commit may not survive as is. A Co-authored-by: trailer for Edward on the final commit would keep it.

    Nothing more is needed from either of you for now. It’s waiting on maintainer review and the 2 pending workflow approvals.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions