Repository navigation
Description
🔴 Required Information
Describe the Bug:
hallucinations_v1 returns NOT_EVALUATED (score None) for a run when the judge can't label one of the responses. That came in with 59b0dd6, so an unlabelled response can't quietly drop out of the average.
With --num_runs > 1 that gets lost again. _get_metric_run_result still passes the run's per-invocation scores to mean_over_invocations, which skips the None and averages the rest. So across runs the metric is scored only on the responses the judge managed to label, and it can pass.
I noticed this while reading the aggregator, then checked it with the real evaluator and a stubbed judge (script below).
This is only on main for now (59b0dd6 isn't in 2.14.1). Judge failures do happen: in a 3-run adk eval of my own agent on main, one run logged Sentence validator returned 0 blocks for 1 sentences; this response will not be scored. That case has one turn per run, so it came out fine. You need a multi-turn case with one unscored turn to hit this.
Steps to Reproduce:
- main at c74df83,
uv sync --all-extras. - Run the script below. No network needed: the judge is stubbed to label the first response and fail on the second.
import asyncio
from unittest import mock
from google.adk.evaluation._eval_case_result_multi_run_aggregator import aggregate_eval_case_results
from google.adk.evaluation.eval_case import Invocation
from google.adk.evaluation.eval_metrics import EvalMetric
from google.adk.evaluation.eval_metrics import EvalMetricResult
from google.adk.evaluation.eval_metrics import EvalMetricResultPerInvocation
from google.adk.evaluation.eval_metrics import HallucinationsCriterion
from google.adk.evaluation.eval_metrics import JudgeModelOptions
from google.adk.evaluation.eval_result import EvalCaseResult
from google.genai import types
with mock.patch("google.adk.evaluation.hallucinations_v1.LLMRegistry"):
from google.adk.evaluation.hallucinations_v1 import HallucinationsV1Evaluator
metric = EvalMetric(
metric_name="hallucinations_v1",
threshold=0.5,
criterion=HallucinationsCriterion(
threshold=0.5,
judge_model_options=JudgeModelOptions(judge_model="gemini-2.5-flash"),
),
)
evaluator = HallucinationsV1Evaluator(metric)
async def judge(nl_response, context):
if nl_response.startswith("Paris"):
return 1.0, "[]"
return None, "unparsable judge output"
evaluator._evaluate_nl_response = judge
def invocation(text):
return Invocation(
user_content=types.Content(parts=[types.Part(text="q")]),
final_response=types.Content(parts=[types.Part(text=text)]),
)
async def run(index):
invocations = [
invocation("Paris is the capital of France."),
invocation("The Eiffel Tower is 900 m tall."),
]
result = await evaluator.evaluate_invocations(invocations)
# Built the way LocalEvalService builds a run's result.
return EvalCaseResult(
eval_set_id="set",
eval_id="case",
final_eval_status=result.overall_eval_status,
overall_eval_metric_results=[
EvalMetricResult(
score=result.overall_score,
eval_status=result.overall_eval_status,
**metric.model_dump(),
)
],
eval_metric_result_per_invocation=[
EvalMetricResultPerInvocation(
actual_invocation=actual,
eval_metric_results=[
EvalMetricResult(
score=r.score,
eval_status=r.eval_status,
**metric.model_dump(),
)
],
)
for actual, r in zip(invocations, result.per_invocation_results)
],
session_id=f"s{index}",
repetition_index=index,
)
async def main():
runs = [await run(i) for i in range(3)]
for n in (1, 3):
case = aggregate_eval_case_results(runs[:n])[0]
m = case.overall_eval_metric_results[0]
print(f"{n} run(s): score={m.score} {m.eval_status.name}, case {case.final_eval_status.name}")
asyncio.run(main())Expected Behavior:
Three runs that are each unscored should be unscored overall, same as one run:
1 run(s): score=None NOT_EVALUATED, case NOT_EVALUATED
3 run(s): score=None NOT_EVALUATED, case NOT_EVALUATED
Observed Behavior:
1 run(s): score=None NOT_EVALUATED, case NOT_EVALUATED
3 run(s): score=1.0 PASSED, case PASSED
The judge never checked the second response (which is false), and the case passes.
Environment Details:
- ADK Library Version: main @ c74df83
- Desktop OS: Windows 10
- Python Version: 3.11.9
Model Information:
- Are you using LiteLLM: No
- Which model is being used: N/A (stubbed judge)
🟡 Optional Information
Regression:
Not in a release yet, see above. #8122 (retrying unscored judge calls) makes this rarer, but once the retries run out the aggregator still does the same thing.
Proposed fix:
In _get_metric_run_result, skip a run's invocation scores when its overall result is NOT_EVALUATED with no score. mean_of_runs and min_over_runs already leave that run out (its overall score is None), so all three score methods would agree. Scored runs still count, so one unscored run among scored ones is just left out. If you'd rather have any unscored run make the whole metric unscored, that's an easy change too.
I have this with a regression test in test_eval_case_result_multi_run_aggregator.py, and the eval unit tests pass. Happy to open a PR if that sounds right.
Activity
somuai commentedon Nov 18, 2026
I investigated this failure mode and opened PR #8238 with a minimal fix and unit regression test.
Root Cause Confirmation
In
_get_metric_run_result(src/google/adk/evaluation/_eval_case_result_multi_run_aggregator.py),invocation_scoreswas unconditionally collected fromeval_metric_result_per_invocationeven when the run's overall metric result wasNOT_EVALUATEDwithscore=None. Becausemean_over_invocationsfilters outNoneturn scores, it ended up computing the average over only the turns that were successfully labeled, falsely producing aPASSEDverdict with a non-null score across multiple runs.Fix & Verification
In
_get_metric_run_result:score=None, NOT_EVALUATED, caseNOT_EVALUATEDscore=1.0, PASSED, casePASSED(falsely passed)score=None, NOT_EVALUATED, caseNOT_EVALUATEDscore=None, NOT_EVALUATED, caseNOT_EVALUATED(correctly preserved)score=0.85, PASSED, casePASSED(unscored run excluded from the mean, matching the behavior ofmean_of_runsandmin_over_runs).All 88 unit tests in
test_eval_case_result_multi_run_aggregator.pyand all 51 tests intest_multi_run_aggregation_methods.pypass cleanly.surajksharma07 commentedon Nov 19, 2026
@EdwardGlockner Reproduced on main with your script: 1 run stays NOT_EVALUATED, but 3 runs come back 1.0 PASSED.
As a workaround, clearing invocation_scores in _get_metric_run_result when the run’s overall result is NOT_EVALUATED with score None fixed it for me. 3 unscored runs now give None / NOT_EVALUATED, and in a mixed set the unscored run is just left out.
Could you try that on your real multi-turn eval with --num_runs 3 and check it holds up, mixed runs included? Please test it thoroughly before raising anything.
#8238 already makes this same change, so it might be easier to review that one than open a second PR. Let us know how it goes.
EdwardGlockner commentedon Nov 19, 2026
Thanks. I ran my repro against #8238: with num_runs=3 it now gives score=None /
NOT_EVALUATED (main gives 1.0 / PASSED), and num_runs=1 is unchanged. My script uses
the real evaluator with a stubbed judge, not a live eval. Agreed, let's go with #8238.
I added an end-to-end test for it: somuai#1
surajksharma07 commentedon Nov 20, 2026
Thanks @EdwardGlockner for checking #8238 and for the end-to-end test. One update: 59b0dd6 is in 2.15.0 now, so this isn’t main-only anymore. 2.15.0 users with multi-turn hallucinations_v1 cases and --num_runs > 1 can hit it.
Merged #8238 (7496eaa) onto today’s main (c81876a). It’s 26 commits behind but merges cleanly. Your repro gives 1.0 PASSED on main and None / NOT_EVALUATED with the PR. In the mixed case the unscored run is left out (0.85 rather than the 0.9 main gives). The new aggregator test fails on main and passes with the PR, and the eval unit tests pass (1086). Your e2e test passes too, along with the rest of test_local_eval_service.py (56 passed). Only mean_over_invocations reads invocation_scores, so mean_of_runs and min_over_runs aren’t touched.
@i-yliu could you review #8238 and approve the 2 pending workflows? Two things to check: whether dropping an unscored run in a mixed set (rather than failing the metric) is the behaviour you want, and that 7496eaa keeps Edward as co-author when it goes through Copybara.
somuai commentedon Nov 20, 2026
@EdwardGlockner Thanks! I have fast-forward merged your commit from somuai#1 directly onto #8238, preserving your authorship intact. The PR branch has been updated with the end-to-end test.
surajksharma07 commentedon Nov 20, 2026
Thanks @somuai. Re-checked the branch and the head is still 7496eaa, the same commit I tested earlier on top of c81876a, so those results still hold. Nothing in src/google/adk/evaluation has changed on main since 2.15.0, so there’s nothing new to rebase over.
One note on credit: Copybara usually lands a PR as a single commit, so Edward’s separate commit may not survive as is. A Co-authored-by: trailer for Edward on the final commit would keep it.
Nothing more is needed from either of you for now. It’s waiting on maintainer review and the 2 pending workflow approvals.