Repository navigation
LongMemEval judge drops a correct answer unless the model returns raw JSON #2525
Description
Pre-submission checklist | 提交前检查
- I have searched existing issues and this hasn't been mentioned before
- I have read the project documentation and confirmed this issue doesn't already exist
- This issue is specific to MemOS and not a general software issue
Bug Description | 问题描述
evaluation/scripts/longmemeval/lme_eval.py lme_grader parses the judge reply with json.loads(message_content)["label"]. The judge prompt in evaluation/scripts/utils/prompts.py (LME_JUDGE_MODEL_TEMPLATE) asks the model to write a one-sentence explanation and then a JSON label. A reply that follows that prompt is not raw JSON, so json.loads raises JSONDecodeError.
process_qa has no handler for that exception. main catches it, prints [ERROR] Processing user failed: ..., and does not record a judgment. evaluate_accuracy then reports No responses were evaluated. LLM-as-a-Judge score: N/A (0/0). lme_metric.py prints that missing result as score 0.0000 over 0 questions.
LoCoMo's grader in evaluation/scripts/locomo/locomo_eval.py already extracts a {"label": ...} object from surrounding text with extract_label_json. The same judge text that LoCoMo accepts is rejected by LongMemEval.
On a one-question LongMemEval smoke run, the generated answer was Business Administration, matching the gold answer, and the judge step failed with Expecting value: line 1 column 1 (char 0). An empty model body and a prose-plus-JSON body both raise that same error, so a transient empty completion and a normal explanation-plus-JSON completion are not distinguishable in the log. Either one drops the sample. The failure is in the parser, not only in the gateway.
How to Reproduce | 如何重现
- Use
evaluation/scripts/longmemeval/lme_eval.pylme_graderon commita7367d07. - Stub the chat client so
message.contentisThe generated answer matches the gold answer.\n{"label": "CORRECT"}. - Call
lme_graderwith questionWhat degree did I graduate with?, goldBusiness Administration, and the same text as the generated answer. - Observed:
JSONDecodeError: Expecting value: line 1 column 1 (char 0). The sample is not stored, and the summary isN/A (0/0). - Pass that same string to LoCoMo
extract_label_json. Observed: it returns{"label": "CORRECT"}. - Repeat step 2 with
message.content == "". Observed: the sameJSONDecodeError, and again no recorded judgment.
Environment | 环境信息
- Python version: 3.12.15
- Operating System: macOS (darwin)
- MemOS version: MemoryOS 2.0.33 (commit
a7367d07)
Willingness to Implement | 实现意愿
- I'm willing to implement this myself
Activity
Memtensor-AI commentedon Oct 21, 2026
🤖 AutoDev has picked up this issue and started working on it.
Task ID:
0ee609f367fb34ccWorking branch:
bugfix/autodev-2525-20261021095904941Target branch:
latest dev* branchWorkflow: opsp (analysis → coding → testing → PR)
I will post the PR link here once done. If I need more information, I will ask in the comments.
fix(evaluation): parse prose+JSON judge replies in LongMemEval grader (…
Memtensor-AI commentedon Oct 21, 2026
✅ AutoDev task
0ee609f367fb34cccompleted.Summary: Fixed issue #2525: LongMemEval's
lme_graderwas passing the full judge chat completion intojson.loads(...)["label"], but the judge prompt (LME_JUDGE_MODEL_TEMPLATE) asks the model for one sentence of reasoning followed by a{"label": "..."}snippet. Any prompt-compliant reply therefore raisedJSONDecodeError: Expecting value: line 1 column 1 (char 0); the exception bubbled tomain, which dropped the sample and degraded the summary toN/A (0/0). LoCoMo had already solved the same prompt shape withextract_label_json; LongMemEval had not.The fix introduces a shared
evaluation/scripts/utils/judge.pywithextract_label_json(same regex semantics as LoCoMo's) andparse_judge_label, which returns the label string on success and raisesValueErrorwith a 200-byte quoted preview of the raw content on empty / no-label replies — making a transient empty completion visibly different from a prose-only reply in the[ERROR] Processing user failed:log line.lme_gradernow callsparse_judge_label;locomo_graderwas rewired to the same helper (behaviour preserved by the existing try/except) so the two codepaths cannot drift again. The judge prompt itself was not changed.Tests: 12 unit tests on
parse_judge_label(raw JSON, prose+JSON reproduction from the issue, single-quoted value, empty, None, no-label, long content preview truncation) plus 4 integration tests drivinglme_graderthrough a mocked OpenAI client — eval-only deps (bert_score,sentence_transformers,rouge_score) are stubbed insys.modulesso the suite runs under the standardtestextras withoutpoetry install --with eval.pytest tests/evaluation/longmemeval: 17/17 passed. Combined with unrelated tests (tests/test_hello_world.py,tests/test_log.py): 33/33 passed.ruff checkandruff format --checkboth clean on all touched files. Nosrc/memos/ordocs/openapi.jsonedits.Base branch:
mainBranch:
bugfix/autodev-2525-20261021095904941Commit:
a0acd63PR: #2528
Assigned to: @CarltonXiang
Reviewers: @endxxxx
fix(evaluation): address OCR review feedback on judge parser