[syzsunshine219]

Restores the change proposed in #2489 for review against main after its former base branch, dev-v2.0.36, was deleted. The source commits and authors are preserved.

Source head: 41269f335d7c08a13cc1242eff662ff688203434

This PR is open for review. Review and CI against the current base are pending; validation claims in the original description below refer to the earlier snapshot. The original discussion and evidence remain in #2489.

---

Original change description

Summary

clean_json_response used a global response.replace("```json", "").replace("```", ""), which removed every triple-backtick sequence — including ones inside JSON string values. Any LLM response whose payload contained a code snippet came out silently corrupted before json.loads.

Root cause

src/memos/mem_os/utils/format_utils.py:1416 (pre-fix):

return response.replace("```json", "").replace("```", "").strip()

Reproducer from the issue:

clean_json_response("```json\n{\"text\": \"```py\\ncode```\"}\n```")
# before:  "{\"text\": \"py\\ncode\"}"      ← fences stripped from the value
# after:   "{\"text\": \"```py\\ncode```\"}"

Both api/handlers/chat_handler.py and api/handlers/suggestion_handler.py pipe responses through this function, so the bug surface was the entire chat-metadata and suggestion pipeline.

Fix

Replace the global string replace with two anchored regex substitutions that only strip the outer fences:

• Opening: re.compile(r"^```[^\n`]*\n?") — covers the optional info string (json or otherwise) and trailing newline
• Closing: re.compile(r"\n?```\s*$") — at the very end only

Interior ``` sequences inside a JSON string value are preserved verbatim. As a side effect the function now correctly handles responses that have only an opening or only a closing fence (the original implementation would leave those stranded mid-string).

Tests

Added four cases to tests/mem_os/test_format_utils.py:

1. test_clean_json_response_preserves_internal_fenced_code — the issue reproducer (red against unpatched main)
2. test_clean_json_response_preserves_unclosed_fence_in_value — a value containing a single unclosed fence
3. test_clean_json_response_handles_only_opening_fence — best-effort handling when the closing fence is missing
4. test_clean_json_response_handles_only_closing_fence — best-effort handling when the opening fence is missing

Existing tests (single backticks in JSON content, plain JSON, complex JSON, etc.) all continue to pass — test_clean_json_response_preserves_internal_backticks was a good seed for the new tests.

Verification

PYTHONPATH=src pytest tests/mem_os/test_format_utils.py -v
============================== 12 passed in 3.09s ==============================

Ruff: ruff format --check and ruff check both clean on touched files (the 2 pre-existing findings were auto-fixed by ruff check --fix during this PR — both were E501 line-too-long in clean_json_response itself, not introduced by this change).

Out of scope

• No new dependencies
• No public API / OpenAPI contract changes
• No DB schema, migration, or graph model changes

Closes #2420

[Memtensor-AI]

🤖 Open Code Review

Target: PR #2590
Task: e45aa70eef2d46db
Base: main
Head: fix/issue-2420-clean-json-fence
Head SHA: 41269f335d7c08a13cc1242eff662ff688203434

✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

Generated by cloud-assistant via Open Code Review.

[Memtensor-AI]

✅ Automated Test Results: PASSED

Required tests passed (14/14 executed, 0 skipped). Duration: 17s [advisory, non-gating] AI-generated tests on branch test/auto-gen-e45aa70eef2d46db-20261110001310: 54/54 passed — these do NOT affect the PR verdict; review the branch manually.

Branch: fix/issue-2420-clean-json-fence

Tested source: 41269f335d7c08a13cc1242eff662ff688203434
Tested base: 3a16e5fe3048a209f722806f4123fa2e533588fd
Tested commit: 6536557b4f9a55d80c8078e0f61b7f9504580d74
These results apply only to this immutable source/base snapshot.