Skip to content

fix(consolidation): expose failed LLM batches - #4164

Closed
koriyoshi2041 wants to merge 1 commit into
vectorize-io:mainfrom
koriyoshi2041:rios/observe-consolidation-batch-failures
Closed

fix(consolidation): expose failed LLM batches#4164
koriyoshi2041 wants to merge 1 commit into
vectorize-io:mainfrom
koriyoshi2041:rios/observe-consolidation-batch-failures

Conversation

@koriyoshi2041

Copy link
Copy Markdown
Contributor

Closes #4151.

What changed

Add hindsight.consolidation.batch_failures, a counter for LLM batches that exhaust consolidation's retry policy. It uses a bounded reason label:

  • response_validation for malformed JSON, schema validation, and output-length failures
  • provider for other exhausted provider failures

The existing failed_consolidation gauge remains unchanged: it counts source memories permanently marked failed after adaptive bisection reaches a singleton. The new counter makes recoverable batch-level failures visible without changing that state-based contract.

Each failed batch increments once after its retry ladder, rather than once per attempt. Successful batches do not increment it.

Verification

  • tests/test_metrics.py: 34 passed
  • TestBuildResponseModel: 10 passed
  • focused Ruff check and format check passed
  • git diff --check passed

The repository hook reached the control-plane lint step but could not load @eslint/js because Node dependencies are not installed in this worktree. The changed Python files and tests passed their focused gates.

@koriyoshi2041

Copy link
Copy Markdown
Contributor Author

Two CI failures look unrelated to this metrics-only diff:

  • verify-generated-files stopped in setup-uv while fetching the latest uv version (fetch failed), before generation ran.
  • free-threaded 3.14 completed 6,987 tests and failed only the existing OpenCode session-header fixture after LLMInterface.call() became a named result (result.ok / payload-vs-.content); the changed consolidation tests passed. PR fix(reflect): enforce max_tokens after forced synthesis #4159 hit the same free-threaded job failure on its independent branch.

I cannot rerun upstream jobs from the fork. Could a maintainer rerun the failed jobs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consolidation responses failing schema validation are not counted in failed_consolidation

2 participants