Skip to content

[codex] record usage costs and refresh local model pricing and limits - #3226

Merged
EKKOLearnAI merged 8 commits into
mainfrom
codex/usage-cost-accounting
Sep 30, 2026
Merged

EKKOLearnAI merged 8 commits into
mainfrom
codex/usage-cost-accounting

Conversation

@EKKOLearnAI

@EKKOLearnAI EKKOLearnAI commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Usage totals previously showed $0 when no cost was recorded. Persist nullable per-call/run USD costs and show reported, estimated, partial, or unknown coverage in total and daily usage.
  • Use a shared local models.dev catalog for automatic cost estimates and context/output limits. Download and validate the catalog in the background on every server startup, replace it atomically, and retain the last good file when offline. Keep the cache under the configured Studio data home. Match Studio’s domestic glm provider to zhipuai-coding-plan for both context limits and pricing. When an older model such as GLM-4.5 is missing from a GLM Coding Plan catalog, resolve its context/output limits and capabilities from the same vendor’s ordinary API catalog; preserve plan metadata and manual overrides, and keep pricing restricted to the plan.
  • Preserve upstream costs first, then profile-scoped custom prices, then exact provider/model catalog prices. Save the effective rates and catalog version with each estimate. Keep manual context overrides authoritative, avoid borrowing unknown relay prices, and leave ambiguous aggregate tier pricing unknown. Ekko subtask totals use run scope, and explicit call counts prevent multi-call totals from triggering a single-request context tier. App Live Activity continues counting disjoint Ekko subtask tokens while excluding other run summaries.
  • Preserve native agent cost metadata, avoid duplicate token/cost accounting, and recover Hermes session costs only for wholly unpriced sessions contained within the selected period. Update API contracts, OpenAPI, all locales, and documentation. Companion App: https://github.com/EKKOLearnAI/hermes-studio-app/pull/135.

Validation

  • Live Activity compatibility: 140 tests passed across notification usage/lifecycle, Ekko chat context, catalog pricing, usage accounting, model context/capabilities, and coding-agent context policy. A new regression first reproduced missing subtask tokens, then passed with foreground/background totals included and other profiles, time windows, estimates, and run summaries excluded. npm run build, npm run harness:check, and git diff --check passed.
  • Aggregate tier fix: 107 tests passed across model catalog, usage accounting, Ekko chat context, native agent usage, and usage store/analytics suites. The new cases first reproduced five failures before the fix and now pass, covering explicit/legacy tiers, cold-start enrichment, and reported/manual price precedence. npm run build, npm run harness:check, and git diff --check passed.
  • GLM-4.5 follow-up: 82 tests passed across model context, catalog pricing, runtime capabilities, and chat context suites. Build and harness checks passed. An isolated controller check against the real local catalog returned 131,072 context tokens and 98,304 output tokens; newer GLM plan metadata retained priority and metered API prices were not borrowed.
  • Two-decimal cost formatting retained; the focused cost presentation tests pass.
  • GLM alias follow-up: 74 focused tests, build and harness checks passed. Replayed two actual GLM records into isolated SQLite: 1,000,000-token context, 131,072-token output limit, and catalog plan rates saved as estimated $0 with pricing snapshots.
  • Focused catalog/context/accounting regression suite: 109 tests passed across 8 files.
  • Live catalog download and isolated SQLite replay: context/output limits resolved, two deduplicated records retained their pricing snapshots, and the expected $0.275978 estimate was persisted. Existing development records remained unchanged.
  • PLAYWRIGHT_PORT=4273 npm run test:e2e -- --workers=4: 255 passed; one XLSX preview assertion timed out and passed on an isolated single-worker rerun.
  • npm run build, npm run harness:check, and git diff --check: passed.
  • PORT=8648 npm run test:coverage -- --maxWorkers=4: 6644 passed, 58 failed, 14 skipped, plus one Vitest worker RPC timeout. All failing assertions also failed before this catalog change. Of these, 51 non-timeout failures were reproduced in the previous isolated baseline check against main 07ccb17a; the seven test timeouts passed when their four files were rerun with one worker (110 tests passed).

Limitations

  • Prices apply to new usage records; catalog updates and price edits do not reprice historical rows. A first launch without either network or a local catalog retains fallback context limits and unknown costs.
  • Reference estimates do not represent subscription charges, custom relay discounts, or reconciled account invoices. Unknown provider/model pairs and missing token-category prices remain unpriced unless custom rates are configured.
  • Native session-level costs use the session start date; daily coverage remains partial where a multi-day bill cannot be attributed accurately.
  • This remains a draft while unrelated full-suite failures are present.

@EKKOLearnAI EKKOLearnAI changed the title [codex] record usage costs and configure model pricing [codex] record usage costs and refresh local model pricing and limits Sep 29, 2026
@EKKOLearnAI
EKKOLearnAI marked this pull request as ready for review September 30, 2026 00:39
@EKKOLearnAI
EKKOLearnAI merged commit c8ea5ba into main Sep 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant