Report real Grok token usage and list-price cost from CLI logs - #3135
Report real Grok token usage and list-price cost from CLI logs#3135olddonkey wants to merge 11 commits into
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09cf7edb0f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
clawsweeper
Bot
added
merge-risk: 🚨 compatibility
|
Codex review: needs maintainer review before merge. Reviewed August 23, 2026, 4:36 PM ET / 20:36 UTC. ClawSweeper reviewWhat this changesThe PR replaces Grok context-window totals with bounded completed-turn CLI-log scanning, adds clearly labeled xAI list-price estimates, and keeps OpenCodex xAI records out of the Grok subscription row without request-time credential provenance. Merge readinessThis PR remains necessary: current main still derives Grok totals from session-end context occupancy, while this branch uses completed-turn records and adds bounded scanning plus explicit estimate provenance. No concrete patch defect was found; owner approval is still needed for the new non-billed list-price presentation. Priority: P2 Review scores
Verification
Live VerificationCommand: Result: FAIL (failed) — execution before step 1 Assertions:
How this fits togetherCodexBar gathers provider snapshots and local CLI history, then projects token and cost summaries into Usage & Spend. This change scans Grok session logs, resolves public catalog prices, and publishes the resulting Grok snapshot to the dashboard. flowchart LR
A[Grok CLI session logs] --> B[Bounded completed-turn scanner]
B --> C[Token and model aggregation]
D[xAI price catalog] --> E[List-price calculation]
C --> E
E --> F[Grok usage snapshot]
F --> G[Usage and Spend dashboard]
H[OpenCodex xAI logs] --> I[Token-only routing]
Decision needed
Why: The scanner and provenance are implemented, but whether this estimate belongs in the Grok product surface is a user-facing cost-policy choice. Before merge
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Land the bounded completed-turn correction only with explicit owner approval to show the separate, clearly labeled public list-price estimate beside Grok usage. Do we have a high-confidence way to reproduce the issue? Not applicable as a feature review; the PR body includes redacted exact-head live scanner and packaged-CLI output, while current main source independently confirms the prior occupancy-based behavior. Is this the best way to solve the issue? Yes technically: completed-turn records plus explicit incomplete-history and price provenance are the narrow maintainable correction; owner approval remains necessary for displaying the estimate. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 63dc1cd520f0. LabelsLabel justifications:
EvidenceWhat I checked:
Likely related people:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (9 earlier review cycles; latest 8 shown)
|
08360b5 to
e3cd3b9
Compare
olddonkey
changed the title
clawsweeper
Bot
added
merge-risk: 🚨 auth-provider
|
Both automated findings are addressed, plus the review's other checklist items. The inline comments were left against P1 — Preserve the Grok fallback on repeated probe failuresFixed in if provider == .grok {
if self.tokenSnapshotPublicationForCurrentProviderConfig(for: provider) == nil {
Task { @MainActor [weak self] in
await self?.scanAndPublishGrokLocalTokenSnapshot(...)
}
}
} else if Self.tokenCostRequiresProviderSnapshot(provider) {
self.clearTokenSnapshot(for: provider)
}Regression coverage is in P2 — Refresh pricing before scanning Grok sessionsCorrect, and thank you — this was a genuine gap and not one the local tests would have surfaced. Fixed in Note the inline comment still points at Coverage: Real-session evidence
The same corpus on Those figures were cross-checked against an independent reimplementation of the pricing formula over the same logs; the two agree to the cent. Merge risk / branch stateRebased onto current One thing deliberately left undone: no |
clawsweeper
Bot
added
rating: 🧂 unranked krab
|
Both new findings addressed at P1 — Do not map every xAI log record to the Grok subscriptionAgreed, and taken as specified rather than argued down. Routing on the prefix alone is right for the case that motivated this — traffic authenticated with the user's Grok account, which is what makes it consume SuperGrok quota — but it silently folds an API-key user's pay-as-you-go xAI spend into the subscription row. CodexBar already models the developer platform as its own The usage log carries no per-record credential evidence; I checked every field emitted for So attribution now requires positive OAuth evidence:
Coverage:
P2 — Republish the Grok snapshot after a missing catalog refreshesI looked at this closely and am deliberately not adding a republish path. Reasoning, so you can overrule it if you disagree: The refresh is fire-and-forget, so the scan that requests it returns whatever the cache currently holds — that part is accurate. But the parse cache stores parsed turns, not prices, so aggregation and pricing re-run on every The alternative — plumbing a completion signal back across the actor boundary into the Happy to build it if you would rather have it. EvidenceThe attribution itself only becomes visible in the app: The two halves stay distinguishable in the UI, which makes the attribution legible rather than something you have to take on trust: the CLI goes through the responses API so its SKU is
Independently recomputing the same corpus agrees to the cent on both halves: 54,121,501 tokens / $50.52 for the CLI logs, and $166.25 across 1,520 OpenCodex The negative direction — API-key traffic staying off the Grok row — is covered by tests rather than a screenshot, since demonstrating it live would mean rewriting the machine's OpenCodex config. StateFull suite on 
|
clawsweeper
Bot
added
proof: 📸 screenshot
olddonkey
changed the title
|
Addressed both current findings in
Validation on the exact pushed head:
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
clawsweeper
Bot
added
merge-risk: 🚨 availability
|
Pushed Final exact-head validation:
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
clawsweeper
Bot
added
rating: 🦐 gold shrimp
|
Final exact-head status for
The only remaining gates are intentionally human: a fresh redacted exact-head live Grok proof (requires explicit authorization under repository policy) and owner approval of the clearly labeled non-billed xAI list-price signal. |
e683918 to
59b1908
Compare
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
clawsweeper
Bot
added
proof: sufficient
|
Final exact-head status for
The only remaining gate is the documented owner decision on whether the clearly labeled, non-billed public xAI list-price estimate belongs in the Grok product surface. |
…on logs two ways and expensive in a third. Wrong tokens: the scanner summed `contextTokensUsed` from `signals.json`, which is the session's ENDING context-window occupancy, not what it consumed. On a real machine that reported 653K where actual consumption was 48.0M. Read the sibling `updates.jsonl` instead, where every `turn_completed` event carries the turn's real usage, and bucket by the per-line timestamp so a session crossing local midnight lands in both days. No cost: `toCostUsageTokenSnapshot` hardcoded nil dollars, and nothing could have priced a Grok model anyway because `codexModelsDevProviderIDs` had no `xai`. Add it, and resolve `grok-<version>-build` onto its base catalog model — the `-build` suffix is an artifact of the responses-API surface, not a separate SKU. `grok-build-0.1` is a real model and is never rewritten. Cost is the public xAI card via models.dev, provenance `.listPriceEstimate`, so Grok stays comparable with Claude and Codex. grok's own `costUsdTicks` is deliberately not used for display. A turn's `usage` is the aggregate of `modelCalls` API calls, so tiering on the turn total would push nearly every multi-call turn into the >=200k bracket. Price on the per-call average instead, in closed form over the two synthetic call groups. This under-tiers slightly when context grows within a turn (measured ~4% below the vendor's own accounting on a 27-turn sample, against ~+10% for aggregate tiering); the trade is documented at the call site and pinned by a test. Main-actor cost: the scan ran synchronously inside `@MainActor UsageStore` on every menu-card build, refresh and dashboard load. It now reads the projection the async probe already produced, and the remaining fallback scans on a detached task with one scan in flight at a time. The probe projects the maximum window and consumers narrow it, so `costUsageHistoryDays` and the dashboard's 365-day request are both honoured. Hardening: `modelCalls` comes from a file, so it is validated before it can size any work; parsing is cached per (path, size, mtime) with entries evicted when a file is no longer visited; the cache lock is not held across file reads. Note for upgraders: adding `xai` to `codexModelsDevProviderIDs` changes the Codex pricing-cache key, so the first launch after this re-prices existing Codex history once. Same one-time cost as when kimi and deepseek were added.
OpenCodex sends inference straight to api.x.ai using the Grok account's OAuth credentials, so it burns the same SuperGrok subscription the Grok provider reports on. It only spawns the `grok` binary to refresh tokens, so those requests never reach ~/.grok/sessions and the local session scanner cannot see them — 1,435 requests on one real machine that CodexBar attributed to nothing. Route the `xai` provider prefix to the Grok subscription, the same way `openai` already routes to Codex. Like that mapping, this routes on the prefix and does not distinguish OAuth from API-key traffic. The `-build` suffix seen in the data is a responses-API protocol artifact, not a separate billing pool, so traffic is not split by it. Routing alone would have produced tokens with no dollars. The aggregator priced the bare `entry.model`, and a name without a route prefix is resolved against the `openai` provider — which is why `gpt-5.6-sol` prices today and `grok-4.6` resolved to `openai/grok-4.6` and missed. Qualify an unprefixed model with its provider before pricing. Codex rows are unaffected (the qualified name resolves to the same target), and providers outside the supported set keep returning nil.
Grok resolved list prices straight out of the cached models.dev catalog, but nothing in its path ever fetched that catalog. The only fetch trigger is CostUsageFetcher.refreshPricingIfAllowed, which is gated to Codex and Claude — and Grok never reaches it at all, because its snapshot comes from the provider probe rather than the shared token-cost pipeline. On a machine where Codex or Claude is also enabled the cache is already there, so this is invisible. Enable only Grok and the file never appears: every price lookup returns nil and the Cost row shows tokens with no money, permanently. Request ModelsDevPricingPipeline.refreshIfNeeded from the Grok scan paths. It is safe to call repeatedly — it returns immediately unless the cache is stale and serialises through its own coordinator — and it is detached rather than awaited, matching how the Codex and Claude paths already treat it: pricing availability must never delay or fail a local scan, and the next refresh fills in the value. `summarize` stays synchronous and side-effect free; the refresh lives in a wrapper so the parse-cache behaviour and existing tests are untouched. Reported as P2 by the automated review on the pull request.
… proof The opt-in live proof scanned real sessions but printed tokens only, which cannot evidence the half of this change that is about money. It now also reports today's and the window's list-price cost, the provenance, the window actually used, and how many days carried a price versus tokens — so an all-unpriced result is visible in the output instead of reading as zero. Still skipped unless CODEXBAR_LIVE_GROK_CATALOG_PROOF=1.
The regression guard drove a single failing refresh after a local publication existed. The defect it covers is specifically about the *second* failure: the first one publishes through the fallback scan, and only the next one arrives with a publication already in place — which is what used to hit the generic clear branch. Drive the failure twice and assert the row and the scan count both hold.
Routing every OpenCodex `xai` record to the Grok subscription is right for the case that motivated it — traffic authenticated with the user's Grok account, which is what makes it burn the SuperGrok quota. It is wrong for anyone using an xAI API key: their pay-as-you-go developer-platform spend gets folded into the subscription row, silently inflating it. CodexBar models that platform as its own xAI provider precisely to keep the two apart. The usage log carries no per-record credential evidence, so the decision has to come from the OpenCodex provider config, which records `authMode` per provider. Read it, and attribute to Grok only when that mode is OAuth; anything else is token-only spend that belongs to no tracked subscription. Fail closed: a missing or malformed config, no `xai` entry, or an absent `authMode` all count as no OAuth evidence and keep the records off the Grok row. The dispatcher stays a pure function — the set of OAuth-backed provider ids is threaded in from the caller rather than read at the routing site — and the gate applies only to `xai`, leaving the other routes exactly as they were. Also records why the Grok pricing refresh stays fire-and-forget: the parse cache holds parsed turns rather than prices, so the next scan reprices against the refreshed catalog, and plumbing completion back to republish was judged disproportionate to a delay Codex and Claude already share. Raised as P1 by the automated review; the owner chose verifiable attribution over prefix-only routing.
59b1908 to
3b64253
Compare
|
Rebased onto current
@clawsweeper re-review |
|
🦞👀 Re-review progress:
|
1 participant
Problem
#3085 enabled Grok token cost, but the local session projection was not measuring actual consumption:
signals.jsonexposes ending context-window occupancy, not per-turn token usage. On a real corpus it reported 653K tokens where the completed turns contained 54.1M.nil, and the models.dev resolver did not include the xAI catalog.updates.jsonlfiles could not safely remain on@MainActor.What this changes
Read the completed-turn usage that the Grok CLI actually records
GrokLocalSessionScannerreadsturn_completedevents from each session'supdates.jsonl, matches bothsession/updateand_x.ai/session/update, and buckets each line by its own timestamp. It preserves raw aggregate token totals and uses the recordedmodelCallscount only to approximate per-call tiered list pricing in O(1).The parser now streams through the shared chunked JSONL reader instead of loading whole files. Production bounds are explicit: a 64 MiB tail per file, 1 MiB per record, 20,000 retained turns per file, and global scan budgets of 256 recent sessions, 256 MiB, and 100,000 turns. The process-wide LRU cache retains at most 64 files or 50,000 turns. Cancellation is checked between chunks, I/O/cancelled results are not cached, and any truncation marks history incomplete rather than presenting a partial total as complete.
Price the Grok models and preserve provenance
The models.dev xAI catalog is now eligible for cost pricing. Responses-API names such as
grok-4.6-buildresolve to the basegrok-4.6SKU after exact lookup, while real independent names such asgrok-build-0.1remain untouched. Cost is published as.listPriceEstimate; Grok's internalcostUsdTicksis not presented as billed spend.On a stale catalog, the scan prices immediately and refreshes in the background. On a fresh install with no catalog artifact, the first scan now awaits the initial best-effort refresh attempt before creating the snapshot, so a successful first refresh is visible in the first publication. A refresh failure still degrades to token-only data rather than failing the local scan.
Keep scanning and publication off the main actor
The provider projection consumes the async probe's snapshot. Remaining fallback paths scan on a detached utility task with a single scan in flight. A maximum-window snapshot is narrowed by each consumer through
CostUsageTokenSnapshot.narrowed(toHistoryDays:calendar:), including the spend dashboard's 365-day request.Keep OpenCodex xAI history out of the Grok subscription row
OpenCodex
usage.jsonlrecords provider/model usage but does not retain the credential mode used for each request. Reading today'sconfig.jsoncannot distinguish older API-key traffic from older OAuth traffic after a configuration switch.For that reason, xAI entries now remain
.tokenOnlyand are not merged into the Grok subscription row until the producer records request-time credential provenance. The current-config reader and its routing parameter were removed, with dispatcher/fan-out regressions and updated documentation. Other existing OpenCodex subscription routes are unchanged.Review findings addressed
bdfc10e0dwith chunked tail reads, per-file and global byte/turn/session budgets, bounded LRU retention, cancellation checks, and incomplete-history propagation. Regressions cover tail truncation, global scan budgets, and cache eviction limits.signals.json/30-day description now documentsupdates.jsonl, the requested window up to 365 days, list-price provenance, and every production bound.signals.jsonis metadata-only and also byte-limited.Real-session evidence
The opt-in proof was rerun on exact head
3b642530aagainst native Grok CLI sessions through the shipped bounded scanner, snapshot projection, pricing, and spend-catalog path:The current local corpus has no completed-turn tokens today and one priced day in the last 30 days; these values naturally change as local logs age. The packaged production CLI on the same exact head also returned
provider=grok,source=grok-cli-proxy,usage=yes, and no provider error. Only redacted aggregate fields are reported here; the cost remains explicitly a public list-price estimate, not billed spend.Testing
3b642530a:swift test --skip-build --filter 'OpenCodexRouteDispatcherTests|OpenCodexUsageFanOutTests|GrokLocalSessionScannerTests'— 35 tests passed.3b642530a:make check— parser hash, manifests, packaging scripts, documentation links, SwiftFormat (0/1985) and SwiftLint (0 violations) passed.5b10ef33c:make test— 923 selections, 77/77 groups, 0 failed groups, 0 retries, 0 timeouts (1066.8 seconds). The subsequent upstream commit63dc1cd52changed onlyCHANGELOG.md; the final rebase did not change production or test sources.3b642530a:./Scripts/package_app.sh— production bundle, signing validation, helper/resource probes, and launch smoke check passed.3b642530a: packagedCodexBarCLI config validate— OK.3b642530a: packagedCodexBarCLI usage --provider grok ...— Grok usage returned fromgrok-cli-proxywith no error.3b642530a:CODEXBAR_LIVE_GROK_CATALOG_PROOF=1 swift test --filter GrokXAISpendCatalogTests— 2 tests passed with the redacted aggregate above.Normal automated tests use temporary homes, injected catalogs/transports, and no live credentials or interactive Keychain reads. The explicitly authorized live checks used the configured Grok provider and local session logs; no credential values were printed.
Adding
xaito the models.dev provider set changes the pricing-cache key, so the first launch re-prices existing Codex history once. The predecessor parser hash remains compatible, so the store is adopted rather than rebuilt from JSONL.Maintainer decision requested
Please confirm whether public xAI list-price estimates derived from local Grok CLI usage should remain visible as a clearly labeled, non-billed signal distinct from SuperGrok subscription credits. The implementation cannot resolve this product-policy sign-off itself.
Changelog: added under 0.54.2 — Unreleased.