Skip to content

Track multi-provider spend and model usage - #2759

Open
Chipagosfinest wants to merge 57 commits into
steipete:mainfrom
Chipagosfinest:agent/cumulative-spend-converged
Open

Track multi-provider spend and model usage#2759
Chipagosfinest wants to merge 57 commits into
steipete:mainfrom
Chipagosfinest:agent/cumulative-spend-converged

Conversation

@Chipagosfinest

@Chipagosfinest Chipagosfinest commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What this changes

CodexBar Usage & Spend now works as a true multi-provider overview instead of stopping at Claude and Codex.

  • Every enabled or configured service stays visible, including providers without spend history yet.
  • The Overview leads with combined estimated spend, tracked tokens, and a clear reporting-provider count.
  • Compact native rows and smooth scrolling keep large provider rosters usable while Codex remains first.
  • Copy Stats and Copy Image produce a shareable local summary with honest partial-coverage disclosure.
  • Top models preserve meaningful variants such as Claude Fable, Opus, and Sonnet.

Unknown spend is never turned into $0. Estimates use ~, currencies remain separate, and incomplete coverage stays explicit.

Screenshots

All values shown here are synthetic.

Combined Overview

CodexBar Overview with combined estimated spend, token total, and six providers

Share card

CodexBar local share card with six tracked services, partial spend coverage, and top models

Wide settings · Narrow settings · Proof notes

OpenRouter scope

OpenRouter stays inside the existing plugin boundary on JavaScriptCore and QuickJS. An ordinary API key continues to report credits and key quota.

Management /activity enrichment is deliberately deferred to a separate follow-up. Removing it restores the existing single optional-request timeout budget and avoids adding a provider-specific history payload or a broader async QuickJS host contract to this PR.

Final integration fixes

  • Merged current main at bff43f8a2cc736a9dd248f6fb275f594b3a472c7.
  • Changed share copy from "connected services" to truthful "tracked services" terminology.
  • Preserved the saved overview ordering while enforcing explicit custom provider caps.
  • Reconciled the exact-line provider architecture gatekeeper with the merged source.
  • Repaired a stale integration assertion to verify the requested history window is forwarded.

Verification

Exact head: 6e7b6178a3cc2ed86e1e9490b7e551b1eed997a8

  • Exact-head CI run 31452605577: all 9 checks passed, including both macOS Swift test shards, Linux x64/arm64, Linux musl, lint, and GitGuardian.
  • Provider architecture catalog: 389 exact anchors validated with zero mismatches after the merge.
  • SwiftFormat and git diff --check passed for the follow-up fixes.
  • Local screenshot proof covers the production Overview, share window, responsive Settings layouts, and Copy Stats flow with synthetic provider data.

This is the current-main successor to #2469 and includes the provider-runtime convergence from #2758. OpenRouter /activity is preserved separately for a clean follow-up after this lands.

Stacked follow-up

#2855 adds the next focused layer: a provider-colored known-token allocation strip and approximate same-row cost per 1M tokens in the Overview summary card.

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. labels Aug 8, 2026
@clawsweeper

clawsweeper Bot commented Aug 8, 2026

Copy link
Copy Markdown

Codex review: needs real behavior proof before merge. Reviewed August 14, 2026, 5:38 AM ET / 09:38 UTC.

ClawSweeper review

What this changes

This PR adds 7-, 30-, and 365-day multi-provider spend reporting across the settings dashboard, menu overview, and local share card with coverage-aware totals.

Merge readiness

Blocked until stronger real behavior proof is added - 7 items remain

Keep open for maintainer product direction and stronger runtime proof. The branch changes an established six-provider fallback into an all-provider default for profiles without a saved selection, which requires explicit approval or an upgrade-preserving migration.

Priority: P2
Reviewed head: d28758308e50523e26256fc3dc4dcf06d374c991
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The feature has broad focused test and screenshot evidence, but merge readiness is limited by the unapproved upgrade behavior and missing live timeout-recovery proof.
Proof confidence 🦐 gold shrimp (3/6) Needs stronger real behavior proof before merge: Synthetic production-component screenshots credibly show the dashboard, menu, and share-card UI, but they do not show after-fix Codex RPC timeout/EOF recovery; add a redacted terminal trace or runtime log from that path. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality 🐚 platinum hermit (4/6) 1 actionable review finding remain.

Verification

Check Result Evidence
Real behavior Needs proof Needs stronger real behavior proof before merge: Synthetic production-component screenshots credibly show the dashboard, menu, and share-card UI, but they do not show after-fix Codex RPC timeout/EOF recovery; add a redacted terminal trace or runtime log from that path. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed 7 items Current default contract: Current main caps the merged overview at six providers when no selection preference exists.
Regression contract: Current main has a focused test that defines an empty selection as the first six of seven active providers.
Proposed behavior: The supplied PR patch changes the limit to all provider cases and changes the empty-selection test to expect all 20 active providers.
Findings 1 actionable finding [P1] Preserve or approve the no-selection fallback
Security None None.

How this fits together

CodexBar gathers local usage and cost snapshots from configured providers, aggregates compatible sources into Usage & Spend, then presents the result in Settings, the menu-bar overview, and a local share card. Provider-selection settings determine which overview rows appear, while cost-history availability controls which sources can contribute to totals.

flowchart LR
A[Configured providers] --> B[Usage and cost snapshots]
B --> C[Spend aggregation]
C --> D[Coverage and currency checks]
D --> E[Settings dashboard]
D --> F[Menu overview]
D --> G[Local share card]
Loading

Decision needed

Question Recommendation
Should an upgrade with no saved Overview selection retain the established six-provider fallback, or intentionally expand to every active provider? Preserve existing menus: Migrate legacy no-selection profiles to their prior first-six selection while using the all-provider default only for new installs.

Why: Current code and tests make absence of a selection a meaningful six-provider behavior, while the PR intentionally changes that behavior for existing users; VISION.md requires maintainer sign-off for this user-facing feature and behavior change.

Before merge

  • Add real behavior proof - Needs stronger real behavior proof before merge: Synthetic production-component screenshots credibly show the dashboard, menu, and share-card UI, but they do not show after-fix Codex RPC timeout/EOF recovery; add a redacted terminal trace or runtime log from that path. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
  • Preserve or approve the no-selection fallback (P1) - Current main defines an absent selection as the first six active providers, but this change makes it every active provider. That silently expands existing users’ Overview menus on upgrade; retain the legacy fallback via migration or obtain explicit product approval before merge.
  • Resolve merge risk (P2) - Existing profiles without a saved overview selection will move from the documented/tested six-provider fallback to every active provider after upgrade.
  • Resolve merge risk (P2) - The screenshots prove the visual flows but do not show the changed Codex RPC timeout/EOF recovery path succeeding in a real run.
  • Resolve merge risk (P1) - The exact PR-head blobs could not be materialized locally from the partial checkout; this review used the supplied PR diff and evidence for the branch-specific assessment.
  • Complete next step (P2) - A maintainer must choose the upgrade/default behavior; contributor proof must also cover the changed timeout-recovery path before merge.

Findings

  • [P1] Preserve or approve the no-selection fallback — Sources/CodexBar/SettingsStore.swift:191
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Patch scope 74 files, +3,464 / -351 lines The multi-surface feature spans provider aggregation, menu behavior, sharing, localization, docs, and tests, so the default change needs deliberate upgrade review.

Root-cause cluster

Relationship: canonical
Canonical: #2759
Summary: This PR is the stated successor to the closed unmerged multi-provider spend PR; the related allocation and durable-history PRs add distinct follow-on behavior.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

  1. Preserve legacy fallback on upgrade (recommended)
    Keep existing no-selection profiles at six overview rows through a tested migration, while new installs may adopt the all-provider default.
  2. Explicitly accept the menu expansion
    Approve the all-provider default as an intentional existing-user behavior change and document that upgrade effect for reviewers.
  3. Split the default change
    Pause the all-provider default until product direction is settled and retain only independently approved dashboard work.

Technical review

Best possible solution:

Preserve the six-provider fallback for existing no-selection profiles through a narrow migration while allowing the all-provider default for new installs, unless the product owner explicitly approves changing existing menus.

Do we have a high-confidence way to reproduce the issue?

Yes—source and the current focused test establish the existing seven-active-provider path resolves to six rows, while the PR patch changes that expected result to all active providers.

Is this the best way to solve the issue?

No until direction is confirmed: preserving the established fallback for existing profiles via a narrow migration is safer than silently expanding their menus.

Full review comments:

  • [P1] Preserve or approve the no-selection fallback — Sources/CodexBar/SettingsStore.swift:191
    Current main defines an absent selection as the first six active providers, but this change makes it every active provider. That silently expands existing users’ Overview menus on upgrade; retain the legacy fallback via migration or obtain explicit product approval before merge.
    Confidence: 0.97

Overall correctness: patch is incorrect
Overall confidence: 0.93

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 1f5abbc7df07.

Labels

Label changes:

  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • remove rating: 🦪 silver shellfish: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.
  • remove merge-risk: 🚨 availability: Current PR review merge-risk labels are merge-risk: 🚨 compatibility.

Label justifications:

  • P2: This is a substantial but non-emergency user-facing feature and upgrade-behavior decision.
  • merge-risk: 🚨 compatibility: The PR replaces the established six-provider fallback for existing profiles that have no saved overview selection.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Synthetic production-component screenshots credibly show the dashboard, menu, and share-card UI, but they do not show after-fix Codex RPC timeout/EOF recovery; add a redacted terminal trace or runtime log from that path. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
  • proof: 📸 screenshot: Contributor real behavior proof includes screenshot evidence. Synthetic production-component screenshots credibly show the dashboard, menu, and share-card UI, but they do not show after-fix Codex RPC timeout/EOF recovery; add a redacted terminal trace or runtime log from that path.

Evidence

What I checked:

Likely related people:

  • Peter Steinberger: Introduced the ProviderInstanceID seam that owns the current SettingsStore provider-selection surface. (role: recent settings and provider-identity contributor; confidence: high; commits: 15a855a2132a; files: Sources/CodexBar/SettingsStore.swift, Sources/CodexBar/SettingsStore+Defaults.swift)
  • Yuxin Qiao: Merged recent integration coverage through the spend dashboard, adjacent to this PR's aggregation and history-window work. (role: recent spend-dashboard integration contributor; confidence: medium; commits: 68f26b5d65c9; files: Sources/CodexBar/SpendDashboardController.swift, Tests/CodexBarTests/SpendDashboardTokenActivityIntegrationTests.swift)
  • Vito Cappello: Recently added a provider, making the behavior of a provider-count-based overview limit especially relevant. (role: recent provider-roster contributor; confidence: medium; commits: 6e4a753340bd; files: Sources/CodexBar/SettingsStore.swift)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Add a redacted runtime trace showing a Codex RPC timeout is classified as timeout rather than a competing EOF.
  • Obtain product-owner direction on the existing no-selection fallback.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (59 earlier review cycles; latest 8 shown)
  • reviewed 2026-08-09T19:35:53.777Z sha 8bcaf4e :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-09T19:49:56.586Z sha 8bcaf4e :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-09T21:12:41.155Z sha 8bcaf4e :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-11T02:20:40.603Z sha 05b58b9 :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-11T02:41:27.885Z sha 6e7b617 :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-11T03:04:56.348Z sha 6e7b617 :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-11T05:46:37.502Z sha 6e7b617 :: needs real behavior proof before merge. :: [P1] Preserve or explicitly approve the no-selection default
  • reviewed 2026-08-14T03:20:29.142Z sha d287583 :: needs real behavior proof before merge. :: [P1] Preserve or explicitly approve the no-selection default

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b03a03bd1a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Sources/CodexBar/SpendDashboardModel.swift
Comment thread Sources/CodexBar/UsageStore+TokenCost.swift Outdated
Comment thread Sources/CodexBar/StatusItemController+OverviewSpend.swift Outdated
@Chipagosfinest

Copy link
Copy Markdown
Contributor Author

Thanks for the sharp review. Two clarifications on the product and architecture decisions:

  • All connected providers in Overview is intentional for both unset fresh and unset existing installs; preserving the six-provider fallback would keep the exact omission this PR is meant to fix. The no-selection path is covered with 20 providers, explicit saved subsets remain scoped, and long rosters use native scrolling.
  • The OpenRouter bridge is intentionally a narrow first-party exception, gated to .openrouter, bounded and validated in Swift, Codable-compatible, fail-open for optional enrichment, and covered across JavaScriptCore/QuickJS. We considered a generic token-history contract, but without a second consumer it would prematurely freeze shared accounting semantics that differ here (completed UTC days, routed models, reasoning-token inclusion, management-key scope).

I also fixed the two concrete hosted-Xcode regressions exposed by the previous run at 65327296f8850f00b96ce5885431f244b716ae43. I will request a fresh review after the exact-head CI is terminal. Pete: the remaining requested decision is whether this narrowly tested first-party plugin exception is acceptable as the durable boundary.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Aug 8, 2026
@clawsweeper clawsweeper Bot removed the rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. label Aug 8, 2026
@clawsweeper clawsweeper Bot removed merge-risk: 🚨 auth-provider 🚨 Merging this PR could break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. labels Aug 9, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. and removed proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Aug 9, 2026
@clawsweeper clawsweeper Bot added merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. and removed merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. labels Aug 9, 2026
@clawsweeper clawsweeper Bot removed the merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. label Aug 11, 2026
@Chipagosfinest

Copy link
Copy Markdown
Contributor Author

@Yuxin-Qiao this is ready for final review/merge when you have a moment.

Exact head 6e7b6178a3cc2ed86e1e9490b7e551b1eed997a8 is merged with current main, all 9 CI checks are green, GitHub reports MERGEABLE / CLEAN, and all 10 review threads are resolved.

@clawsweeper clawsweeper Bot added the merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. label Aug 11, 2026
@Chipagosfinest

Copy link
Copy Markdown
Contributor Author

@Yuxin-Qiao @steipete when you have a moment, could you take a maintainer look at this base PR?

Exact head 6e7b6178a3cc2ed86e1e9490b7e551b1eed997a8 has all 9 checks passing and zero unresolved review threads. The remaining decision is the intentional upgrade from the six-provider fallback to all active providers; ClawSweeper also asks for live timeout/recovery proof before merge. Direction on either point would help close the final gap.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Aug 14, 2026
@Chipagosfinest

Copy link
Copy Markdown
Contributor Author

Current-main integration is complete at exact head d28758308e50523e26256fc3dc4dcf06d374c991.

  • GitHub reports CLEAN / MERGEABLE against current main (3a848565bcd28426f680821de985fc1fc01e7068).
  • Hosted CI is fully green: both macOS test shards, Linux x64/arm64/musl, lint, aggregate, and GitGuardian.
  • The conflict resolution preserves the selected dashboard history window (effectiveCostUsageHistoryDays) while retaining upstream same-scope restart, cancellation/stop cleanup, and cyclic-stall protection.
  • Independent integration review found no blockers; there are zero unresolved review threads.

@Yuxin-Qiao @steipete — this is ready for maintainer review. No merge action was taken.

Copy link
Copy Markdown
Contributor

Thanks for the detailed updates and for looping me in. Just to clarify, I’m not a maintainer of CodexBar and don’t have authority to approve or merge these PRs, so I may not be the right person to make the product-scope decision. I’ll leave that to the repository maintainers. Apologies for any confusion, and thank you for the thorough work.

@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. labels Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature: ✨ showcase ClawSweeper spotlight: unusually compelling feature idea for maintainer attention. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants