Cost instrumentation: per-role spend capture, the reference cost ledger, and a per-experiment rollup (#343) by ealt · Pull Request #346 · ealt/eden

ealt · 2026-07-30T00:20:01Z

Summary

EDEN could report what a run did, never what it cost. The R2 report's economics section hand-estimated ~$35–45 and the live R3 run has the same blindness — while the executor's Claude Code agent logs have been emitting total_cost_usd per attempt the whole time and the platform dropped it. This wires up all three milestones of Cost instrumentation: persist per-role token/$ so runs can report what they cost #343: capture in every worker host, a cost ledger to hold it, and a per-experiment rollup to read it.
Cost lands in a non-normative /_reference/ ledger, not on the Variant — because giving it a normative home means amending spec/v0 MUSTs + conformance, which this work was explicitly scoped away from. The scoped spec-change plan is a comment on #343 instead.
Every failure mode in the capture path is a no-op, never an error. A malformed log, a missing key, a deadline-killed agent, an unreachable ledger — none of them may fail an otherwise-good variant. Cost is bookkeeping about an attempt, not part of it.

Companion producer PR (this repo can't see what a worker spent): ealt/eden-experiments#11.

Follow-up round (commit 09c2514), after review: per-idea attribution, per-model token splits preserved instead of collapsed, and dollars derivable from tokens × an operator-supplied rate table — with derived figures labelled so they can never pass for reported ones. No spec change was needed for any of the three; the ledger stays in /_reference/. Details in the section below.

Milestone coverage

	What landed
M1 — executor `total_cost_usd`	`agent_cost.py` parses the terminal `{"type":"result"}` record of a Claude Code `--output-format stream-json` log; executor and evaluator hosts record through one shared helper. `cost.py` / `_ops/cost.py` / `cost_entry` table on all three backends / `POST`+`GET /_reference/experiments/{E}/cost` / `StoreClient` pair.
M2 — ideator gateway tokens	Same two keys, carried on the JSON-line terminator (`ideation-done` and `ideation-error`). Bridge half is in the eden-experiments PR.
M3 — per-experiment rollup	`summarize()` (pure reduction) + `uv run python -m eden_service_common.cost_report`, joined to each variant's status + evaluation payload so DCI-per-dollar is a local computation. Operator docs: `docs/observability.md` §2.10.
F1 — per-idea attribution	Evaluator stamps `idea_id` from its variant; ideator stamps it when a dispatch produced exactly one idea. New `by_idea` rollup bucket + report section pairing each idea's spend with the variants it produced.
F2 — per-model splits	`modelUsage` preserved as a `models` list on the entry (was collapsed to a label, and dropped entirely for multi-model runs) + cache writes captured per TTL tier. New `by_model` bucket.
F3 — derived dollars	`pricing.py` + `--price-table`: tokens × per-class-per-model rates for attempts no provider priced, with `basis` labelling and gap reporting throughout.

One PR rather than three because M2 and M3 are ~200 lines of delta on M1's surface and touch the same doc sections; the commit is milestone-ordered internally and the CHANGELOG entry separates them.

The design decision worth reviewing

spec/v0 has no home for spend. That was verified rather than assumed — #343 named the eval-payload exact-key-match constraint as the known obstacle and asked whether it also binds the execution path. Probed against a real store:

Evaluation path: an undeclared key is rejected loudly (InvalidPrecondition: evaluation key 'total_cost_usd' is not in the experiment's evaluation_schema).
Execution path: no key validation and no free-form field — submission_from_payload reads named keys only, so an extra cost key is silently dropped. No error, no persistence.

Different mechanisms, same conclusion: no smuggling route, and the execution path's silence means an implementation that tried would look like it worked. Hence a ledger behind the /_reference/ surface chapter 7 §5 sanctions for extensions, kept off the Store protocol (the ArtifactStore precedent — a reference extension does not belong in the interface a conforming implementation is measured against; the routers cast exactly like the §16 artifact router).

Two ledger properties are load-bearing and worth a second opinion:

First-write-wins on entry_id, keyed per attempt. A re-record after a transport failure must not double the reported spend; and the key must identify an attempt rather than a task, because a reclaimed-and-rerun task really did spend twice. Executor keys on its freshly-minted variant_id, evaluator on (task_id, variant_id), ideator on a per-dispatch nonce (an ideation dispatch has no stable per-attempt id; nothing retries that call, so one-row-per-dispatch holds by construction rather than by key).
Attribution, not aggregation. One row per spend event; every rollup is a read-time reduction, so no stored aggregate can drift from the ledger. Cost rows carry no event — like artifact metadata they aren't bound to any transition, so the chapter-5 §2 invariant has nothing to pair them with (and the v0 registry is closed).

Follow-up round: the three design calls worth reviewing

Per-idea attribution stops where it would become invention. The executor and evaluator both know the idea behind their variant, so those attribute cleanly. An ideation dispatch that produced several ideas spent one indivisible gateway call on all of them — so it is attributed to none, because picking one or splitting N ways are both fabrications. by_idea is therefore a documented partial partition, and reports how many entries it excluded (unattributed.by_idea) so "no idea cost anything" can't be confused with "attribution wasn't available". Landing this required moving the ideator's single cost-record into a finally after _persist_ideas mints the ids; that trades one failure window for a better one (a crash between submit and record loses a cost row; the old order would have lost the submission).

Per-model splits are kept as structure, not a label. modelUsage is now preserved whole as a models list on the entry — cost stays attributed per attempt, and the split is structure the read-time reduction slices. by_model deliberately carries no num_turns / duration_ms (they belong to the attempt; a per-model share would be fabricated), and it does cover single-model attempts by synthesizing their one slice — attributing a single-model attempt's whole usage to its one model is exact, not an allocation. Cache writes are additionally captured per TTL tier from usage.cache_creation, which is the precondition for pricing them at all.

Derived dollars are labelled, never blended. Three enforced properties, each because its absence is a specific way to mislead:

Nothing writes a computed figure into the ledger. CostEntry.total_cost_usd keeps meaning "the provider said so"; derivation is read-time, lands in separate reported_cost_usd / derived_cost_usd fields, and every bucket carries a basis of reported / derived / mixed / unpriced. The table render says DERIVED in words.
Rates are configuration with provenance: a table MUST declare source + as_of, both echoed into the report so a figure is auditable against the rates that produced it — not against whatever the table says later. An unfilled table prices nothing and is marked UNFILLED in the output.
An unpriced class is a reported gap, never a zero — including cache writes whose TTL tier was never reported, which is why rates are per token class per model (cache reads run ~an order of magnitude cheaper than fresh input; 1-hour writes materially dearer than 5-minute), not one blended number.

Two bugs from this round worth naming, both caught by driving the real path: models as a tuple is unbuildable from a JSON array under strict=True (it made every multi-model entry both unrecordable and un-POSTable), and a per-model row whose basis said derived displayed $0.0000 until DerivedCost.per_model credited each model its own tokens' cost — "derived, $0.0000" reads as "this model was free", which is worse than no row.

Two refactors the complexity gate forced (both worth having)

_execute_and_validate crossed the 100-line threshold → phases 2d–2e split into _validated_commit_from_outcome. postgres.py crossed 800 SLOC → the ledger primitives moved to a _postgres_cost.py sibling (the _postgres_schema.py / _postgres_views.py precedent), which also puts the whole non-normative extension behind one clearly-labeled file. In the follow-up round render_table crossed the length gate → split into per-section helpers, and the rollup moved into its own rollup.py (records → pricing → rollup is a clean three-layer stack; keeping the reduction in cost.py would have been a pricing↔cost import cycle). No # slop-allow annotations added.

What this does NOT cover

Cost is not in checkpoints. A restore starts with an empty ledger, so a run that survived one reports only its post-restore spend. The chapter-10 archive layout is normative and extending it is spec surgery → Cost ledger rows are not carried in checkpoint export/import (#343 deferral) #344 (with the three options + a recommendation, and a note that the gap should at minimum be made visible if it persists — smoke-checkpoint.sh passed while Checkpoint archives carry an empty git bundle under Compose (no --repo-path on task-store-server) #294 existed for exactly this reason).
No cost_entry_unpacked Postgres view for EDEN_READONLY_STORE_URL analysis consumers, so direct-SQL readers cast JSON by hand → Add a cost_entry_unpacked Postgres view for readonly analysis consumers (#343 deferral) #345.
No normative home for cost (a Variant field or first-class record). Four options + a recommendation are on Cost instrumentation: persist per-role token/$ so runs can report what they cost #343; the short version is "first-class record, timed with a v1 lineage, because the checkpoint format bump argues for a lineage boundary".
AWS cost-allocation tags: propose-only on Cost instrumentation: persist per-role token/$ so runs can report what they cost #343 — needs AWS permissions, and the R3 box is off-limits while the run is live. The proposal includes the two non-obvious parts: activation is management-account-only and not retroactive (so it's worth doing before the next run), and tags are per-resource, so shared RDS / shared S3 buckets cannot be split per experiment at all. Also argues infra cost should not go in this ledger (per-attempt vs per-time-window granularity).
Budget cap: propose-only on Cost instrumentation: persist per-role token/$ so runs can report what they cost #343. It did not drop out trivially: the obvious home (a termination_policy kind) is a closed enum in the normative config schema, so it needs either spec surgery or a deployment-level CLI flag; the cap necessarily overshoots by the in-flight work; and unreported cost forces a fail-open-vs-fail-closed decision (recommendation: fail-open with a loud warning, because fail-closed halts every uninstrumented run).
Timed-out attempts under-report. A deadline-killed agent's log has no terminal result record, so its spend is unrecoverable from the log — the attempt shows in the event log but not the ledger. Attempts that errored but finished do record. Documented in docs/observability.md §2.10 rather than papered over; recovering it would need per-turn accumulation, which disagrees with the result totals.
No real prices are shipped. reference/pricing/price-table.example.json has every rate null and as_of: "unset", and in that state it derives nothing. Deliberate per the guard on this work: a number I can't source would be wrong for someone (list vs negotiated, direct vs Bedrock, region) and a wrong rate that looks authoritative is worse than an explicit gap. This is the one thing left for you: supply the rates you actually pay (and their source + date) and every unpriced ideator attempt becomes a dollar figure.
Cache-write pricing needs the TTL split, which per-model data lacks. usage.cache_creation carries the 5m/1h breakdown at attempt level (the real capture shows this run's writes were all 1-hour), but modelUsage reports only an aggregate per model — so cache writes on a multi-model attempt are reported as a gap rather than priced at a guessed tier. Same for any source that reports only the aggregate.
Ideator per-idea attribution is single-idea-only. A dispatch that emits several ideas is attributed to none (see above). R3's bridge emits one per dispatch, so this is the common case today; a run configured with EDEN_IDEAS_PER_IDEATION > 1 will see those entries under unattributed.by_idea.
The wire rejects unknown fields, so host and server must upgrade together. CostEntry is extra="forbid" on both sides; a new host POSTing models at a pre-follow-up server would 400. Not an issue for Compose/Helm (one image), worth knowing for a partial rollout.
Multi-model runs report no model label. More than one entry in modelUsage means no single honest label; the aggregate total_cost_usd and token counts are still exact. Per-model cost splitting is out of scope (cost attribution here is per attempt).
No conformance assertions. Deliberate: the ledger is non-normative, so nothing in conformance/ may depend on it (that would codify a reference-impl quirk as contract). The suite stays green — 260 passed.
Postgres rows not exercised locally. No Postgres server available in this environment, so the postgres parametrizations skip. CI's python-test-postgres job covers them. The one structural risk the _postgres_cost.py extraction introduced (MRO position — a bases reorder would silently route to _StoreCore's NotImplementedError stubs) is covered by a server-free guard test.
Compose / Helm smokes not run locally. No Docker daemon available in this environment. Per AGENTS.md these are the gate most worth running and the most often skipped, so flagging it explicitly: CI runs them on this PR, and I have not independently verified the ledger under Compose. The changed hosts are on the smoke path, so a regression there would surface in compose-smoke / compose-smoke-subprocess.

Fresh-operator walkthrough

The report CLI is a new operator-facing surface, so it was driven end-to-end against a real task-store-server (in-memory store, fixture experiment config, random port) rather than only unit-tested:

Started the server, seeded an idea + variant + two cost entries through StoreClient, then re-recorded one entry to exercise idempotency over the wire: 3 writes → 2 rows.
--format table:

experiment: exp_01hzzzzzzzzzzzzzzzzzzzzzzz
total: $0.1233 over 2 attempt(s) — 1 attempt(s) reported tokens but no dollar figure

role         attempts  no_usd          usd       in_tok    out_tok
executor            1       0      $0.1233            6        637
ideator             1       1      $0.0000         4200        830

variant                    status                      usd evaluation
variant-1                  starting                $0.1233 -

--role executor (JSON): filter forwarded to the wire, by_variant carries status + idea_id + evaluation, absent figures absent rather than null.
Walkthrough finding, fixed: the first table render showed the ideator row as $0.0000, which reads as "this role was free" when the truth is "this role reported tokens but no dollar figure". Added the per-role unpriced column — the header caveat alone wasn't enough, because the per-role rows are what a reader actually scans.

Follow-up round walkthrough (re-run against a live server with two ideas, a variant, one provider-priced executor attempt carrying a two-model split, and two token-only gateway attempts) — three passes: no table, the shipped unfilled template, and a filled one.

=== (c) filled rate table ===
total: $4.9233 [mixed] over 3 attempt(s)
  of which $0.1233 was reported by the provider (1 attempt(s)) and $4.8000 was DERIVED from tokens x rates (2 attempt(s))
  rates: walkthrough fixture — NOT real prices (as of 2026-07-30, usd_per_mtok)

role         attempts unpriced          usd basis           in_tok    out_tok
executor            1        0      $0.1233 reported             6        637
ideator             2        0      $4.8000 derived        1300000      60000

model                               slices          usd basis           in_tok    out_tok   cache_rd
anthropic/gateway-model                  2      $4.8000 derived        1300000      60000          0
claude-sonnet-4-6                        1      $0.1233 reported             6        637      47413

idea                       slug           state                usd basis     variants
idea-dud                   p1             drafting         $3.3000 derived          0
idea-good                  p0             drafting         $1.6233 mixed            1

idea-dud cost $3.30 and produced zero variants — the row the whole per-idea bucket exists to produce. Two more findings fixed from this pass: idea rows had no basis column (a $0.0000 unpriced idea read as free, the same trap as the role rows), and passing the unfilled template printed its placeholder source without saying it was doing nothing (now marked UNFILLED, nothing derived, plus a stderr warning).

Test plan

Follow-up-round tests: test_cost_pricing.py covers the rate table as configuration (provenance required, per-token unit rejected, malformed table raises rather than silently reading as "no table", unfilled template prices nothing, alias resolution), the per-class arithmetic including both cache tiers priced differently and cache reads far cheaper than fresh input, every gap path (untiered cache writes, missing class rate, unknown model, no model label), multi-model derivation with one unknown model not voiding the others, and the rollup's reported/derived/mixed/unpriced basis + per-model credit. Host-level: single-idea dispatch attributes, multi-idea dispatch attributes to none, failed dispatch records without an idea, evaluator's idea_id comes from its variant. Parser: the real capture's per-model split and its 5m/1h tiers verbatim, multi-model preservation, malformed modelUsage entries skipped not fatal.

Original-round tests: ledger semantics parametrized across all three backends (idempotency, cross-backend read order, no-events, experiment-id mismatch, partial entries, validation); the stream-json parser against a real captured log from an eden-experiments belief-state-recovery execution task (reduced to one line per record type, prose/session-ids/hook-output redacted, every result number verbatim) plus synthesized degradation cases (killed agent, interleaved stderr, truncated final line, tail-cap boundary in both directions, NaN/Infinity, multi-model); wire round-trip incl. self-gated bearer auth on both cost routes; host-level tests driving the real _handle_one for all three roles — including the relative-agent_log case, which is the only way to catch a regression that moved extraction after worktree cleanup.

Related issues

Refs #343 — cost instrumentation (not closed: AWS tags + budget cap remain propose-only, and the normative-home decision is open)
Refs #344 — cost rows not carried in checkpoints (filed with this PR)
Refs #345 — cost_entry_unpacked Postgres view (filed with this PR)
Companion: ealt/eden-experiments#11 — the producer half (deploy-order independent; additive keys an older host ignores)

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y8jE6s96VcMT8MxxPvnxD3

Closes the blindness #343 describes: EDEN could report a run's duration and throughput but not what it cost, so the R2 report hand-estimated ~$35-45 and the live R3 run has no better answer. The executor's Claude Code agent logs already emit `total_cost_usd` per attempt; the platform dropped it. All three of #343's milestones, in one PR because the second two are small deltas on the first's surface. Where cost lives, and why not on the Variant. spec/v0 has no home for spend, verified rather than assumed (#343 asked whether the eval-payload exact-key-match constraint also binds the execution path): on the evaluation path an undeclared key is rejected loudly; on the execution path there is no key validation AND no free-form field, so an extra `cost` key is silently dropped by `submission_from_payload`. Different mechanisms, same conclusion — no smuggling route, and a normative field would mean amending spec MUSTs + conformance, which the balloon-guard on this work put out of scope (the spec-change plan is a comment on #343). So: a non-normative ledger behind the `/_reference/` surface chapter 7 §5 sanctions. `CostLedger` protocol with record/list, a `cost_entry` table on all three backends (SQLite v10 + mirrored Postgres migration), `POST`/`GET /_reference/experiments/{E}/cost` with a matching `StoreClient` pair so a host writes cost identically in-process or across the wire. Kept off the `Store` protocol on purpose (the `ArtifactStore` precedent): a reference extension does not belong in the interface a conforming implementation is measured against. Two ledger properties are load-bearing. First-write-wins on `entry_id` so a re-record after a transport failure cannot double the reported spend — and the key is per-ATTEMPT, not per-task, because a reclaimed and rerun task really did spend twice. Attribution, not aggregation: one row per spend event with the role/task/variant/idea, so every rollup is a read-time reduction. Cost rows carry no event; like artifact metadata they are not bound to any transition, so the chapter-5 §2 invariant has nothing to pair them with. Capture. A host never talks to an LLM itself, so the user's `*_command` reports via two optional keys on the outcome JSON the host already reads (`agent_log`, a stream-json log the host parses; or `cost`, normalized figures) — for the R3-shaped experiment that is one added key naming a log it already writes. The ideator's arrives on its JSON-line terminator, honored on `ideation-error` too since a failed attempt still burned tokens. Recording happens inside the worktree's lifetime and regardless of how the attempt terminalizes. Every failure mode is a no-op, never an error, including an unreachable ledger. Rollup. `summarize()` is a pure reduction into per-role and per-variant totals — not a stored aggregate and not a server-side endpoint, so it cannot drift from the ledger. `cost_report` prints it joined to each variant's status + evaluation payload, which is what makes DCI-per- dollar local instead of a two-source join every consumer rewrites. `entries_missing_cost_usd` keeps a partial total from reading as a complete one. Two refactors the size gate forced, both worth having: `_execute_and_validate` split at the run/validate seam, and the Postgres ledger primitives moved to a `_postgres_cost.py` sibling (the `_postgres_schema.py` precedent) — which also puts the whole non-normative extension behind one labeled file. No slop-allow added. Deferred: cost in checkpoints (#344, needs a normative format bump), a `cost_entry_unpacked` Postgres view (#345). Propose-only on #343: a normative home for cost, AWS cost-allocation tags, an orchestrator budget cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8jE6s96VcMT8MxxPvnxD3

Two failures the CI typecheck caught (and I mis-reported as clean — I read a truncated pyright tail, not the error-count line): - `sql.SQL(...) + Composed` is not typed as addable in psycopg's stubs. Build the query with `sql.Composed(list[Composable])` instead, which is both well-typed and the shape psycopg documents for assembling fragments. - `test_cost_ledger.py` calls the ledger through the `make_store` fixture, which is typed `Store` — and `CostLedger` is deliberately not on that Protocol, so every call is an unknown attribute. Widen with the file-level pragma the repo already uses for exactly this (tests/test_no_op_variant.py), with a comment naming why. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8jE6s96VcMT8MxxPvnxD3

Three review additions, all inside the same non-normative surface (no spec change was needed for any of them). Per-idea attribution. `CostEntry.idea_id` existed but only the executor populated it and the rollup had no per-idea bucket, so the ideation-efficiency question — which expensive ideas produced nothing — was unanswerable. The evaluator stamps it from `variant.idea_id`; the ideator stamps it when a dispatch produced exactly ONE idea, because a dispatch that emitted three spent one indivisible gateway call on all three and both picking one and splitting three ways would be inventions. Those entries stay at role/task level and `by_idea` reports how many it excluded. Landing it meant restructuring `handle_ideation_task` so the single record happens in a `finally` after `_persist_ideas` mints the ids: recording last risks losing a cost row on a crash, recording first risked losing the submission, which is worse. Per-model splits preserved. `modelUsage` was collapsed to one label and dropped entirely for multi-model runs. The whole map is now a `models` list on the entry — per-attempt cost stays per-attempt, the split is structure the read-time reduction slices into `by_model` — deliberately without `num_turns` / `duration_ms`, which belong to the attempt and whose per-model share would be fabricated. `by_model` also covers single-model attempts by synthesizing their one slice, which is exact rather than an allocation. Cache writes are additionally captured per TTL tier from `usage.cache_creation`, which is what makes correct cache pricing possible at all. Derived dollars, labelled as derived. `pricing.py` prices tokens against an operator-supplied rate table for attempts no provider priced. Rates are per token class per model (input / cache write 5m / cache write 1h / cache read / output) in USD per Mtok, because cache reads run ~10x cheaper than fresh input and 1h writes materially dearer than 5m — one blended rate can be wrong by a large multiple. Three properties are enforced, not just documented: nothing writes a computed figure into the ledger (derivation is read-time; `total_cost_usd` keeps meaning "the provider said so"), a table MUST carry `source` + `as_of` and both travel into the report so a figure is auditable against the rates that made it, and an unpriced class is a reported gap never a zero. The shipped template has every rate null so it prices nothing until filled in — no unsourced numbers are baked in. Two structural moves: the rollup got its own `rollup.py` (records -> pricing -> rollup is a clean three-layer stack; inlining pricing would have been an import cycle), and `render_table` split into per-section helpers at the length gate. Two bugs worth naming. `models` as a `tuple` was unbuildable from a JSON array under `strict=True` — it made every multi-model entry both unrecordable and un-POSTable. And a per-model row whose basis said `derived` showed $0 until `DerivedCost.per_model` credited each model its own tokens' cost; "derived, $0.0000" reads as "this model was free". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8jE6s96VcMT8MxxPvnxD3

/codex-review on this chunk never ran: Codex's bwrap sandbox cannot create a mount namespace in this container, so every command — including a read-only pwd — failed before execution and it read nothing. Eric elected to ship without the review. Committing the record rather than deleting it, for two reasons. The absence of a codex-review on a planless chunk should be explicit rather than left for a reviewer to wonder about. And the diagnosis is worth more than the failed round: bare bwrap fails the same way with the harness sandbox off, this shell shares PID 1's namespaces, the project is config-trusted, and there is no seccomp filter — so it is none of the usual suspects. It is the container's capability set (no CAP_SYS_ADMIN, so making / slave returns EPERM), and the leading hypothesis is that installing system bubblewrap flipped Codex off a bundled fallback that had been working. BLOCKED.md carries the verbatim errors, the eliminations, the fix options, and a two-second preflight the skill should adopt so a future session fails fast instead of burning a round that reads nothing. The brief (0.md) is complete and accurate; a retry needs only a working sandbox. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8jE6s96VcMT8MxxPvnxD3

Eric measured the premise and it does not hold: `which -a bwrap` returns /usr/bin/bwrap and /bin/bwrap (same inode), and there is no bwrap in ~/.local/bin at all — so nothing was shadowed, PATH order never mattered, and removing the bubblewrap install would fix nothing. That was the one action the hypothesis recommended, which is why this is a correction in place rather than a deletion. Two things the measurements replaced it with, both sharper: - "This pod cannot be sandboxed" was too strong. The kernel offers Landlock ABI v7 (errno 0), which needs no CAP_SYS_ADMIN. The accurate statement is that *Codex* can no longer reach the sandbox this pod can provide: `use_linux_sandbox_bwrap` is removed and `use_legacy_landlock` deprecated, and the decisive test on that escape hatch produced no file and reported the sandbox failing before execution. - Codex on this pod has been observed both failing honestly and burning 15,303 tokens, replying DONE, and running nothing. So a round's validity cannot be read off its prose: zero tool-execution events means void, not passing. Recorded as a postflight check alongside the existing preflight suggestion. The correction notes what the original inference got right (the string evidence) and what it skipped (testing the displacement claim before writing it down), and marks the replacement version-change explanation as inference rather than measurement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8jE6s96VcMT8MxxPvnxD3

ealt mentioned this pull request Jul 30, 2026

Cost instrumentation: persist per-role token/$ so runs can report what they cost #343

Open

ealt and others added 4 commits July 30, 2026 00:35

Navigation Menu

Cost instrumentation: per-role spend capture, the reference cost ledger, and a per-experiment rollup (#343) - #346

Cost instrumentation: per-role spend capture, the reference cost ledger, and a per-experiment rollup (#343)#346
ealt wants to merge 5 commits into
mainfrom
feat/cost-instrumentation

ealt commented Jul 30, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ealt commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Milestone coverage

The design decision worth reviewing

Follow-up round: the three design calls worth reviewing

Two refactors the complexity gate forced (both worth having)

What this does NOT cover

Fresh-operator walkthrough

Test plan

Related issues

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ealt commented Jul 30, 2026 •

edited

Loading