Cost instrumentation: per-role spend capture, the reference cost ledger, and a per-experiment rollup (#343)

AGENTS.md

@@ -59,6 +59,7 @@ At Phase 10 chunk 10d follow-up A, markdown linting, JSON Schema validation, and
 | `python3 -m eden_task_store_server --store-url :memory: --experiment-id exp-1 --experiment-config tests/fixtures/experiment/.eden/config.yaml --port 0` | Run the reference task-store-server (announces `EDEN_TASK_STORE_LISTENING host=… port=…` on stdout). `--store-url` accepts `:memory:`, `sqlite:///<path>`, `postgresql://…`, or a bare path. |
 | `python3 -m eden_orchestrator …` / `python3 -m eden_ideator_host …` / `python3 -m eden_executor_host …` / `python3 -m eden_evaluator_host …` / `python3 -m eden_web_ui …` | Run each reference service (see each service's `README.md` for full flag list). The web-ui announces `EDEN_WEB_UI_LISTENING host=… port=…` on stdout, mirroring the task-store-server convention. Pass `--repo-path <bare-repo>` to the web-ui to enable the executor module; omit it for an ideator+evaluator deployment. |
 | `python3 -m eden_service_common.repo_init --repo-path <dir>` | Idempotent bare-repo seed; emits `EDEN_REPO_SEEDED sha=<hex>` (or `EDEN_REPO_ALREADY_SEEDED`). Used by setup-experiment. |
+| `uv run python -m eden_service_common.cost_report --task-store-url <url> --experiment-id <id> [--price-table <path>] [--format table]` | Issue #343 per-experiment cost rollup: per-role / per-variant / per-idea / per-model spend from the reference cost ledger, joined against each variant's status + evaluation payload. JSON by default (machine contract); `--format table` for humans. `--price-table` derives dollars from token counts for attempts no provider priced (template: [`reference/pricing/price-table.example.json`](reference/pricing/price-table.example.json)) — derived figures are labelled, never blended into reported ones. Auth from `EDEN_ADMIN_TOKEN` / `EDEN_BEARER`, never argv. See [`docs/observability.md`](docs/observability.md) §2.10. |
 | `python3 scripts/spec-xref-check.py` | Validate every `§N.M` reference in `spec/v0/*.md` resolves to a real section heading in its target chapter. Run before committing a normative spec change. |
 | `python3 scripts/check-rename-discipline.py` | Fail if any of the legacy-vocab patterns enumerated at the top of the script (pre-rename role / artifact / kind names and intermediate verb-form survivors) appear outside the allowlist. Mirrors CI's `rename-discipline` job. Pass `--write-baseline` to dump all hits when extending the allowlist. |
 | `EDEN_TEST_POSTGRES_DSN=postgresql://… uv run pytest -q reference/packages/eden-storage/tests` | Run the parametrized backend conformance tests against a live Postgres (CI's `python-test-postgres` does this). Without the env var, postgres rows skip. |

CHANGELOG.md

@@ -8,6 +8,42 @@ Per-chunk entries preserve the full implementation record: contract amendments,
 
 ## [Unreleased]
 
+### Cost instrumentation: per-role spend capture, the reference cost ledger, and a per-experiment rollup (issue #343)
+
+Planless chunk (no roadmap chunk); all three milestones of [#343](https://github.com/ealt/eden/issues/343) — executor/evaluator capture + ledger, ideator capture, per-experiment rollup — in one PR because the second two are small deltas on the first's surface. EDEN had no token/dollar capture at all, so run economics could only be reported as duration / throughput / counts — the R2 report's economics section had to hand-estimate ~$35–45, and the live R3 run has the same blindness. The cheapest available win was that the executor's Claude Code agent logs **already** emit `total_cost_usd` per attempt and the platform dropped it on the floor. This chunk stops dropping it.
+
+**Where cost lives, and why it is not on the `Variant`.** `spec/v0` has no home for spend: the chapter-3 submission shapes carry no cost field, the chapter-2 `Variant` record has no cost property, and the chapter-5 event registry is closed at v0. Verified empirically rather than assumed (#343 flagged the eval-payload exact-key-match constraint as the known obstacle and asked whether it also binds the execution path): on the **evaluation** path an undeclared key is rejected loudly (`InvalidPrecondition: evaluation key 'total_cost_usd' is not in the experiment's evaluation_schema`), while on the **execution** path there is no key validation *and* no free-form field — `submission_from_payload` reads named keys only, so an extra `cost` key is **silently dropped**. Different mechanisms, same conclusion: there is no smuggling route, and inventing a normative field would mean amending `spec/v0` MUSTs + conformance, which this chunk deliberately does not do (the scoped spec-change plan is commented on #343 instead).
+
+So the reference impl keeps its own **non-normative ledger** behind the `/_reference/` extension surface chapter 7 §5 sanctions. New `CostLedger` protocol ([`cost.py`](reference/packages/eden-storage/src/eden_storage/cost.py), [`_ops/cost.py`](reference/packages/eden-storage/src/eden_storage/_ops/cost.py)) with `record_cost` / `list_cost_entries`, a `cost_entry` table on all three backends (SQLite v10 + the mirroring Postgres migration; Postgres primitives in the [`_postgres_cost.py`](reference/packages/eden-storage/src/eden_storage/_postgres_cost.py) sibling), and `POST` / `GET /_reference/experiments/{E}/cost` with a matching `StoreClient` pair so a subprocess-mode host writes cost identically in-process or across the wire. `CostLedger` is kept **off** the `Store` protocol on purpose (the `ArtifactStore` precedent): a reference extension does not belong in the structural interface a conforming implementation is measured against, and the routers cast exactly like the §16 artifact router does.
+
+**Two ledger properties are load-bearing.** (1) **First-write-wins on `entry_id`** — a host that re-records after a transport failure must not double the reported spend; the caller owns the key and it is per-*attempt*, not per-task, because a reclaimed-and-rerun task really did spend twice (executor keys on its freshly-minted `variant_id`, evaluator on `(task_id, variant_id)`). (2) **Attribution, not aggregation** — one row per spend event carrying role / task / variant / idea, so every rollup is a read-time reduction. Cost rows carry **no event**: like an artifact-metadata row they are not bound to any task/idea/variant transition, so the chapter-5 §2 transactional invariant has nothing to pair them with (and the v0 registry is closed).
+
+**Capture path.** A worker host never talks to an LLM itself, so the user's `*_command` reports via two OPTIONAL keys on the outcome JSON the host already parses ([worker-host binding](spec/v0/reference-bindings/worker-host-subprocess.md) §11, informative): `agent_log` (a path to a Claude Code `--output-format stream-json` log — the host parses the last `{"type": "result"}` record) or `cost` (already-normalized figures, for non-Claude providers; wins when both are present). For the R3-shaped experiment, adopting this is **one added key naming a log file it already writes**. Shared extraction in [`agent_cost.py`](reference/services/_common/src/eden_service_common/agent_cost.py) drives both the executor and evaluator hosts through one `record_outcome_cost` helper rather than a per-host copy.
+
+Three behaviors the parser gets right on purpose: it reads the **aggregate** `usage` totals from the terminal `result` record (not per-turn sums, which disagree); it tolerates everything a real log throws at it — interleaved stderr (the reference `execution.py` merges the two streams), a partial final line from a SIGKILLed agent, non-JSON hook noise, and a head past the 8 MiB tail cap; and **every** failure mode is a no-op rather than an error, including an unreachable ledger — cost is bookkeeping *about* an attempt, so a malformed log must never fail an otherwise-good variant. Recording runs inside the per-task worktree's lifetime (a relative `agent_log` resolves against it) and **regardless of how the attempt terminalizes** — money spent on a variant that errored is still money spent, and the ledger says so.
+
+**Tests.** The happy path runs against a real captured stream-json log from an eden-experiments belief-state-recovery execution task, reduced to one line per record type with prose / session ids / hook output redacted and every `result` number verbatim — so the assertions pin the real field names rather than an invented shape. Ledger semantics (idempotency, cross-backend read order, no-events, experiment-id mismatch) are parametrized across all three backends; the wire round-trip covers filters, `exclude_none` shaping, and the self-gated bearer auth on the two `/_reference/` cost routes (the auth middleware skips `/_reference/`, and one of these routes writes). Host-level tests drive the real `_handle_one` — including the relative-`agent_log` case, which is the only way to catch a regression that moved extraction after worktree cleanup.
+
+**Refactors the size gate forced (both worth having).** `_execute_and_validate` crossed the 100-line function threshold, so phases 2d–2e split into `_validated_commit_from_outcome`; `postgres.py` crossed 800 SLOC, so the ledger primitives moved to the `_postgres_cost.py` sibling (the `_postgres_schema.py` / `_postgres_views.py` precedent) — which also puts the whole non-normative extension behind one clearly-labeled file. No `# slop-allow` annotations added.
+
+**Ideator capture (milestone 2).** The ideator's spend arrives on the JSON-line terminator instead of an outcome file — the same two keys, honored on `ideation-error` as well as `ideation-done`, because a failed ideation attempt still burned gateway tokens. Its attempt key is a per-dispatch **nonce** rather than a deterministic id: unlike the executor's `variant_id`, an ideation dispatch has no stable per-attempt identifier, and a task re-dispatched after a reclaim really did spend twice; nothing retries the record call, so one-row-per-dispatch holds by construction rather than by key. `IdeatorSubprocess` gained a read-only `cwd` property so a relative `agent_log` resolves the way the subprocess wrote it.
+
+The bridge half of milestone 2 is not in this repo: the OpenClaw gateway response the R3 ideator drives is read by `fraxl-ideator.py` in eden-experiments, and whether that response populates an OpenAI-style `usage` object could not be determined from this environment (the gateway runs on the experiment box, which is off-limits while R3 is live). The companion eden-experiments PR normalizes `usage` when present and — per #343's "document what it carries, don't estimate" constraint — logs the response's actual top-level keys once when it is absent, so the next run's log answers the question definitively instead of a heuristic guessing at it.
+
+**Rollup (milestone 3).** `summarize(experiment_id, entries)` is a pure read-time reduction into per-role and per-variant totals — deliberately **not** a stored aggregate or a server-side summary endpoint, so it cannot drift from the ledger it reads, and one implementation serves both an in-process consumer and one reading over the wire. `CostTotals` carries `entries_missing_cost_usd` so a token-only entry can't make a partial total read as a complete one. `python3 -m eden_service_common.cost_report` prints the rollup joined against each variant's status + evaluation payload (JSON by default — it exists to feed analysis; `--format table` for humans), which is what turns DCI-per-dollar into a local computation instead of a two-source join every consumer rewrites. Auth comes from `EDEN_ADMIN_TOKEN` / `EDEN_BEARER`, never argv. Two accounting properties are asserted rather than assumed: ideation spend (no `variant_id`) counts in `totals` + `by_role` while appearing in no `by_variant` bucket, and spend attributed to a variant the store no longer has is still reported (`status: null`) rather than dropped — understating a run's cost is the one failure this report must not have. Operator-facing docs at [`docs/observability.md`](docs/observability.md) §2.10, including the gaps: labeled-incomplete totals, deadline-killed attempts under-reporting (a killed agent's log has no terminal `result` record), no cost in checkpoints, and inference-only scope.
+
+**Follow-up round (per-idea attribution, per-model splits, derived dollars).** Three additions after review, all inside the same non-normative surface:
+
+- **Per-idea attribution.** `CostEntry.idea_id` existed but only the executor populated it, and the rollup had no per-idea bucket — so "which ideas cost what", the ideation-efficiency question, was unanswerable. The evaluator now stamps it from `variant.idea_id`; the ideator stamps it when a dispatch produced **exactly one** idea (a dispatch that emitted three spent one indivisible gateway call on all three, so picking one or splitting three ways would both be inventions — those entries stay at role/task level and the rollup counts them as unattributed). Landing that meant restructuring `handle_ideation_task` so the single cost-record happens in a `finally` **after** `_persist_ideas` mints the ids: recording last means a crash between submit and record loses a cost row, where recording first would lose the *submission*. The report's `by_idea` section pairs each idea's total with the variants it produced, so an expensive idea that produced nothing is a row with a cost and an empty variant list.
+- **Per-model token splits preserved.** `modelUsage` was being collapsed to a single label and **dropped entirely** when a run spanned models. The whole map is now kept as a `models` list on the entry (per-attempt cost stays per-attempt; the split is structure the reduction slices into `by_model`), deliberately without `num_turns` / `duration_ms` — those belong to the attempt and a per-model share would be fabricated. Cache writes are additionally captured **per TTL tier** from `usage.cache_creation` (the real capture carries `ephemeral_5m` / `ephemeral_1h`), which is what makes correct cache pricing possible at all. One bug worth naming: `models` was first typed as a `tuple`, which `strict=True` refuses to build from a JSON array — it made every multi-model entry both unrecordable *and* un-POSTable until it became a `list`.
+- **Derived dollars, labelled as derived.** [`pricing.py`](reference/packages/eden-storage/src/eden_storage/pricing.py) prices token counts against an operator-supplied rate table (`--price-table`; template at [`reference/pricing/price-table.example.json`](reference/pricing/price-table.example.json)) for attempts no provider priced. Rates are **per token class per model** — fresh input, cache write at 5m, cache write at 1h, cache read — in USD per million tokens, because cache reads run ~an order of magnitude cheaper than fresh input and 1h writes materially dearer than 5m, so one blended rate can be wrong by a large multiple. Three properties are enforced rather than documented: nothing writes a computed figure into the ledger (`total_cost_usd` keeps meaning "the provider said so"; derivation is read-time), a table MUST carry `source` + `as_of` and both travel into the report so a figure is auditable against the rates that made it, and an unpriced class is a **reported gap, never a zero** — including cache writes whose TTL tier was never reported, and every rate in the shipped template, which is null on purpose so the template prices nothing until filled in. Buckets carry a `basis` (`reported` / `derived` / `mixed` / `unpriced`), the table render says `DERIVED` in words, and `entries_unpriced` makes a total a stated floor.
+
+The rollup moved to its own [`rollup.py`](reference/packages/eden-storage/src/eden_storage/rollup.py) (cost records → pricing → rollup is now a clean three-layer stack; the alternative was a cost↔pricing import cycle), and `render_table` split into per-section helpers when it crossed the length gate.
+
+**Deferrals.** Checkpoint coverage — cost rows are **not** in checkpoint export/import, because the chapter-10 archive layout is normative and extending it is spec surgery → [#344](https://github.com/ealt/eden/issues/344). A `cost_entry_unpacked` Postgres convenience view for `EDEN_READONLY_STORE_URL` analysis consumers → [#345](https://github.com/ealt/eden/issues/345). A **normative home** for cost (a `Variant` field or first-class record, which would also make it round-trip through checkpoints and be assertable by conformance) is scoped as a comment on #343 rather than done here — the balloon-guard on this work was explicitly "implement within current spec, propose the spec change". AWS cost-allocation tags and an orchestrator budget cap stay **propose-only** on #343: the first needs AWS permissions this work does not have, and the second is a policy surface (chapter 3 §6 decision types + a termination-policy-shaped config block), not a trivial fall-out of the accounting.
+
+**Validation.** `ruff` / `pyright` / full `pytest` / markdownlint / `spec-xref-check` / `check-rename-discipline` / `check-complexity` all green locally. **Not** run locally: the Postgres-backed rows (no server available in this environment — the `postgres` parametrizations skip; a server-free MRO guard covers the one structural risk the extraction introduced) and the Compose / Helm smokes (no Docker daemon available). Both are CI-covered on the PR.
+
 ### Repo review: docs refresh, architecture doc, ground-up design review, ideator submit-readback fix
 
 A review-and-cleanup pass (no roadmap chunk). Four parts: