feat(sdk): add per-call prompt token composition metrics

.pr/evidence/analysis.md

@@ -0,0 +1,61 @@
+# Evidence: per-call prompt composition on live agent runs
+
+> `.pr/` is PR-only reviewer context per repository policy (`.github/workflows/pr-artifacts.yml`);
+> this directory is removed after PR approval. Nothing here is meant to merge to `main`.
+
+Per-call records from live runs of the default agent (`get_default_agent`, 19 tools,
+browser-on) on tasks from the published harness-benchmark short suite
+(<https://github.com/rajshah4/harness-benchmark>). Each `calls.jsonl` row carries the
+`PromptComposition` estimate and the provider-reported `TokenUsage` for one LLM call
+(counts and response ids only; no message content, no credentials).
+
+## Lanes
+
+- `minimax-m3/` — `openai/MiniMax-M3` (chat completions; litellm has no tokenizer mapping for
+  this model, so estimates use litellm's fallback tokenizer). Tasks p09-task-01 (trivial
+  rename, 5 calls), p09-task-07 (medium refactor, 26 calls), p09-task-10 (hard cache
+  implementation, 10 calls).
+- `gpt-4o-mini/` — `gpt-4o-mini` (chat completions; litellm maps this model to its real
+  `o200k_base` tokenizer — verified by comparing `token_counter` framing overhead against
+  `tiktoken.get_encoding("o200k_base")` vs `cl100k_base` on divergent inputs). Tasks
+  p09-task-01 (9 calls, verifier PASS) and p09-task-07 (28 calls, verifier FAIL — the model
+  left a syntax error in the repo; a completed run, labeled a model-quality failure).
+
+## gpt-4o-mini summary (mapped tokenizer)
+
+| task | calls | avg system | avg tools | avg history | avg latest | avg provider input | est/provider median |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| p09-task-01 | 9 | 3,340 | 5,702 | 987 | 175 | 10,231 | 1.00 |
+| p09-task-07 | 28 | 3,340 | 5,702 | 18,357 | 792 | 28,739 | 0.98 |
+
+- Per-call est/provider ratio band: 0.99–1.00 (task-01), 0.97–0.99 (task-07) — versus
+  0.85–0.97 on the unmapped MiniMax-M3 lane. The mapped-tokenizer band tightens to ~1.0, so
+  the MiniMax underestimate was dominated by the fallback tokenizer; a residual ~1–3%
+  underestimate remains, consistent with uncounted request framing and litellm's tool
+  serialization convention.
+- One notable event: between calls 9 and 10 of task-07, history dropped 30.8K → 10.0K tokens
+  with **no intervening LLM call** — view-property enforcement
+  (`View.enforce_properties`: batch/observation/atomicity properties drop events without a
+  summarization call), not an LLM-summarizing condensation. The composition tracked the
+  shrunken view exactly (ratio stayed 0.99 across the drop), which is itself evidence the
+  history bucket measures the payload actually sent.
+- Total spend for this lane (litellm-computed `accumulated_cost`, PAYG key): $0.0081
+  (task-01) + $0.0808 (task-07) ≈ **$0.089**; 87% of prompt tokens were cache reads.
+
+
+## MiniMax-M3 summary
+
+| task | calls | avg system | avg tools | avg history | avg latest | avg provider input | est/provider median |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| p09-task-01 | 5 | 3,331 | 5,749 | 533 | 205 | 11,478 | 0.85 |
+| p09-task-07 | 26 | 3,331 | 5,749 | 10,462 | 381 | 21,176 | 0.94 |
+| p09-task-10 | 10 | 3,331 | 5,749 | 2,605 | 215 | 13,506 | 0.86 |
+
+- The standing preamble (system + tool schemas ≈ 9,080 estimated tokens) is constant per call:
+  ~79% of the average call on the trivial task (avg provider input 11,478), ~84% of first calls.
+- Re-sent tool schemas totaled 149,474 estimated tokens on the 26-call task — 27% of that run's
+  provider-reported input (550,568).
+- Estimated component sum per call runs 0.85–0.97× the provider-reported `prompt_tokens`,
+  rising as history grows. The residual gap is consistent with request framing and tokenizer
+  mapping differences; the component split is the finding, not the absolute sum.
+- MiniMax's automatic prefix caching served 79–94% of prompt tokens as cache reads.

.pr/evidence/gpt-4o-mini/p09-task-01-calls.jsonl

@@ -0,0 +1,9 @@
+{"seq": 0, "elapsed_s": 3.12, "usage": {"model": "gpt-4o-mini", "prompt_tokens": 9172, "completion_tokens": 38, "cache_read_tokens": 0, "cache_write_tokens": 0, "reasoning_tokens": 0, "context_window": 128000, "per_turn_token": 9210, "response_id": "chatcmpl-EGVEG8NSZIEywDIyNQnqt99X5R4tJ"}, "composition": {"model": "gpt-4o-mini", "system_prompt_tokens": 3340, "tool_tokens": 5702, "history_tokens": 0, "latest_message_tokens": 38, "is_estimate": true, "response_id": "chatcmpl-EGVEG8NSZIEywDIyNQnqt99X5R4tJ"}, "latency_s": 2.5493357181549072}
+{"seq": 1, "elapsed_s": 4.67, "usage": {"model": "gpt-4o-mini", "prompt_tokens": 9675, "completion_tokens": 63, "cache_read_tokens": 9088, "cache_write_tokens": 0, "reasoning_tokens": 0, "context_window": 128000, "per_turn_token": 9738, "response_id": "chatcmpl-EGVEICEloV8BjwzBVpfYkPDKSNhVb"}, "composition": {"model": "gpt-4o-mini", "system_prompt_tokens": 3340, "tool_tokens": 5702, "history_tokens": 73, "latest_message_tokens": 486, "is_estimate": true, "response_id": "chatcmpl-EGVEICEloV8BjwzBVpfYkPDKSNhVb"}, "latency_s": 1.5301110744476318}
+{"seq": 2, "elapsed_s": 7.12, "usage": {"model": "gpt-4o-mini", "prompt_tokens": 9797, "completion_tokens": 61, "cache_read_tokens": 9600, "cache_write_tokens": 0, "reasoning_tokens": 0, "context_window": 128000, "per_turn_token": 9858, "response_id": "chatcmpl-EGVEJHP3YvqiBxIXWUbhpgqhTOhE7"}, "composition": {"model": "gpt-4o-mini", "system_prompt_tokens": 3340, "tool_tokens": 5702, "history_tokens": 620, "latest_message_tokens": 77, "is_estimate": true, "response_id": "chatcmpl-EGVEJHP3YvqiBxIXWUbhpgqhTOhE7"}, "latency_s": 2.4455978870391846}
+{"seq": 3, "elapsed_s": 8.73, "usage": {"model": "gpt-4o-mini", "prompt_tokens": 10054, "completion_tokens": 63, "cache_read_tokens": 9728, "cache_write_tokens": 0, "reasoning_tokens": 0, "context_window": 128000, "per_turn_token": 10117, "response_id": "chatcmpl-EGVEMHTRF4V3DUDvosBEZHNMSya0m"}, "composition": {"model": "gpt-4o-mini", "system_prompt_tokens": 3340, "tool_tokens": 5702, "history_tokens": 755, "latest_message_tokens": 213, "is_estimate": true, "response_id": "chatcmpl-EGVEMHTRF4V3DUDvosBEZHNMSya0m"}, "latency_s": 1.593902826309204}
+{"seq": 4, "elapsed_s": 10.05, "usage": {"model": "gpt-4o-mini", "prompt_tokens": 10309, "completion_tokens": 63, "cache_read_tokens": 9984, "cache_write_tokens": 0, "reasoning_tokens": 0, "context_window": 128000, "per_turn_token": 10372, "response_id": "chatcmpl-EGVENFynBkjyDGNnJZvpU8I5yjIcr"}, "composition": {"model": "gpt-4o-mini", "system_prompt_tokens": 3340, "tool_tokens": 5702, "history_tokens": 1028, "latest_message_tokens": 212, "is_estimate": true, "response_id": "chatcmpl-EGVENFynBkjyDGNnJZvpU8I5yjIcr"}, "latency_s": 1.288510799407959}
+{"seq": 5, "elapsed_s": 10.95, "usage": {"model": "gpt-4o-mini", "prompt_tokens": 10539, "completion_tokens": 32, "cache_read_tokens": 10240, "cache_write_tokens": 0, "reasoning_tokens": 0, "context_window": 128000, "per_turn_token": 10571, "response_id": "chatcmpl-EGVEO5r1bOirNFiH5qiMMXW33T1og"}, "composition": {"model": "gpt-4o-mini", "system_prompt_tokens": 3340, "tool_tokens": 5702, "history_tokens": 1301, "latest_message_tokens": 187, "is_estimate": true, "response_id": "chatcmpl-EGVEO5r1bOirNFiH5qiMMXW33T1og"}, "latency_s": 0.8807761669158936}
+{"seq": 6, "elapsed_s": 13.23, "usage": {"model": "gpt-4o-mini", "prompt_tokens": 10669, "completion_tokens": 29, "cache_read_tokens": 10496, "cache_write_tokens": 0, "reasoning_tokens": 0, "context_window": 128000, "per_turn_token": 10698, "response_id": "chatcmpl-EGVEQBtKdhnHab1gSiwzkourerokO"}, "composition": {"model": "gpt-4o-mini", "system_prompt_tokens": 3340, "tool_tokens": 5702, "history_tokens": 1517, "latest_message_tokens": 118, "is_estimate": true, "response_id": "chatcmpl-EGVEQBtKdhnHab1gSiwzkourerokO"}, "latency_s": 1.3414945602416992}
+{"seq": 7, "elapsed_s": 15.53, "usage": {"model": "gpt-4o-mini", "prompt_tokens": 10787, "completion_tokens": 97, "cache_read_tokens": 10624, "cache_write_tokens": 0, "reasoning_tokens": 0, "context_window": 128000, "per_turn_token": 10884, "response_id": "chatcmpl-EGVESk7f9F0Vx4trTv6Cr3B7g2hM8"}, "composition": {"model": "gpt-4o-mini", "system_prompt_tokens": 3340, "tool_tokens": 5702, "history_tokens": 1661, "latest_message_tokens": 110, "is_estimate": true, "response_id": "chatcmpl-EGVESk7f9F0Vx4trTv6Cr3B7g2hM8"}, "latency_s": 1.62056303024292}
+{"seq": 8, "elapsed_s": 18.22, "usage": {"model": "gpt-4o-mini", "prompt_tokens": 11074, "completion_tokens": 57, "cache_read_tokens": 10752, "cache_write_tokens": 0, "reasoning_tokens": 0, "context_window": 128000, "per_turn_token": 11131, "response_id": "chatcmpl-EGVEVrPlyWeD92ZjuW892YFNWzZvj"}, "composition": {"model": "gpt-4o-mini", "system_prompt_tokens": 3340, "tool_tokens": 5702, "history_tokens": 1929, "latest_message_tokens": 135, "is_estimate": true, "response_id": "chatcmpl-EGVEVrPlyWeD92ZjuW892YFNWzZvj"}, "latency_s": 1.2971453666687012}

.pr/evidence/gpt-4o-mini/p09-task-01-summary.json

@@ -0,0 +1,28 @@
+{
+  "task_id": "p09-task-01",
+  "prompt": "Rename the local variable `raw_items` to `source_items` in `toyapp/pagination.py`. Keep behavior unchanged and do not edit unrelated files.",
+  "model": "gpt-4o-mini",
+  "status": "completed",
+  "conversation_status": "ConversationExecutionStatus.FINISHED",
+  "wall_clock_s": 18.22,
+  "max_iterations_cap": 30,
+  "calls_recorded": 9,
+  "condenser_shares_metrics": true,
+  "metrics_snapshot": {
+    "model_name": "gpt-4o-mini",
+    "accumulated_cost": 0.0080748,
+    "max_budget_per_task": null,
+    "accumulated_token_usage": {
+      "model": "gpt-4o-mini",
+      "prompt_tokens": 92076,
+      "completion_tokens": 503,
+      "cache_read_tokens": 80512,
+      "cache_write_tokens": 0,
+      "reasoning_tokens": 0,
+      "context_window": 128000,
+      "per_turn_token": 11131,
+      "response_id": ""
+    }
+  },
+  "error": null
+}