From 991a59cf66ef7f52140f89ef5cd9f0412a59ad67 Mon Sep 17 00:00:00 2001 From: george larson Date: Mon, 24 Aug 2026 14:53:22 -0400 Subject: [PATCH 1/5] docs(sdk): document per-call prompt token composition metrics Companion to the PromptComposition feature on software-agent-sdk feat/llm-prompt-composition-metrics: per-call decomposition of prompt tokens into system prompt, tool schemas, conversation history, and latest message, recorded in LLM Metrics. Co-authored-by: openhands --- docs.json | 3 +- sdk/guides/llm-prompt-composition.mdx | 71 +++++++++++++++++++++++++++ sdk/guides/metrics.mdx | 2 + 3 files changed, 75 insertions(+), 1 deletion(-) create mode 100644 sdk/guides/llm-prompt-composition.mdx diff --git a/docs.json b/docs.json index 41479408b..d06686b2e 100644 --- a/docs.json +++ b/docs.json @@ -361,7 +361,8 @@ "sdk/guides/llm-image-input", "sdk/guides/llm-error-handling", "sdk/guides/llm-fallback", - "sdk/guides/llm-profile-store" + "sdk/guides/llm-profile-store", + "sdk/guides/llm-prompt-composition" ] }, { diff --git a/sdk/guides/llm-prompt-composition.mdx b/sdk/guides/llm-prompt-composition.mdx new file mode 100644 index 000000000..11562daf2 --- /dev/null +++ b/sdk/guides/llm-prompt-composition.mdx @@ -0,0 +1,71 @@ +--- +title: Prompt Token Composition +description: Break down prompt tokens per LLM call into system prompt, tool schemas, conversation history, and the latest message. +--- + +## Overview + +Every LLM call records a per-call decomposition of its prompt tokens, so you can see where the input budget goes on each step of an agent run: + +- `system_prompt_tokens` - Tokens in system messages +- `tool_tokens` - Tokens in the tool schemas included with the call +- `history_tokens` - Tokens in conversation history (all non-system messages except the latest one) +- `latest_message_tokens` - Tokens in the latest observation or user message + +The decomposition is computed automatically on every call - both the Chat Completions and Responses API paths - and recorded into the LLM's [Metrics](/sdk/guides/metrics). No configuration is required. + +## Accessing the Composition + +Each call appends one `PromptComposition` record to `llm.metrics.prompt_compositions`. The most recent record is available as `llm.metrics.latest_prompt_composition`: + +```python icon="python" +conversation.run() + +composition = llm.metrics.latest_prompt_composition +assert composition is not None +print(f"System prompt: {composition.system_prompt_tokens}") +print(f"Tool schemas: {composition.tool_tokens}") +print(f"History: {composition.history_tokens}") +print(f"Latest message: {composition.latest_message_tokens}") +``` + +Each record carries the `response_id` of its call, so you can join it with the provider-reported record in `llm.metrics.token_usages`: + +```python icon="python" +for composition, usage in zip( + llm.metrics.prompt_compositions, llm.metrics.token_usages +): + estimated = ( + composition.system_prompt_tokens + + composition.tool_tokens + + composition.history_tokens + + composition.latest_message_tokens + ) + print( + f"{composition.response_id}: estimated {estimated}, " + f"provider reported {usage.prompt_tokens}" + ) +``` + +Agent steps always include the agent's tool list, so calls from the primary agent loop have `tool_tokens > 0`. Auxiliary calls that pass no tools - for example the [context condenser](/sdk/guides/context-condenser) or title generation - are recorded with `tool_tokens == 0`. + +## Estimates vs Provider-Reported Usage + +Composition counts are **client-side estimates**, computed with the model's tokenizer before the request is sent. The provider-reported `TokenUsage` remains the authoritative accounting: + +- `is_estimate` is `True` on records produced by the client-side estimator. +- Each component is counted independently, so per-message framing overhead is included in every component and the components do not necessarily sum exactly to the provider-reported `prompt_tokens`. +- When no tokenizer is available for a model, composition recording is skipped for that call; the call itself is unaffected. + + + Treat the composition as a breakdown of *where* prompt tokens go, and `token_usages` as the record of *how many* tokens the provider billed. + + +## Performance + +The estimator adds roughly 10-20 ms per call on a typical agent payload (measured with the default 19-tool agent on a ~27K-token prompt) - negligible next to network latency - so it is always on. + +## Next Steps + +- **[Metrics Tracking](/sdk/guides/metrics)** - Token usage, costs, and latency metrics for your agents +- **[Context Condenser](/sdk/guides/context-condenser)** - How OpenHands keeps history within the context window diff --git a/sdk/guides/metrics.mdx b/sdk/guides/metrics.mdx index 023306602..e4035318d 100644 --- a/sdk/guides/metrics.mdx +++ b/sdk/guides/metrics.mdx @@ -41,6 +41,7 @@ The `llm.metrics` object is an instance of the [Metrics class](https://github.co - `costs` - List of individual cost records per API call - `token_usages` - List of detailed token usage records per API call - `response_latencies` - List of response latency metrics per API call +- `prompt_compositions` - List of per-call [prompt token composition](/sdk/guides/llm-prompt-composition) estimates (system prompt, tool schemas, history, latest message) For more details on the available metrics and methods, refer to the [source code](https://github.com/OpenHands/software-agent-sdk/blob/main/openhands-sdk/openhands/sdk/llm/utils/metrics.py). @@ -409,5 +410,6 @@ for usage_id, metrics in conversation.conversation_stats.usage_to_metrics.items( ## Next Steps +- **[Prompt Token Composition](/sdk/guides/llm-prompt-composition)** - Break down prompt tokens per call into system prompt, tool schemas, history, and latest message - **[Context Condenser](/sdk/guides/context-condenser)** - Learn about context management and how it uses separate LLMs - **[LLM Routing](/sdk/guides/llm-routing)** - Optimize costs with smart routing between different models From 99bcd6e354c4d13dbe12ed29fa3a73b5eee1d4d2 Mon Sep 17 00:00:00 2001 From: george larson Date: Mon, 24 Aug 2026 16:02:33 -0400 Subject: [PATCH 2/5] docs(sdk): fix prompt composition join example and estimator caveats Address review findings: join compositions to token usage by response_id instead of positionally, qualify tool_tokens > 0 as native-FC agent steps (mock-tools renders schemas into prompt text), note litellm's tool serialization convention and fallback tokenizer, and document the linear-in-prompt-size counting cost. Co-authored-by: openhands --- sdk/guides/llm-prompt-composition.mdx | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/sdk/guides/llm-prompt-composition.mdx b/sdk/guides/llm-prompt-composition.mdx index 11562daf2..2203632c2 100644 --- a/sdk/guides/llm-prompt-composition.mdx +++ b/sdk/guides/llm-prompt-composition.mdx @@ -12,7 +12,7 @@ Every LLM call records a per-call decomposition of its prompt tokens, so you can - `history_tokens` - Tokens in conversation history (all non-system messages except the latest one) - `latest_message_tokens` - Tokens in the latest observation or user message -The decomposition is computed automatically on every call - both the Chat Completions and Responses API paths - and recorded into the LLM's [Metrics](/sdk/guides/metrics). No configuration is required. +The decomposition is computed automatically on every call - both the Chat Completions and Responses API paths - and recorded into the LLM's [Metrics](/sdk/guides/metrics). No configuration is required. On the Responses API path, the decomposition is computed on the finalized payload (instructions plus input items), so the record reflects what the provider received. ## Accessing the Composition @@ -29,12 +29,15 @@ print(f"History: {composition.history_tokens}") print(f"Latest message: {composition.latest_message_tokens}") ``` -Each record carries the `response_id` of its call, so you can join it with the provider-reported record in `llm.metrics.token_usages`: +Each record carries the `response_id` of its call. Join records with the provider-reported usage in `llm.metrics.token_usages` by `response_id` rather than by position - the two lists can diverge when a composition is skipped or a response carries no usage: ```python icon="python" -for composition, usage in zip( - llm.metrics.prompt_compositions, llm.metrics.token_usages -): +usage_by_id = {u.response_id: u for u in llm.metrics.token_usages} + +for composition in llm.metrics.prompt_compositions: + usage = usage_by_id.get(composition.response_id) + if usage is None: + continue estimated = ( composition.system_prompt_tokens + composition.tool_tokens @@ -47,7 +50,11 @@ for composition, usage in zip( ) ``` -Agent steps always include the agent's tool list, so calls from the primary agent loop have `tool_tokens > 0`. Auxiliary calls that pass no tools - for example the [context condenser](/sdk/guides/context-condenser) or title generation - are recorded with `tool_tokens == 0`. +Agent steps that send tools as native function-calling schemas have `tool_tokens > 0`. Auxiliary calls that pass no tools - for example the [context condenser](/sdk/guides/context-condenser) or title generation - are recorded with `tool_tokens == 0`. + + + On models without native function calling, the SDK renders tool schemas into the prompt text instead of sending them as tool parameters. Those agent steps are recorded with `tool_tokens == 0` and the schema tokens appear in the message buckets instead. + ## Estimates vs Provider-Reported Usage @@ -55,7 +62,9 @@ Composition counts are **client-side estimates**, computed with the model's toke - `is_estimate` is `True` on records produced by the client-side estimator. - Each component is counted independently, so per-message framing overhead is included in every component and the components do not necessarily sum exactly to the provider-reported `prompt_tokens`. -- When no tokenizer is available for a model, composition recording is skipped for that call; the call itself is unaffected. +- Tool schema counts follow litellm's `token_counter` serialization convention for tools, which can differ from the provider's wire-format tokenization. +- For models litellm has no tokenizer mapping for, counts use litellm's fallback tokenizer and may deviate more from the provider's counts. +- When token counting fails or is disabled (for example `litellm.disable_token_counter`), composition recording is skipped for that call; the call itself is unaffected. Treat the composition as a breakdown of *where* prompt tokens go, and `token_usages` as the record of *how many* tokens the provider billed. @@ -63,7 +72,7 @@ Composition counts are **client-side estimates**, computed with the model's toke ## Performance -The estimator adds roughly 10-20 ms per call on a typical agent payload (measured with the default 19-tool agent on a ~27K-token prompt) - negligible next to network latency - so it is always on. +The estimator's cost scales linearly with prompt size: roughly 10-20 ms per call on a typical agent payload, measured at ~31 ms for a ~100K-token prompt and ~61 ms for ~190K tokens (19 tools, gpt-4o tokenizer) - negligible next to network latency, so it is always on. ## Next Steps From 3784e9582fd43f9bb78edc4d19456dbc896b88f3 Mon Sep 17 00:00:00 2001 From: george larson Date: Mon, 24 Aug 2026 17:25:40 -0400 Subject: [PATCH 3/5] docs(sdk): note subscription-mode bucket behavior for prompt composition Mirrors the SDK docstring caveat (OpenHands/software-agent-sdk#4623): subscription mode folds the system prompt into the first user message, so those tokens count as history/latest on that transport. Co-authored-by: openhands --- sdk/guides/llm-prompt-composition.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/sdk/guides/llm-prompt-composition.mdx b/sdk/guides/llm-prompt-composition.mdx index 2203632c2..063177b3c 100644 --- a/sdk/guides/llm-prompt-composition.mdx +++ b/sdk/guides/llm-prompt-composition.mdx @@ -65,6 +65,7 @@ Composition counts are **client-side estimates**, computed with the model's toke - Tool schema counts follow litellm's `token_counter` serialization convention for tools, which can differ from the provider's wire-format tokenization. - For models litellm has no tokenizer mapping for, counts use litellm's fallback tokenizer and may deviate more from the provider's counts. - When token counting fails or is disabled (for example `litellm.disable_token_counter`), composition recording is skipped for that call; the call itself is unaffected. +- Buckets follow the wire, not the logical prompt: in subscription mode the system prompt is folded into the first user message before transport, so those tokens are counted in `history_tokens` or `latest_message_tokens` rather than `system_prompt_tokens` on that path. Treat the composition as a breakdown of *where* prompt tokens go, and `token_usages` as the record of *how many* tokens the provider billed. From d62f87ec426f3a935b679d4ef4a62043dbf0257a Mon Sep 17 00:00:00 2001 From: george larson Date: Tue, 25 Aug 2026 03:38:22 -0400 Subject: [PATCH 4/5] docs(sdk): make prompt composition opt-in via enable_prompt_composition Address rajshah4's review on software-agent-sdk#4623: composition recording is now opt-in (default off), so the page leads with enabling the flag and examples show it; the metrics field list notes the gate. Co-authored-by: openhands --- sdk/guides/llm-prompt-composition.mdx | 16 ++++++++++++---- sdk/guides/metrics.mdx | 2 +- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/sdk/guides/llm-prompt-composition.mdx b/sdk/guides/llm-prompt-composition.mdx index 063177b3c..81dbc717e 100644 --- a/sdk/guides/llm-prompt-composition.mdx +++ b/sdk/guides/llm-prompt-composition.mdx @@ -5,18 +5,26 @@ description: Break down prompt tokens per LLM call into system prompt, tool sche ## Overview -Every LLM call records a per-call decomposition of its prompt tokens, so you can see where the input budget goes on each step of an agent run: +When enabled, every LLM call records a per-call decomposition of its prompt tokens, so you can see where the input budget goes on each step of an agent run: - `system_prompt_tokens` - Tokens in system messages - `tool_tokens` - Tokens in the tool schemas included with the call - `history_tokens` - Tokens in conversation history (all non-system messages except the latest one) - `latest_message_tokens` - Tokens in the latest observation or user message -The decomposition is computed automatically on every call - both the Chat Completions and Responses API paths - and recorded into the LLM's [Metrics](/sdk/guides/metrics). No configuration is required. On the Responses API path, the decomposition is computed on the finalized payload (instructions plus input items), so the record reflects what the provider received. +Recording is **opt-in**: set `enable_prompt_composition=True` on the LLM. When off (the default), no tokenization pass runs and no records are appended. When on, the decomposition is computed on every call - both the Chat Completions and Responses API paths - and recorded into the LLM's [Metrics](/sdk/guides/metrics). On the Responses API path, the decomposition is computed on the finalized payload (instructions plus input items), so the record reflects what the provider received. + +```python icon="python" focus={4} +llm = LLM( + model="anthropic/claude-sonnet-4-5-20250929", + api_key=SecretStr(os.getenv("LLM_API_KEY")), + enable_prompt_composition=True, +) +``` ## Accessing the Composition -Each call appends one `PromptComposition` record to `llm.metrics.prompt_compositions`. The most recent record is available as `llm.metrics.latest_prompt_composition`: +With the flag enabled, each call appends one `PromptComposition` record to `llm.metrics.prompt_compositions`. The most recent record is available as `llm.metrics.latest_prompt_composition` (`None` when the flag is off): ```python icon="python" conversation.run() @@ -73,7 +81,7 @@ Composition counts are **client-side estimates**, computed with the model's toke ## Performance -The estimator's cost scales linearly with prompt size: roughly 10-20 ms per call on a typical agent payload, measured at ~31 ms for a ~100K-token prompt and ~61 ms for ~190K tokens (19 tools, gpt-4o tokenizer) - negligible next to network latency, so it is always on. +The estimator's cost scales linearly with prompt size: roughly 10-20 ms per call on a typical agent payload, measured at ~31 ms for a ~100K-token prompt and ~61 ms for ~190K tokens (19 tools, gpt-4o tokenizer) - negligible next to network latency. Because the feature is opt-in, this cost is only paid when `enable_prompt_composition=True`. ## Next Steps diff --git a/sdk/guides/metrics.mdx b/sdk/guides/metrics.mdx index e4035318d..ecd44ca3e 100644 --- a/sdk/guides/metrics.mdx +++ b/sdk/guides/metrics.mdx @@ -41,7 +41,7 @@ The `llm.metrics` object is an instance of the [Metrics class](https://github.co - `costs` - List of individual cost records per API call - `token_usages` - List of detailed token usage records per API call - `response_latencies` - List of response latency metrics per API call -- `prompt_compositions` - List of per-call [prompt token composition](/sdk/guides/llm-prompt-composition) estimates (system prompt, tool schemas, history, latest message) +- `prompt_compositions` - List of per-call [prompt token composition](/sdk/guides/llm-prompt-composition) estimates (system prompt, tool schemas, history, latest message); populated only when `enable_prompt_composition=True` on the LLM For more details on the available metrics and methods, refer to the [source code](https://github.com/OpenHands/software-agent-sdk/blob/main/openhands-sdk/openhands/sdk/llm/utils/metrics.py). From 9ddf874b696f5362b80c27791c55568679b9a268 Mon Sep 17 00:00:00 2001 From: george larson Date: Mon, 31 Aug 2026 14:23:52 -0400 Subject: [PATCH 5/5] docs(sdk): rework prompt composition guide for offline report script The SDK PR was reshaped from a runtime opt-in metric to an offline analysis script (scripts/prompt_composition_report.py) that ingests LLM(log_completions=True) logs. Rework the guide to document enabling completion logging, running the report (--root/--out/--no-chart), the four buckets (tool_tokens is now tool_schema_tokens), the est/provider median ratio, and the tool_schema_counted=false caveat for logs written before tool schemas were logged in finalized form. Drop the enable_prompt_composition flag, the metrics surface references, and the runtime join example; note the report is the measurement baseline for deferred tool loading (software-agent-sdk#4083). Remove the prompt_compositions bullet from the metrics guide since the runtime field no longer exists. Co-authored-by: openhands --- sdk/guides/llm-prompt-composition.mdx | 82 +++++++++++++-------------- sdk/guides/metrics.mdx | 3 +- 2 files changed, 42 insertions(+), 43 deletions(-) diff --git a/sdk/guides/llm-prompt-composition.mdx b/sdk/guides/llm-prompt-composition.mdx index 81dbc717e..48f1e2793 100644 --- a/sdk/guides/llm-prompt-composition.mdx +++ b/sdk/guides/llm-prompt-composition.mdx @@ -1,87 +1,87 @@ --- title: Prompt Token Composition -description: Break down prompt tokens per LLM call into system prompt, tool schemas, conversation history, and the latest message. +description: Break down prompt tokens per LLM call into system prompt, tool schemas, conversation history, and the latest message with the offline composition report. --- ## Overview -When enabled, every LLM call records a per-call decomposition of its prompt tokens, so you can see where the input budget goes on each step of an agent run: +The prompt composition report is an **offline analysis script** in the [software-agent-sdk repository](https://github.com/OpenHands/software-agent-sdk/blob/main/scripts/prompt_composition_report.py). It reads the completion logs written by the SDK during a run and rebuilds, for every LLM call, a decomposition of its prompt tokens, so you can see where the input budget went on each step of an agent run: - `system_prompt_tokens` - Tokens in system messages -- `tool_tokens` - Tokens in the tool schemas included with the call +- `tool_schema_tokens` - Tokens in the tool schemas included with the call - `history_tokens` - Tokens in conversation history (all non-system messages except the latest one) - `latest_message_tokens` - Tokens in the latest observation or user message -Recording is **opt-in**: set `enable_prompt_composition=True` on the LLM. When off (the default), no tokenization pass runs and no records are appended. When on, the decomposition is computed on every call - both the Chat Completions and Responses API paths - and recorded into the LLM's [Metrics](/sdk/guides/metrics). On the Responses API path, the decomposition is computed on the finalized payload (instructions plus input items), so the record reflects what the provider received. +Because the analysis runs after the fact on logged payloads, it adds **no runtime cost and no runtime API surface** - there is nothing to enable on the LLM beyond completion logging, which the SDK already supports. On the Responses API path, the decomposition is computed on the finalized logged payload (instructions plus input items), so the report reflects what the provider received. -```python icon="python" focus={4} +A typical question this answers: how many prompt tokens per call are re-sent tool schemas? That number is the baseline for evaluating deferred tool loading (see [software-agent-sdk#4083](https://github.com/OpenHands/software-agent-sdk/issues/4083)). + +## Enable Completion Logging + +Run your agent with `log_completions=True` so each LLM call is written to a JSON log file: + +```python icon="python" focus={4-5} llm = LLM( model="anthropic/claude-sonnet-4-5-20250929", api_key=SecretStr(os.getenv("LLM_API_KEY")), - enable_prompt_composition=True, + log_completions=True, + log_completions_folder="logs/completions", ) ``` -## Accessing the Composition +`log_completions_folder` defaults to `logs/completions`. Each call lands in its own `*.json` file containing the finalized request payload (messages and OpenAI-format tool schemas on the Chat Completions path; `instructions`, `input`, and Responses-format tool schemas on the Responses API path) plus the provider-reported usage, latency, and timestamp. The report accepts either a single folder of log files or a directory containing several such run folders. -With the flag enabled, each call appends one `PromptComposition` record to `llm.metrics.prompt_compositions`. The most recent record is available as `llm.metrics.latest_prompt_composition` (`None` when the flag is off): +## Run the Report -```python icon="python" -conversation.run() +From the root of the `software-agent-sdk` repository: -composition = llm.metrics.latest_prompt_composition -assert composition is not None -print(f"System prompt: {composition.system_prompt_tokens}") -print(f"Tool schemas: {composition.tool_tokens}") -print(f"History: {composition.history_tokens}") -print(f"Latest message: {composition.latest_message_tokens}") +```bash icon="terminal" +uv run python scripts/prompt_composition_report.py --root logs/completions ``` -Each record carries the `response_id` of its call. Join records with the provider-reported usage in `llm.metrics.token_usages` by `response_id` rather than by position - the two lists can diverge when a composition is skipped or a response carries no usage: - -```python icon="python" -usage_by_id = {u.response_id: u for u in llm.metrics.token_usages} - -for composition in llm.metrics.prompt_compositions: - usage = usage_by_id.get(composition.response_id) - if usage is None: - continue - estimated = ( - composition.system_prompt_tokens - + composition.tool_tokens - + composition.history_tokens - + composition.latest_message_tokens - ) - print( - f"{composition.response_id}: estimated {estimated}, " - f"provider reported {usage.prompt_tokens}" - ) +- `--root` (required) - A run folder of `*.json` completion logs, or a root directory containing run folders. +- `--out DIR` - Also write the per-call rows to `calls.jsonl` and the aggregate numbers to `summary.json` in `DIR`. +- `--no-chart` - Skip the text visualization on stdout. + +The script prints a stacked bar per call (`S`=system, `T`=tool_schema, `H`=history, `L`=latest) and a trend table over the call sequence: + +```text icon="terminal" + seq composition est_total provider + 0 SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSTTLL 675 702 + 1 SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSTTHHHHLL 754 784 + 2 SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSTTHHHHHHHHHHLL 870 904 ``` -Agent steps that send tools as native function-calling schemas have `tool_tokens > 0`. Auxiliary calls that pass no tools - for example the [context condenser](/sdk/guides/context-condenser) or title generation - are recorded with `tool_tokens == 0`. +Each `calls.jsonl` row carries `seq`, `usage` (provider-reported, joined by response id), `composition` (the four estimated buckets), `latency_s`, and `tool_schema_counted`. The `summary.json` aggregates per-bucket averages and the `est_provider_median_ratio` - the median of estimated-total over provider-reported `prompt_tokens` across calls, which shows how much of the billed prompt the breakdown covers. + +Agent steps that send tools as native function-calling schemas show `tool_schema_tokens > 0`. Auxiliary calls that pass no tools - for example the [context condenser](/sdk/guides/context-condenser) or title generation - show `tool_schema_tokens == 0`. - On models without native function calling, the SDK renders tool schemas into the prompt text instead of sending them as tool parameters. Those agent steps are recorded with `tool_tokens == 0` and the schema tokens appear in the message buckets instead. + On models without native function calling, the SDK renders tool schemas into the prompt text instead of sending them as tool parameters. The report detects this from the log and leaves `tool_schema_tokens` at 0 for those calls; the schema tokens appear in the message buckets instead, with no double counting. + + Completion logs written by SDK versions before tool schemas were logged in finalized form record only tool names and descriptions, not parameter schemas. The report cannot reconstruct the tool bucket from those and flags the affected rows with `tool_schema_counted=false` (marked `*` in the chart) rather than miscounting them; flagged calls are excluded from the est/provider median ratio. Re-run with a current SDK to populate the bucket. + + ## Estimates vs Provider-Reported Usage -Composition counts are **client-side estimates**, computed with the model's tokenizer before the request is sent. The provider-reported `TokenUsage` remains the authoritative accounting: +Composition counts are **client-side estimates**, computed offline with the model's tokenizer from the logged payload. The provider-reported usage in each log remains the authoritative accounting: - `is_estimate` is `True` on records produced by the client-side estimator. - Each component is counted independently, so per-message framing overhead is included in every component and the components do not necessarily sum exactly to the provider-reported `prompt_tokens`. - Tool schema counts follow litellm's `token_counter` serialization convention for tools, which can differ from the provider's wire-format tokenization. - For models litellm has no tokenizer mapping for, counts use litellm's fallback tokenizer and may deviate more from the provider's counts. -- When token counting fails or is disabled (for example `litellm.disable_token_counter`), composition recording is skipped for that call; the call itself is unaffected. +- When token counting fails or is disabled (for example `litellm.disable_token_counter`), the affected log file is skipped and counted under `skipped_files` in the summary; other calls are unaffected. - Buckets follow the wire, not the logical prompt: in subscription mode the system prompt is folded into the first user message before transport, so those tokens are counted in `history_tokens` or `latest_message_tokens` rather than `system_prompt_tokens` on that path. - Treat the composition as a breakdown of *where* prompt tokens go, and `token_usages` as the record of *how many* tokens the provider billed. + Treat the composition as a breakdown of *where* prompt tokens go, and the provider-reported usage as the record of *how many* tokens the provider billed. ## Performance -The estimator's cost scales linearly with prompt size: roughly 10-20 ms per call on a typical agent payload, measured at ~31 ms for a ~100K-token prompt and ~61 ms for ~190K tokens (19 tools, gpt-4o tokenizer) - negligible next to network latency. Because the feature is opt-in, this cost is only paid when `enable_prompt_composition=True`. +The counting cost is paid at analysis time, not during the agent run. It scales linearly with prompt size: roughly 10-20 ms per call on a typical agent payload, measured at ~31 ms for a ~100K-token prompt and ~61 ms for ~190K tokens (19 tools, gpt-4o tokenizer). ## Next Steps diff --git a/sdk/guides/metrics.mdx b/sdk/guides/metrics.mdx index ecd44ca3e..36a57f2df 100644 --- a/sdk/guides/metrics.mdx +++ b/sdk/guides/metrics.mdx @@ -41,7 +41,6 @@ The `llm.metrics` object is an instance of the [Metrics class](https://github.co - `costs` - List of individual cost records per API call - `token_usages` - List of detailed token usage records per API call - `response_latencies` - List of response latency metrics per API call -- `prompt_compositions` - List of per-call [prompt token composition](/sdk/guides/llm-prompt-composition) estimates (system prompt, tool schemas, history, latest message); populated only when `enable_prompt_composition=True` on the LLM For more details on the available metrics and methods, refer to the [source code](https://github.com/OpenHands/software-agent-sdk/blob/main/openhands-sdk/openhands/sdk/llm/utils/metrics.py). @@ -410,6 +409,6 @@ for usage_id, metrics in conversation.conversation_stats.usage_to_metrics.items( ## Next Steps -- **[Prompt Token Composition](/sdk/guides/llm-prompt-composition)** - Break down prompt tokens per call into system prompt, tool schemas, history, and latest message +- **[Prompt Token Composition](/sdk/guides/llm-prompt-composition)** - Offline report that breaks down prompt tokens per call into system prompt, tool schemas, history, and latest message - **[Context Condenser](/sdk/guides/context-condenser)** - Learn about context management and how it uses separate LLMs - **[LLM Routing](/sdk/guides/llm-routing)** - Optimize costs with smart routing between different models