Add blog post announcing the Agent LLM Traces dataset (10K agent runs, 6 domains)

blogs/agent-llm-traces-v2/blog.md

@@ -0,0 +1,129 @@
+---
+title: "What Agents Do: A Large Scale Public Dataset of Agent Runs"
+subtitle: "10K real agent runs, in one standard format, so anyone can study how agents behave and not just what they scored."
+author: ""
+date: 2026-07-01
+reading_time: 8 min
+slug: agent-llm-traces-v2
+---
+
+AI agents now book our travel, write our code, and answer our customers. Two can finish the same task and look identical on paper — one taking three clean steps, the other opening a file it shouldn't, crashing a service, and recovering by luck. We already know whether they succeed and what they cost. It's time we saw how they get there.
+
+Today we're releasing [Exgentic Agent LLM Traces](https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2): 10K real agent runs, across five leading models and six kinds of task, with every model call preserved in one standard format. It's the part of the run that normally gets thrown away.
+
+> We've always known whether agents succeed or failed. Now we can see how.
+
+## Why this matters
+
+Agents are moving into real work: writing code, handling customer support, doing research, automating tasks across hundreds of apps. Benchmarks are how we study that work before it reaches the people who depend on it — each one a controlled mirror of a real job, a stand-in for the messy thing we actually care about. So the question is never really how an agent scores on the mirror. It is how the agent will behave doing the real thing: what it costs, where it gives up, whether it recovers when something breaks. And almost everything we know about that comes from a single number per run, because that number is usually all anyone keeps.
+
+That leaves a gap. We study *models* on mountains of public data. We study *agents* — the multi-step, tool-using, failing-and-recovering systems people actually deploy — on almost nothing public. The behavior that decides whether an agent is worth shipping lives in the full record of what it did, and that record rarely leaves the lab that produced it.
+
+The gap exists for a concrete reason: producing real agent runs at scale is slow, costly, and fragmented. It means running the strongest available models across many different environments, thousands of times over, and paying for every token they generate. And even once you have the runs, every tool records them in its own shape, so what data exists stays private and mutually incompatible. There has been no large, uniform, public record of what capable agents actually do.
+
+> Model behavior is studied everywhere. Agent behavior is barely studied at all, because the data isn't public.
+
+This release is a piece of that missing record. Real runs, at scale, from frontier models, across the domains agents are deployed in, in one format anyone can load. That combination didn't exist before today.
+
+## What you can do with it
+
+Preserving the full run, not just the verdict, is what makes the data useful. Each of these was previously gated behind running the evaluations yourself:
+
+- **Train and fine-tune agents on real runs.** Long, tool-using runs from strong models are the kind of data that is hard to obtain and expensive to generate.
+- **Build tools that evaluate agents, using real failures.** Study where agents actually break instead of on made-up examples.
+- **Replay a recorded moment against a new model.** Feed the exact situation an agent faced to a different model and compare, without re-running the whole evaluation.
+- **Debug your own agent against a reference set.** Compare its behavior to how the strongest models handle the same kinds of tasks.
+- **Load-test inference infrastructure against realistic traffic.** Standard load tests send independent requests; real agent runs are dependency chains where the output of one call shapes the input of the next. Because these traces are already in OTel format, they work directly with tools that replay recorded workloads against live endpoints.
+
+## What's in it
+
+The dataset is 10K runs and 242K model calls, filtered down from a full corpus of 10.5K runs and 626K calls. Roughly two-thirds of the raw calls were removed on purpose: the scaffolding around the agent under test.
+
+Each run passes through a filter that strips out the supporting machinery — the simulated user, the automatic grader — collapses repeated identical retries, and keeps only the calls the tested model actually made. What remains is the model under test, and nothing the evaluation wrapped around it. The totals for steps, tokens, and cost are recomputed from the calls that survive, so they describe the agent's own footprint and not the machinery around it.
+
+Every call is written in one open, standard format for recording model calls (the [OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/), which a growing part of the ecosystem already emits). A call carries the models used, the full input and output messages, the tools that were available, token usage, why the call stopped, and an error type when it failed. Each row also carries run-level context — which benchmark, which agent design, the score, the cost, the number of steps — so you can slice the data however you need. Because every run speaks the same format, these traces accumulate into something comparable instead of fragmenting into one dialect per tool — the value compounds as more are added.
+
+The coverage spans the range of real agent work — software engineering, deep research, personal-task automation across hundreds of apps, and policy-bound customer service and technical support. The token averages alone show how differently these tasks load a model:
+
+| Domain | Benchmark | Runs | Avg tokens / run |
+|--------|-----------|-----:|-----------------:|
+| Software engineering | SWE-bench | 2K | 2.1M |
+| Deep research | BrowseComp Plus | 1.9K | 1.6M |
+| Personal-task automation | AppWorld | 1.5K | 1.7M |
+| Technical support | τ²-bench Telecom | 1.8K | 332K |
+| Customer service | τ²-bench Retail | 1.8K | 198K |
+| Customer service | τ²-bench Airline | 1K | 255K |
+
+Five frontier models are represented in comparable volume — DeepSeek-V3.2 (2.3K runs), Kimi-K2.5 (2.3K), GPT-5.2 (2.1K), Claude Opus 4.5 (1.9K), and Gemini 3 Pro (1.4K) — each run through up to five different agent designs. Because the format is identical across all of them, comparing how two models handled the same task, or how one model's behavior shifts between agent designs, is a filter, not a data-cleaning project.
+
+## A closer look
+
+Three views of the data, each answering a different question.
+
+**What kind of work is this?** Task complexity differs sharply across benchmarks:
+
+| Benchmark | Turns (median / mean / max) | Avg tool calls / run | Responses with >1 tool call (%) | Avg duration (min) |
+|-----------|----------------------------:|---------------------:|--------------------------------:|-------------------:|
+| SWE-bench | 38 / 46.8 / 259 | 31.7 | 10% | 16 |
+| AppWorld | 19 / 32.3 / 300 | 27.8 | 10% | 14 |
+| BrowseComp Plus | 13 / 19.7 / 167 | 15.4 | 14% | 15 |
+| τ²-bench Telecom | 13 / 16.3 / 198 | 5.9 | 3% | 8 |
+| τ²-bench Airline | 10 / 11.4 / 151 | 6.2 | 9% | 5 |
+| τ²-bench Retail | 11 / 11.9 / 151 | 5.5 | 6% | 6 |
+
+Turns are the number of model calls in a run. Tool calls are how many tool invocations the model issued in total across those calls. The third column counts the fraction of model calls where the model returned more than one tool call in a single response — batching multiple requests at once rather than one at a time. Duration is the wall-clock time from the first to the last model call in a run.
+
+Taken together, these four numbers show how structurally different the workloads are: SWE-bench and AppWorld runs are long, tool-heavy, and slow; τ²-bench runs are short, light on tools, and fast. The workloads are not interchangeable, and which ones you pull matters for training, evaluation, and infrastructure work.
+
+**How much does it cost?** Token consumption varies as much by model as by task:
+
+| Model | Avg tokens / model call | Avg tokens / run |
+|-------|------------------------:|-----------------:|
+| Claude Opus 4.5 | 66.8K | 2.4M |
+| Gemini 3 Pro | 41.8K | 1.1M |
+| DeepSeek-V3.2 | 29.0K | 880K |
+| GPT-5.2 | 26.3K | 552K |
+| Kimi-K2.5 | 23.1K | 655K |
+
+A Claude Opus 4.5 run consumes on average 4× the tokens of a GPT-5.2 run — a difference that compounds across hundreds of calls. Selecting traces by model matters if you're estimating replay costs, building token-budget-aware training pipelines, or studying how different models use their context window.
+
+**How does context evolve?** Because agent design determines how the prompt is constructed on each call, it is the primary driver of how context accumulates — not the task type. Prefix retention measures what fraction of messages carry over from one call to the next: 1.0 means a single growing conversation; lower values mean the agent reconstructs the context window more aggressively each time.
+
+| Agent design | Avg turns / run | Prefix retention |
+|--------------|----------------:|-----------------:|
+| claude_code | 33.0 | 0.90 |
+| smolagents_code | 21.3 | 0.87 |
+| tool_calling | 22.2 | 0.87 |
+| openai_solo | 18.3 | 0.86 |
+| tool_calling_with_shortlisting | 50.1 | 0.03 |
+
+Four of the five designs maintain high retention throughout a run — they accumulate a single long conversation. The exception is `tool_calling_with_shortlisting`, which drops to 0.03: it selects a fresh subset of tools for each call, rebuilding the prompt from scratch every time. Despite having the highest average turn count by far, it never builds up a long conversational context. This makes its traces structurally different from the others at the message level — worth knowing before using them for fine-tuning or prefix caching analysis.
+
+## Inference benchmarking: an ongoing example
+
+One use of this data is already underway. A team working on LLM inference infrastructure is using these traces to benchmark serving configurations under realistic agentic load — the kind of workload that synthetic benchmarks systematically miss.
+
+The tool at the center of that effort is [inference-perf](https://github.com/kubernetes-sigs/inference-perf), a Kubernetes SIG benchmarking tool that replays OTel traces as directed acyclic graphs. Rather than sending independent requests at a fixed rate, it reconstructs the dependency structure of a real agent run — the order and dependencies between calls, and how context accumulated across them — and drives that structure against the inference endpoint under test. Downstream calls receive the actual live-generated output from upstream calls, so KV-cache behavior and context growth stay realistic throughout the replay.
+
+Because the Exgentic traces are already in OTel format, they plug into this workflow without any conversion. The result is end-to-end run latency measured against real agent traffic, not a synthetic proxy for it — exactly the signal needed to evaluate whether a serving configuration change is worth shipping.
+
+This is one example of what becomes possible when traces are public and in a standard format. The work is ongoing; we'll share results as they come in.
+
+## What it doesn't cover
+
+The dataset is deliberately narrow in a few ways worth stating plainly. It captures the model's own chat calls, not the non-model actions around them, so an agent's file edits or environment steps show up only through the calls that produced them. The models are specific snapshots evaluated at one point in time, not standing verdicts on any model family. And six benchmarks, however varied, are not the full range of work agents will eventually do. If you need the surrounding scaffolding — the user-simulator turns and grading calls we filtered out — the full corpus below keeps all of it.
+
+## See for yourself
+
+The runs are public and ready to load — one open format, 236 MB, [on Hugging Face](https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2). Whatever you are building agents to do, or worried they might do, you can now start from what they actually did instead of a single line of results. Open a few runs and follow how they get there.
+
+> The how was always there, inside every run. Now anyone can read it.
+
+## Sources
+
+- Exgentic Agent LLM Traces: [huggingface.co/datasets/Exgentic/agent-llm-traces-v2](https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2)
+- Full corpus: [huggingface.co/datasets/Exgentic/traces-v2](https://huggingface.co/datasets/Exgentic/traces-v2)
+- OpenTelemetry GenAI semantic conventions: [opentelemetry.io](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
+- inference-perf: [github.com/kubernetes-sigs/inference-perf](https://github.com/kubernetes-sigs/inference-perf) · [Benchmarking LLM Inference with Production Agent Traces](https://medium.com/inference-perf/benchmarking-llm-inference-with-production-agent-traces-f47f7f994aff)
+- Exgentic & Open Agent Leaderboard: [arXiv:2602.22953](https://arxiv.org/abs/2602.22953) · [exgentic.ai](https://www.exgentic.ai)
+</content>

index.html

@@ -198,6 +198,16 @@ <h2 class="section-title">Open Source. Open Data. Open Leaderboard.</h2>
     <h2 class="section-title">Blog</h2>
     <p class="section-subtitle">Research updates, benchmarks, and findings from the Exgentic team.</p>
     <div class="blog-cards-grid">
+      <div class="blog-card-home glass">
+        <div class="blog-card-home-meta">
+          <time>Jul 1, 2026</time>
+          <span>&middot;</span>
+          <span>8 min read</span>
+        </div>
+        <h3><a href="/blog/agent-llm-traces-v2/">What Agents Do: A Large Scale Public Dataset of Agent Runs</a></h3>
+        <p>10K real agent runs, in one standard format, so anyone can study how agents behave and not just what they scored.</p>
+        <a href="/blog/agent-llm-traces-v2/" class="blog-card-home-link">Read more &rarr;</a>
+      </div>
       <div class="blog-card-home glass">
         <div class="blog-card-home-meta">
           <time>May 13, 2026</time>

scripts/build_blog.py

@@ -117,9 +117,7 @@
     <h1 class="blog-post-title">{title}</h1>
     <p class="blog-post-subtitle">{subtitle}</p>
     <div class="blog-post-meta">
-      <span>{author}</span>
-      <span>&middot;</span>
-      <time datetime="{date_iso}">{date_display}</time>
+      {author_block}<time datetime="{date_iso}">{date_display}</time>
       <span>&middot;</span>
       <span>{reading_time} read</span>
       <button class="blog-share-btn" id="shareBtn" title="Copy link">
@@ -482,7 +480,11 @@ def build_blog(slug: str) -> tuple[dict, str] | None:
         + ARTICLE_HEADER_TEMPLATE.format(
             title=meta["title"],
             subtitle=meta["subtitle"],
-            author=meta["author"],
+            author_block=(
+                f'<span>{meta["author"]}</span>\n      <span>&middot;</span>\n      '
+                if str(meta["author"]).strip()
+                else ""
+            ),
             date_iso=date_iso,
             date_display=date_display,
             reading_time=meta["reading_time"],