tracewright

Capture Pydantic AI OpenTelemetry model spans as local, versionable pydantic-evals datasets.

Tracewright removes the manual step between observing a useful agent run and turning it into an eval case. It preserves the prompt, expected output, model, token usage, timing, and trace identity. Pydantic Evals continues to own dataset storage, evaluators, execution, and reports.

Pydantic AI run -> OpenTelemetry model span -> Pydantic Evals Case

Quick start

The example uses Pydantic AI's TestModel, so it needs no provider key or hosted observability service.

pip install "tracewright[pydantic-ai,pydantic-evals]"

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from pydantic_ai import Agent, InstrumentationSettings
from pydantic_ai.capabilities import Instrumentation
from pydantic_ai.models.test import TestModel

from tracewright import PydanticAITraceExporter, to_pydantic_evals_dataset

exporter = PydanticAITraceExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))

agent = Agent(
    TestModel(),
    instructions="Answer concisely.",
    capabilities=[Instrumentation(settings=InstrumentationSettings(tracer_provider=provider))],
)
agent.run_sync("What is 2+2?")

dataset = to_pydantic_evals_dataset(exporter.rows, name="agent-regression")
dataset.to_file("agent-regression.yaml")
print(exporter.stats)

The resulting file is a normal Pydantic Evals dataset. Review it, commit it with the code it protects, and evaluate future agent versions against it.

The complete runnable example is examples/pydantic_ai_to_evals.py.

Contract

PydanticAITraceExporter is a standard OpenTelemetry SpanExporter. Attach it to the tracer provider already used by Pydantic AI. Its rows property returns an immutable snapshot suitable for to_pydantic_evals_dataset.

Each replayable model span becomes one pydantic_evals.Case:

Case field	Source
inputs	Last user message
expected_output	Assistant text output
metadata	The complete TraceRow, including system instructions, model, tokens, timing, and trace IDs

Non-model spans are ignored. Model spans with missing, redacted, or malformed content are counted in exporter.stats and are never silently promoted into eval cases.

Content capture is a privacy decision. Pydantic AI instrumentation with include_content=False still produces useful telemetry, but it cannot produce replayable prompt and output pairs.

Boundaries

Tracewright deliberately does not:

• replace Pydantic Evals evaluators, execution, or reports;
• replace Logfire trace storage or hosted datasets;
• query the Logfire API or decode OTLP payloads;
• reconstruct tool calls, full multi-turn state, or SpanTree objects.

parse_pydantic_ai_jsonl remains available for files containing exactly one serialized span object per line. It is not a generic Logfire export or OTLP reader. Enriched Tracewright/f3dx JSONL and the original replay CLI remain as compatibility surfaces.

tracewright replay traces.jsonl \
  --candidate myapp.replay:my_candidate \
  --candidate-model candidate-v2 \
  --report html=report.html \
  --budget "pass_rate=>=1.0,latency_p95=+10%"

New integrations should use PydanticAITraceExporter and Pydantic Evals.

Development

python -m pip install -e ".[dev]"
ruff check src tests examples
mypy src
pytest -q
python examples/pydantic_ai_to_evals.py

CI runs on Linux, macOS, and Windows with Python 3.10 and 3.12. It also replays Tracewright's serialized TraceRow contract across supported Pydantic versions using Bumpcheck.

License

MIT