ci: trusted validation for #2860

.changeset/datadog-eval-reporter.md

@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Add a Datadog eval reporter that creates one LLM Observability Experiment per eve eval run and submits eval assertion metrics through the optional `dd-trace` package. Opted-in eval inputs are pushed as versioned dataset records and linked to their experiment rows. The integration is tested against the public dataset and external Experiment APIs in `dd-trace@6.13.0`.

apps/fixtures/datadog-eval-reporter/README.md

@@ -0,0 +1,22 @@
+# Datadog eval reporter fixture
+
+This deterministic mock-model app exercises the Datadog eval reporter against
+`dd-trace@6.13.0`.
+
+```bash
+export DD_API_KEY="..."
+export DD_APP_KEY="..."
+export DD_SITE="datadoghq.com"
+
+pnpm --filter datadog-eval-reporter-fixture eval
+```
+
+Without both Datadog keys, the fixture omits the reporter so the shared local,
+Postgres, and Vercel e2e matrices remain hermetic.
+
+The credentialed run creates a Datadog dataset with one record containing `Say hello.`,
+creates one LLM Observability Experiment against that dataset version, links the
+record to the smoke eval's synthetic experiment span, attaches the assertion
+metrics, and prints the Dataset and Experiment URLs. The fixture does not export
+the agent runtime's OpenTelemetry spans; Datadog's public dataset and external
+Experiment APIs generate the record, trace, and span identifiers.

apps/fixtures/datadog-eval-reporter/agent/agent.ts

@@ -0,0 +1,9 @@
+import { e2eAgentConfig } from "@eve-e2e/config";
+import { defineAgent } from "eve";
+import { mockModel } from "eve/evals";
+
+export default defineAgent({
+  ...e2eAgentConfig(),
+  model: mockModel(),
+  modelContextWindowTokens: 1_000_000,
+});

apps/fixtures/datadog-eval-reporter/agent/instructions.md

@@ -0,0 +1 @@
+You are a deterministic test agent used to develop the Datadog eval reporter.

apps/fixtures/datadog-eval-reporter/evals/evals.config.ts

@@ -0,0 +1,8 @@
+import { defineEvalConfig } from "eve/evals";
+import { Datadog } from "eve/evals/reporters";
+
+const hasDatadogCredentials = Boolean(process.env.DD_API_KEY && process.env.DD_APP_KEY);
+
+export default defineEvalConfig({
+  reporters: hasDatadogCredentials ? [Datadog({ recordInputs: true })] : [],
+});

apps/fixtures/datadog-eval-reporter/evals/smoke.eval.ts

@@ -0,0 +1,10 @@
+import { defineEval } from "eve/evals";
+
+export default defineEval({
+  description: "Datadog reporter smoke eval.",
+
+  async test(t) {
+    await t.send("Say hello.");
+    t.succeeded();
+  },
+});

apps/fixtures/datadog-eval-reporter/package.json

@@ -0,0 +1,15 @@
+{
+  "name": "datadog-eval-reporter-fixture",
+  "version": "0.0.0",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "eval": "eve eval --strict"
+  },
+  "dependencies": {
+    "@eve-e2e/config": "workspace:*",
+    "@workflow/world-postgres": "catalog:",
+    "dd-trace": "6.13.0",
+    "eve": "workspace:*"
+  }
+}

docs/evals/reporters.mdx

@@ -1,6 +1,6 @@
 ---
 title: "Reporters"
-description: "Ship eval results to Braintrust experiments or JUnit XML. eve runs and scores everything itself."
+description: "Ship eval results to Braintrust, Datadog, or JUnit XML. eve runs and scores everything itself."
 ---
 
 eve runs and grades everything itself; reporters ship the results out. The CLI prints a console summary by default (one line per eval, with failed assertions and their messages), and reporters from `eve/evals/reporters` add destinations on top.
@@ -44,6 +44,25 @@ A reporter instance observes the evals that reference it. Share one instance acr
 
 Braintrust needs its SDK installed in the app and credentials in the environment: install the `braintrust` package (`npm install braintrust`) and set `BRAINTRUST_API_KEY`. Pass `--skip-report` to run the eval without shipping results, which also suppresses config reporters and is useful locally when iterating.
 
+## Datadog
+
+`Datadog(...)` uploads eval assertion scores to a Datadog LLM Observability Experiment. Put one instance in the config so it covers the whole run:
+
+```ts title="evals/evals.config.ts"
+import { defineEvalConfig } from "eve/evals";
+import { Datadog } from "eve/evals/reporters";
+
+export default defineEvalConfig({
+  reporters: [Datadog({ projectName: "weather-agent" })],
+});
+```
+
+The reporter creates one Datadog Experiment, submits one synthetic experiment span per completed eve eval, associates that eval's assertion scores as experiment metrics, and prints the Datadog Experiment URL after the run completes. Without input recording, the external Experiment uses a placeholder dataset and spans are submitted as evals finish. With `recordInputs: true`, the reporter instead creates and pushes a versioned Datadog dataset after the evals finish, adds one record per eval, starts the Experiment against that dataset version, passes each dataset record id to its corresponding experiment span, and prints both Dataset and Experiment URLs. Buffering is necessary because an imperative eval's first `t.send(...)` input is only known after that eval executes. `dd-trace` generates dataset record, experiment trace, and experiment span identifiers; the reporter does not mutate or link the agent runtime's OpenTelemetry spans.
+
+Assertion metrics use descriptive assertion names by default: characters outside letters, numbers, underscores, and hyphens are normalized to underscores, gate labels receive a `gate_` prefix, and repeated or reserved labels receive numeric suffixes. For example, `succeeded` becomes `gate_succeeded` and `calledTool(get_stock_quote)` becomes `gate_calledTool_get_stock_quote`. Authors can use `.label("stable name")` on an assertion handle to choose a stable metric name. Because assertion names can contain authored expectations, review them before exporting sensitive evals. Pass `recordAssertionDetails: true` only when the destination is also approved for raw assertion-name tags and failed assertion messages in row metadata.
+
+Datadog needs `dd-trace` installed in the app and Datadog credentials in the environment. The reporter is tested against `dd-trace@6.13.0`; install it with `npm install dd-trace@6.13.0` and set `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` as appropriate. By default, the reporter records assertion scores, eval metadata, and the target URL origin only. Pass `recordInputs: true`, `recordOutputs: true`, `recordExpectedOutputs: true`, or `recordErrors: true` if your destination is approved for eval prompts, outputs, authored expectations, or execution error messages. `recordInputs` stores the eval input in both the linked dataset record and experiment span. Input is read from the eval's first `t.send(...)` message, falling back to the eval description when no message event is available. When `recordExpectedOutputs` is enabled, expected output is read from eval `metadata.expectedOutput`, `metadata.expected`, or `metadata.expected_output` and stored in both the dataset record and experiment span; when no expected output is authored, the input-only dataset record is still valid. Those expected-output keys are excluded from the general metadata payload. Target URL credentials, paths, query parameters, and fragments are never reported.
+
 ## JUnit
 
 `JUnit({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same thing without touching the eval file, usually the better fit because CI owns the output path, not the eval:

packages/eve/package.json

@@ -498,6 +498,7 @@
     "chat": "4.34.0",
     "chokidar": "5.0.0",
     "commander": "14.0.3",
+    "dd-trace": "6.13.0",
     "emulate": "0.6.0",
     "env-runner": "0.1.16",
     "eventsource-parser": "3.1.0",
@@ -527,6 +528,7 @@
     "@opentelemetry/api": "^1.0.0",
     "ai": "catalog:",
     "braintrust": "^3.0.0",
+    "dd-trace": "^6.13.0",
     "just-bash": "^3.1.0",
     "microsandbox": "^0.5.0"
   },
@@ -537,6 +539,9 @@
     "braintrust": {
       "optional": true
     },
+    "dd-trace": {
+      "optional": true
+    },
     "just-bash": {
       "optional": true
     },

packages/eve/src/evals/reporters/index.ts

@@ -1,5 +1,6 @@
 export { Braintrust, type BraintrustReporterConfig } from "#evals/runner/reporters/braintrust.js";
 export { Console, type ConsoleReporterConfig } from "#evals/runner/reporters/console.js";
+export { Datadog, type DatadogReporterConfig } from "#evals/runner/reporters/datadog.js";
 export { JUnit, type JUnitReporterConfig } from "#evals/runner/reporters/junit.js";
 export type {
   EvalReporter,