diff --git a/.changeset/evals-readme.md b/.changeset/evals-readme.md new file mode 100644 index 000000000..15cfc9f06 --- /dev/null +++ b/.changeset/evals-readme.md @@ -0,0 +1,5 @@ +--- +"@voltagent/evals": patch +--- + +Add README documentation diff --git a/packages/evals/README.md b/packages/evals/README.md new file mode 100644 index 000000000..495e078f3 --- /dev/null +++ b/packages/evals/README.md @@ -0,0 +1,133 @@ +
+ +voltagent + + +

+AI Agent Engineering Platform +

+ +
+ Home Page | + Documentation | + Examples +
+
+ +
+ +
+ +[![GitHub issues](https://img.shields.io/github/issues/voltagent/voltagent)](https://github.com/voltagent/voltagent/issues) +[![GitHub pull requests](https://img.shields.io/github/issues-pr/voltagent/voltagent)](https://github.com/voltagent/voltagent/pulls) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) +[![npm version](https://img.shields.io/npm/v/@voltagent/evals.svg)](https://www.npmjs.com/package/@voltagent/evals) +[![npm downloads](https://img.shields.io/npm/dm/@voltagent/evals.svg)](https://www.npmjs.com/package/@voltagent/evals) +[![Discord](https://img.shields.io/discord/1361559153780195478.svg?label=&logo=discord&logoColor=ffffff&color=7389D8&labelColor=6A7EC2)](https://s.voltagent.dev/discord) + +
+ +## @voltagent/evals + +Experiment orchestration utilities for VoltAgent. Define a dataset, a runner, and a set of scorers with `createExperiment`, then execute them with `runExperiment` to get aggregated pass/fail and score summaries — locally, in CI, or against a VoltOps-managed dataset. + +--- + +## Install + +```bash +npm install @voltagent/core @voltagent/evals @voltagent/scorers @voltagent/sdk +# or +yarn add @voltagent/core @voltagent/evals @voltagent/scorers @voltagent/sdk +# or +pnpm add @voltagent/core @voltagent/evals @voltagent/scorers @voltagent/sdk +``` + +## Usage + +Define an experiment with `createExperiment`: + +```typescript +import { Agent } from "@voltagent/core"; +import { createExperiment } from "@voltagent/evals"; + +const supportAgent = new Agent({ + name: "support-agent", + instructions: "You are a helpful assistant that answers questions concisely and accurately.", + model: "openai/gpt-4o-mini", +}); + +export const experiment = createExperiment({ + id: "offline-smoke", + label: "Offline Regression Smoke Test", + description: "Demonstrates createExperiment + runExperiment without VoltOps connectivity.", + dataset: { + name: "support-qa", + items: [ + { input: "How do I reset my password?", expected: "Use the reset link on the login page." }, + ], + // To use a VoltOps-managed dataset instead, create one at https://console.voltagent.dev/evals/datasets + }, + runner: async ({ item }) => { + const result = await supportAgent.generateText(item.input); + return { output: result.text }; + }, + scorers: [], // see "Scorers" below + passCriteria: { + type: "meanScore", + min: 0.5, + }, +}); +``` + +Then run it with `runExperiment`: + +```typescript +import { runExperiment } from "@voltagent/evals"; +import { experiment } from "./experiment"; + +const result = await runExperiment(experiment, { + onProgress: ({ completed, total }) => { + console.log(`Processed ${completed}/${total ?? "?"} items`); + }, +}); + +console.log({ + success: result.summary.successCount, + failures: result.summary.failureCount, + errors: result.summary.errorCount, + meanScore: result.summary.meanScore, + passRate: result.summary.passRate, +}); +``` + +### `RunExperimentOptions` + +| Option | Type | Description | +| --------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------- | +| `concurrency` | `number` | Maximum number of dataset items to process in parallel | +| `signal` | `AbortSignal` | Abort the experiment run | +| `voltOpsClient` | client instance | Used to resolve VoltOps-managed datasets and report results to VoltOps | +| `onItem` | `(event) => void \| Promise` | Called after each dataset item finishes, with its result and score | +| `onProgress` | `(event: { completed: number; total?: number }) => void \| Promise` | Called after each dataset item finishes, with overall progress | + +## Scorers + +Pass scorer definitions from [`@voltagent/scorers`](https://github.com/VoltAgent/voltagent/tree/main/packages/scorers) (or your own custom scorers) via `scorers` on `createExperiment`. Each scorer receives the runner's input/output and returns a score that feeds into the experiment summary and `passCriteria`. + +## VoltOps Integration + +If a `voltOpsClient` is provided (either via `runExperiment` options or `config.voltOps.client`), datasets can be loaded from and experiment runs reported to your [VoltOps](https://console.voltagent.dev) project. + +## Example + +See [`examples/with-offline-evals`](https://github.com/VoltAgent/voltagent/tree/main/examples/with-offline-evals) for a complete, runnable experiment with a local dataset and scorers. + +## Documentation + +- [VoltAgent Documentation](https://voltagent.dev/docs/) +- [Evals Overview](https://voltagent.dev/docs/evals/overview/) + +## License + +Licensed under the MIT License, Copyright © 2026-present VoltAgent.