Skip to content

Commit a989617

Browse files
authored
feat: add requestTransform for deterministic matching and recording (#79)
## Summary Adds `requestTransform` to `MockServerOptions` — a function that normalizes requests before both matching and recording, solving the problem of dynamic data (timestamps, UUIDs, session IDs) causing fixture mismatches on replay. Full credit to @iskhakovt for the patch. ```typescript const mock = new LLMock({ requestTransform: (req) => ({ ...req, messages: req.messages.map(m => ({ ...m, content: typeof m.content === "string" ? m.content.replace(/\d{4}-\d{2}-\d{2}T[\d:.+Z]+/g, "") : m.content, })), }), }); ``` **Recording**: saves the transformed match key — no timestamps in fixture. **Matching**: transforms incoming request before comparison — same clean key. When `requestTransform` is set, string matching switches from `includes` (substring) to `===` (exact equality). Without a transform, existing `includes` behavior is preserved (backward compatible). ## Changes - `types.ts`: Add `requestTransform` to `MockServerOptions` and `HandlerDefaults` - `router.ts`: Optional 4th param on `matchFixture`, exact match when transform set - `server.ts`: Thread transform into defaults - `recorder.ts`: Apply transform before match key extraction - All 15 provider handlers: Pass `defaults.requestTransform` to `matchFixture` - `docs/record-replay.html`: Document the feature with use case and example - 16 new tests covering basic transform, exact vs includes, backward compat, embeddings, predicates, streaming, tool calls ## Attribution Based on the design by @iskhakovt in #63, rebuilt cleanly on the 1.7.0 codebase. The original PR had 100 commits and was conflicting after the v1.7.0 rebrand — this is a fresh implementation of the same concept. ## Test plan - [x] `npx vitest run` — 2064 tests pass (58 files) - [x] `npx tsc --noEmit` — clean - [x] Backward compatible — no transform = existing `includes` behavior unchanged
2 parents 7aac519 + aafe82a commit a989617

17 files changed

Lines changed: 583 additions & 36 deletions

docs/record-replay.html

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -355,6 +355,55 @@ <h2>CI Pipeline Workflow</h2>
355355
run: docker stop aimock</code></pre>
356356
</div>
357357

358+
<h2>Request Transform</h2>
359+
<p>
360+
Prompts often contain dynamic data &mdash; timestamps, UUIDs, session IDs &mdash; that
361+
changes between runs. This causes fixture mismatches on replay because the recorded key no
362+
longer matches the live request. The <code>requestTransform</code> option normalizes
363+
requests before both matching and recording, stripping out the volatile parts.
364+
</p>
365+
366+
<div class="code-block">
367+
<div class="code-block-header">
368+
Strip timestamps before matching <span class="lang-tag">ts</span>
369+
</div>
370+
<pre><code>import { LLMock } from "@copilotkit/aimock";
371+
372+
const mock = new LLMock({
373+
requestTransform: (req) => ({
374+
...req,
375+
messages: req.messages.map((m) => ({
376+
...m,
377+
content:
378+
typeof m.content === "string"
379+
? m.content.replace(/\d{4}-\d{2}-\d{2}T[\d:.+Z-]+/g, "")
380+
: m.content,
381+
})),
382+
}),
383+
});
384+
385+
// Fixture uses the cleaned key (no timestamp)
386+
mock.onMessage("tell me the weather ", { content: "Sunny" });
387+
388+
// Request with a timestamp still matches after transform
389+
await mock.start();</code></pre>
390+
</div>
391+
392+
<p>
393+
When <code>requestTransform</code> is set, string matching for
394+
<code>userMessage</code> and <code>inputText</code> switches from substring
395+
(<code>includes</code>) to exact equality (<code>===</code>). This prevents shortened keys
396+
from accidentally matching unrelated prompts. Without a transform, the existing
397+
<code>includes</code> behavior is preserved for backward compatibility.
398+
</p>
399+
400+
<p>
401+
The transform is applied in both directions: <strong>recording</strong> saves the
402+
transformed match key (no timestamps in the fixture file), and
403+
<strong>matching</strong> transforms the incoming request before comparison. This means
404+
recorded fixtures and live requests always use the same normalized key.
405+
</p>
406+
358407
<h2>Building Fixture Sets</h2>
359408
<p>A practical workflow for building and maintaining fixture sets:</p>
360409
<ol>
Lines changed: 342 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,342 @@
1+
import { describe, it, expect, afterEach } from "vitest";
2+
import http from "node:http";
3+
import { matchFixture } from "../router.js";
4+
import { LLMock } from "../llmock.js";
5+
import type { ChatCompletionRequest, Fixture } from "../types.js";
6+
7+
// ---------------------------------------------------------------------------
8+
// Helpers
9+
// ---------------------------------------------------------------------------
10+
11+
function makeReq(overrides: Partial<ChatCompletionRequest> = {}): ChatCompletionRequest {
12+
return {
13+
model: "gpt-4o",
14+
messages: [{ role: "user", content: "hello" }],
15+
...overrides,
16+
};
17+
}
18+
19+
function makeFixture(
20+
match: Fixture["match"],
21+
response: Fixture["response"] = { content: "ok" },
22+
): Fixture {
23+
return { match, response };
24+
}
25+
26+
async function httpPost(url: string, body: object): Promise<{ status: number; body: string }> {
27+
return new Promise((resolve, reject) => {
28+
const req = http.request(
29+
url,
30+
{
31+
method: "POST",
32+
headers: { "Content-Type": "application/json" },
33+
},
34+
(res) => {
35+
const chunks: Buffer[] = [];
36+
res.on("data", (c) => chunks.push(c));
37+
res.on("end", () =>
38+
resolve({
39+
status: res.statusCode!,
40+
body: Buffer.concat(chunks).toString(),
41+
}),
42+
);
43+
},
44+
);
45+
req.on("error", reject);
46+
req.write(JSON.stringify(body));
47+
req.end();
48+
});
49+
}
50+
51+
/** Strip ISO timestamps from text content. */
52+
const stripTimestamps = (req: ChatCompletionRequest): ChatCompletionRequest => ({
53+
...req,
54+
messages: req.messages.map((m) => ({
55+
...m,
56+
content:
57+
typeof m.content === "string"
58+
? m.content.replace(/\d{4}-\d{2}-\d{2}T[\d:.+Z-]+/g, "")
59+
: m.content,
60+
})),
61+
});
62+
63+
// ---------------------------------------------------------------------------
64+
// Unit tests — matchFixture with requestTransform
65+
// ---------------------------------------------------------------------------
66+
67+
describe("matchFixture — requestTransform", () => {
68+
it("matches after transform strips dynamic data", () => {
69+
const fixture = makeFixture({ userMessage: "tell me the weather" });
70+
const req = makeReq({
71+
messages: [{ role: "user", content: "tell me the weather 2026-04-02T10:30:00.000Z" }],
72+
});
73+
74+
// Without transform — exact match would fail, but includes works
75+
expect(matchFixture([fixture], req)).toBe(fixture);
76+
77+
// With transform — also matches (exact match against stripped text)
78+
const transformedFixture = makeFixture({ userMessage: "tell me the weather " });
79+
expect(matchFixture([transformedFixture], req, undefined, stripTimestamps)).toBe(
80+
transformedFixture,
81+
);
82+
});
83+
84+
it("uses exact equality (===) when transform is provided", () => {
85+
// Fixture matches a substring — without transform, includes would match
86+
const fixture = makeFixture({ userMessage: "hello" });
87+
const req = makeReq({
88+
messages: [{ role: "user", content: "hello world" }],
89+
});
90+
91+
// Without transform — includes matches
92+
expect(matchFixture([fixture], req)).toBe(fixture);
93+
94+
// With transform (identity) — exact match fails because "hello world" !== "hello"
95+
const identity = (r: ChatCompletionRequest): ChatCompletionRequest => r;
96+
expect(matchFixture([fixture], req, undefined, identity)).toBeNull();
97+
});
98+
99+
it("exact match succeeds when text matches precisely", () => {
100+
const fixture = makeFixture({ userMessage: "hello world" });
101+
const req = makeReq({
102+
messages: [{ role: "user", content: "hello world" }],
103+
});
104+
105+
const identity = (r: ChatCompletionRequest): ChatCompletionRequest => r;
106+
expect(matchFixture([fixture], req, undefined, identity)).toBe(fixture);
107+
});
108+
109+
it("preserves includes behavior when no transform is provided", () => {
110+
const fixture = makeFixture({ userMessage: "hello" });
111+
const req = makeReq({
112+
messages: [{ role: "user", content: "say hello to me" }],
113+
});
114+
115+
// No transform — includes matching
116+
expect(matchFixture([fixture], req)).toBe(fixture);
117+
});
118+
119+
it("applies transform to inputText (embedding) matching with exact equality", () => {
120+
const fixture = makeFixture({ inputText: "embed this text" });
121+
const req = makeReq({ embeddingInput: "embed this text plus extra" });
122+
123+
// Without transform — includes matches
124+
expect(matchFixture([fixture], req)).toBe(fixture);
125+
126+
// With identity transform — exact match fails
127+
const identity = (r: ChatCompletionRequest): ChatCompletionRequest => r;
128+
expect(matchFixture([fixture], req, undefined, identity)).toBeNull();
129+
130+
// With identity transform — exact match succeeds
131+
const exactFixture = makeFixture({ inputText: "embed this text plus extra" });
132+
expect(matchFixture([exactFixture], req, undefined, identity)).toBe(exactFixture);
133+
});
134+
135+
it("regex matching still works with transform", () => {
136+
const fixture = makeFixture({ userMessage: /weather/i });
137+
const req = makeReq({
138+
messages: [{ role: "user", content: "tell me the weather 2026-04-02T10:30:00.000Z" }],
139+
});
140+
141+
// Regex always uses .test(), not exact match
142+
expect(matchFixture([fixture], req, undefined, stripTimestamps)).toBe(fixture);
143+
});
144+
145+
it("predicate receives original (untransformed) request", () => {
146+
let receivedContent: string | null = null;
147+
const fixture = makeFixture({
148+
predicate: (r) => {
149+
const msg = r.messages.find((m) => m.role === "user");
150+
receivedContent = typeof msg?.content === "string" ? msg.content : null;
151+
return true;
152+
},
153+
});
154+
155+
const originalContent = "hello 2026-04-02T10:30:00.000Z";
156+
const req = makeReq({
157+
messages: [{ role: "user", content: originalContent }],
158+
});
159+
160+
matchFixture([fixture], req, undefined, stripTimestamps);
161+
// Predicate should see the original request, not the transformed one
162+
expect(receivedContent).toBe(originalContent);
163+
});
164+
165+
it("transform applies to model matching", () => {
166+
const fixture = makeFixture({ model: "cleaned-model" });
167+
const req = makeReq({ model: "original-model" });
168+
169+
const modelTransform = (r: ChatCompletionRequest): ChatCompletionRequest => ({
170+
...r,
171+
model: "cleaned-model",
172+
});
173+
174+
expect(matchFixture([fixture], req, undefined, modelTransform)).toBe(fixture);
175+
});
176+
177+
it("identity transform does not break tool call matching", () => {
178+
const fixture = makeFixture({ toolName: "get_weather" });
179+
const req = makeReq({
180+
tools: [
181+
{
182+
type: "function",
183+
function: { name: "get_weather", description: "Get weather" },
184+
},
185+
],
186+
});
187+
188+
const identity = (r: ChatCompletionRequest): ChatCompletionRequest => r;
189+
expect(matchFixture([fixture], req, undefined, identity)).toBe(fixture);
190+
});
191+
192+
it("identity transform does not break toolCallId matching", () => {
193+
const fixture = makeFixture({ toolCallId: "call_123" });
194+
const req = makeReq({
195+
messages: [
196+
{ role: "user", content: "hi" },
197+
{ role: "tool", content: "result", tool_call_id: "call_123" },
198+
],
199+
});
200+
201+
const identity = (r: ChatCompletionRequest): ChatCompletionRequest => r;
202+
expect(matchFixture([fixture], req, undefined, identity)).toBe(fixture);
203+
});
204+
205+
it("sequenceIndex still works with transform", () => {
206+
const fixture = makeFixture({ userMessage: "cleaned", sequenceIndex: 1 });
207+
const req = makeReq({
208+
messages: [{ role: "user", content: "cleaned" }],
209+
});
210+
211+
const identity = (r: ChatCompletionRequest): ChatCompletionRequest => r;
212+
const counts = new Map<Fixture, number>();
213+
214+
// First call (count 0) — sequenceIndex 1 should not match
215+
expect(matchFixture([fixture], req, counts, identity)).toBeNull();
216+
217+
// Simulate count increment
218+
counts.set(fixture, 1);
219+
expect(matchFixture([fixture], req, counts, identity)).toBe(fixture);
220+
});
221+
});
222+
223+
// ---------------------------------------------------------------------------
224+
// Integration tests — LLMock server with requestTransform
225+
// ---------------------------------------------------------------------------
226+
227+
let mock: LLMock | null = null;
228+
229+
afterEach(async () => {
230+
if (mock) {
231+
await mock.stop();
232+
mock = null;
233+
}
234+
});
235+
236+
describe("LLMock server — requestTransform", () => {
237+
it("matches fixture after transform strips timestamps from request", async () => {
238+
mock = new LLMock({
239+
requestTransform: stripTimestamps,
240+
});
241+
242+
// Fixture expects the cleaned message (no timestamp)
243+
mock.onMessage("tell me the weather ", { content: "It will be sunny" });
244+
245+
const url = await mock.start();
246+
247+
const res = await httpPost(`${url}/v1/chat/completions`, {
248+
model: "gpt-4",
249+
messages: [
250+
{
251+
role: "user",
252+
content: "tell me the weather 2026-04-02T10:30:00.000Z",
253+
},
254+
],
255+
});
256+
257+
expect(res.status).toBe(200);
258+
const body = JSON.parse(res.body);
259+
expect(body.choices[0].message.content).toBe("It will be sunny");
260+
});
261+
262+
it("uses exact equality with transform — prevents false positive substring matches", async () => {
263+
mock = new LLMock({
264+
requestTransform: (req) => req, // identity
265+
});
266+
267+
// "hello" is a substring of "hello world" — but with transform,
268+
// exact match is used, so this should NOT match
269+
mock.onMessage("hello", { content: "should not match" });
270+
mock.onMessage("hello world", { content: "correct match" });
271+
272+
const url = await mock.start();
273+
274+
const res = await httpPost(`${url}/v1/chat/completions`, {
275+
model: "gpt-4",
276+
messages: [{ role: "user", content: "hello world" }],
277+
});
278+
279+
expect(res.status).toBe(200);
280+
const body = JSON.parse(res.body);
281+
expect(body.choices[0].message.content).toBe("correct match");
282+
});
283+
284+
it("works without requestTransform — backward compatible includes matching", async () => {
285+
mock = new LLMock();
286+
287+
mock.onMessage("hello", { content: "matched via includes" });
288+
289+
const url = await mock.start();
290+
291+
const res = await httpPost(`${url}/v1/chat/completions`, {
292+
model: "gpt-4",
293+
messages: [{ role: "user", content: "say hello to everyone" }],
294+
});
295+
296+
expect(res.status).toBe(200);
297+
const body = JSON.parse(res.body);
298+
expect(body.choices[0].message.content).toBe("matched via includes");
299+
});
300+
301+
it("transform works with streaming responses", async () => {
302+
mock = new LLMock({
303+
requestTransform: stripTimestamps,
304+
});
305+
306+
mock.onMessage("weather ", { content: "sunny" });
307+
308+
const url = await mock.start();
309+
310+
const res = await httpPost(`${url}/v1/chat/completions`, {
311+
model: "gpt-4",
312+
stream: true,
313+
messages: [{ role: "user", content: "weather 2026-01-01T00:00:00Z" }],
314+
});
315+
316+
expect(res.status).toBe(200);
317+
// Streaming responses have SSE format — just verify it returned 200
318+
expect(res.body).toContain("sunny");
319+
});
320+
321+
it("transform works with embedding requests", async () => {
322+
mock = new LLMock({
323+
requestTransform: (req) => ({
324+
...req,
325+
embeddingInput: req.embeddingInput?.replace(/\d{4}-\d{2}-\d{2}T[\d:.+Z-]+/g, ""),
326+
}),
327+
});
328+
329+
mock.onEmbedding("embed this ", { embedding: [0.1, 0.2, 0.3] });
330+
331+
const url = await mock.start();
332+
333+
const res = await httpPost(`${url}/v1/embeddings`, {
334+
model: "text-embedding-3-small",
335+
input: "embed this 2026-04-02T10:30:00Z",
336+
});
337+
338+
expect(res.status).toBe(200);
339+
const body = JSON.parse(res.body);
340+
expect(body.data[0].embedding).toEqual([0.1, 0.2, 0.3]);
341+
});
342+
});

0 commit comments

Comments
 (0)