Skip to content

Commit 2f39489

Browse files
committed
feat: add requestTransform for deterministic matching and recording
Normalizes requests before matching and recording to handle dynamic data (timestamps, UUIDs, session IDs) in prompts. When set, string matching switches from includes to exact equality to prevent false positives from shortened keys. Applied across all 15 provider handlers + recorder. 16 new tests. Based on the design by @iskhakovt in #63, rebuilt on the 1.7.0 codebase.
1 parent 7aac519 commit 2f39489

19 files changed

+610
-41
lines changed

docs/migrate-from-mokksy.html

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,16 @@ <h2>The quick switch</h2>
154154
<span class="str">"-p"</span>, <span class="str">"4010:4010"</span>, <span class="str">"-v"</span>, <span class="str">"./fixtures:/fixtures"</span>,
155155
<span class="str">"ghcr.io/copilotkit/aimock"</span>, <span class="str">"-f"</span>, <span class="str">"/fixtures"</span>)
156156
.<span class="fn">start</span>().<span class="fn">waitFor</span>()
157+
<span class="cm">// Wait for server to be ready</span>
158+
<span class="kw">repeat</span>(<span class="num">30</span>) {
159+
<span class="kw">try</span> {
160+
<span class="type">java.net.URL</span>(<span class="str">"http://localhost:4010/__aimock/health"</span>)
161+
.<span class="fn">readText</span>()
162+
<span class="kw">return</span>
163+
} <span class="kw">catch</span> (_: <span class="type">Exception</span>) {
164+
<span class="type">Thread</span>.<span class="fn">sleep</span>(<span class="num">200</span>)
165+
}
166+
}
157167
}
158168

159169
<span class="kw">@AfterAll</span>

docs/migrate-from-python-mocks.html

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -248,17 +248,22 @@ <h3>aimock (after)</h3>
248248

249249
<span class="op">@pytest.fixture</span>(scope=<span class="str">"session"</span>)
250250
<span class="kw">def</span> <span class="fn">aimock_server</span>():
251-
<span class="cm"># Start aimock via Docker</span>
252251
proc = subprocess.Popen([
253252
<span class="str">"docker"</span>, <span class="str">"run"</span>, <span class="str">"--rm"</span>,
254253
<span class="str">"-p"</span>, <span class="str">"4010:4010"</span>,
255254
<span class="str">"-v"</span>, <span class="str">f"{os.getcwd()}/fixtures:/fixtures"</span>,
256255
<span class="str">"ghcr.io/copilotkit/aimock:latest"</span>,
257256
<span class="str">"-f"</span>, <span class="str">"/fixtures"</span>
258257
])
259-
time.sleep(<span class="num">2</span>) <span class="cm"># wait for server</span>
258+
<span class="cm"># Wait for health endpoint</span>
259+
<span class="kw">import</span> requests
260+
<span class="kw">for</span> _ <span class="kw">in</span> range(<span class="num">30</span>):
261+
<span class="kw">try</span>:
262+
<span class="kw">if</span> requests.get(<span class="str">"http://localhost:4010/__aimock/health"</span>).ok:
263+
<span class="kw">break</span>
264+
<span class="kw">except</span> requests.ConnectionError:
265+
time.sleep(<span class="num">0.2</span>)
260266

261-
<span class="cm"># Point OpenAI SDK at the mock</span>
262267
os.environ[<span class="str">"OPENAI_BASE_URL"</span>] = <span class="str">"http://localhost:4010/v1"</span>
263268
os.environ[<span class="str">"OPENAI_API_KEY"</span>] = <span class="str">"mock-key"</span>
264269

@@ -489,9 +494,16 @@ <h2>Alternative: npx fixture (no Docker)</h2>
489494
<span class="kw">def</span> <span class="fn">aimock_server</span>():
490495
proc = subprocess.Popen(
491496
[<span class="str">"npx"</span>, <span class="str">"aimock"</span>, <span class="str">"-p"</span>, <span class="str">"4010"</span>, <span class="str">"-f"</span>, <span class="str">"./fixtures"</span>],
492-
stdout=subprocess.PIPE, stderr=subprocess.PIPE
497+
stdout=subprocess.PIPE, stderr=subprocess.STDOUT
493498
)
494-
time.sleep(<span class="num">2</span>) <span class="cm"># wait for server</span>
499+
<span class="cm"># Wait for health endpoint</span>
500+
<span class="kw">import</span> requests
501+
<span class="kw">for</span> _ <span class="kw">in</span> range(<span class="num">30</span>):
502+
<span class="kw">try</span>:
503+
<span class="kw">if</span> requests.get(<span class="str">"http://localhost:4010/__aimock/health"</span>).ok:
504+
<span class="kw">break</span>
505+
<span class="kw">except</span> requests.ConnectionError:
506+
time.sleep(<span class="num">0.2</span>)
495507

496508
os.environ[<span class="str">"OPENAI_BASE_URL"</span>] = <span class="str">"http://localhost:4010/v1"</span>
497509
os.environ[<span class="str">"OPENAI_API_KEY"</span>] = <span class="str">"mock-key"</span>

docs/record-replay.html

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -355,6 +355,55 @@ <h2>CI Pipeline Workflow</h2>
355355
run: docker stop aimock</code></pre>
356356
</div>
357357

358+
<h2>Request Transform</h2>
359+
<p>
360+
Prompts often contain dynamic data &mdash; timestamps, UUIDs, session IDs &mdash; that
361+
changes between runs. This causes fixture mismatches on replay because the recorded key no
362+
longer matches the live request. The <code>requestTransform</code> option normalizes
363+
requests before both matching and recording, stripping out the volatile parts.
364+
</p>
365+
366+
<div class="code-block">
367+
<div class="code-block-header">
368+
Strip timestamps before matching <span class="lang-tag">ts</span>
369+
</div>
370+
<pre><code>import { LLMock } from "@copilotkit/aimock";
371+
372+
const mock = new LLMock({
373+
requestTransform: (req) => ({
374+
...req,
375+
messages: req.messages.map((m) => ({
376+
...m,
377+
content:
378+
typeof m.content === "string"
379+
? m.content.replace(/\d{4}-\d{2}-\d{2}T[\d:.+Z-]+/g, "")
380+
: m.content,
381+
})),
382+
}),
383+
});
384+
385+
// Fixture uses the cleaned key (no timestamp)
386+
mock.onMessage("tell me the weather ", { content: "Sunny" });
387+
388+
// Request with a timestamp still matches after transform
389+
await mock.start();</code></pre>
390+
</div>
391+
392+
<p>
393+
When <code>requestTransform</code> is set, string matching for
394+
<code>userMessage</code> and <code>inputText</code> switches from substring
395+
(<code>includes</code>) to exact equality (<code>===</code>). This prevents shortened keys
396+
from accidentally matching unrelated prompts. Without a transform, the existing
397+
<code>includes</code> behavior is preserved for backward compatibility.
398+
</p>
399+
400+
<p>
401+
The transform is applied in both directions: <strong>recording</strong> saves the
402+
transformed match key (no timestamps in the fixture file), and
403+
<strong>matching</strong> transforms the incoming request before comparison. This means
404+
recorded fixtures and live requests always use the same normalized key.
405+
</p>
406+
358407
<h2>Building Fixture Sets</h2>
359408
<p>A practical workflow for building and maintaining fixture sets:</p>
360409
<ol>

0 commit comments

Comments
 (0)