feat(eve): replace Workflow with durable code mode - #3002
Conversation
a13ba1c to
5a2b9cd
Compare
Bundle + Package Summary:
|
| Area | Metric | Baseline | Current | Delta |
|---|---|---|---|---|
| Package | Packed tarball | 8.65 MB | 8.65 MB | +2.6 kB |
| Package | Unpacked publish size | 32.59 MB | 32.60 MB | +8.5 kB |
| Package | Installed footprint | 77.67 MB | 77.68 MB | +8.5 kB |
| Package | Published files | 3847 | 3845 | -2 |
| Package | Installed files | 7745 | 7743 | -2 |
| Package | Installed package instances | 33 | 33 | 0 |
| Package | Distinct installed package names | 32 | 32 | 0 |
| Package | Installed dependency edges | 51 | 51 | 0 |
| Package | Installed optional peer edges | 7 | 7 | 0 |
| Runtime | Unique function payloads | 2 | 2 | 0 |
| Runtime | Total function bytes | 20.72 MB | 21.10 MB | +380.3 kB |
| Runtime | Public routes | 18 | 18 | 0 |
Changed function payloads vs main (4b5fad4) (2)
| Function | Status | Baseline | Current | Delta | Route changes |
|---|---|---|---|---|---|
functions/__server.func |
changed | 10.36 MB | 10.55 MB | +190.1 kB |
none |
functions/.well-known/workflow/v1/flow.func |
changed | 10.36 MB | 10.55 MB | +190.1 kB |
none |
eve init install
| Metric | Baseline | Current | Delta |
|---|---|---|---|
| Installed footprint | 116.11 MB | 116.12 MB | +8.5 kB |
| Installed packages | 113 | 113 | 0 |
| dependencies | 4 | 4 | 0 |
| devDependencies | 2 | 2 | 0 |
| Dependency package bytes | 48.72 MB | 48.73 MB | +8.5 kB |
| devDependency package bytes | 5.04 MB | 5.04 MB | 0 B ➖ |
Build Metadata
- Preset:
vercel - Nitro:
nitro@3.0.260903-beta - Output directory:
apps/fixtures/weather-agent/.vercel/output - Build metadata timestamp: 2026-09-07T03:54:09.218Z
- Route aliases: 18 public, 1 internal (19 total aliases)
- Vercel routes in config: 21
- Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small
Package Drill-Down
Package Details
- Package:
eve@0.52.2 - Package directory:
packages/eve - Tarball: 8.65 MB (
eve-0.52.2.tgz) - Unpacked payload: 32.60 MB across 3845 published files
- Installed footprint: 77.68 MB across 7743 installed files
- Installed root package: 31.17 MB
- Installed dependencies: 46.51 MB
- Installed package instances: 33
- Distinct installed package names: 32
- Installed dependency edges: 51
- Installed optional peer edges: 7
- Runtime dependencies: 2
- Peer dependencies: 5 (4 optional)
Installed footprint is measured from an isolated temporary npm install of the packed tarball.
Graph metrics read only package.json files in package directories directly beneath a node_modules boundary, including nested boundaries. Each directory is one package instance; distinct names come from those manifests. Dependency edges count each unique name in dependencies or optionalDependencies per instance; optional peer edges count peerDependencies marked optional.
Heavy installed dependencies
eve: 31.17 MB (40.1%)@rolldown/binding-linux-x64-gnu: 19.31 MB (24.9%)ai: 7.01 MB (9.0%)zod: 6.41 MB (8.3%)undici: 3.51 MB (4.5%)
Publish payload breakdown
Published file size
🔴 dist/src/compiled/shadcn-registry/index.js [#############...........] 9.76 MB 29.9%
🟠 dist/src/compiled/@photon-ai/chat-adapter-ime... [###.....................] 2.27 MB 7.0%
🟠 dist/src/compiled/@ai-sdk/code-mode/index.js [#.......................] 1.03 MB 3.1%
🟡 dist/src/compiled/@vercel/blob/index.js [#.......................] 901.4 kB 2.8%
🟡 dist/src/compiled/_chunks/workflow/signal-exi... [#.......................] 514.6 kB 1.6%
🔴 Other published files [########################] 18.14 MB 55.6%
Installed footprint breakdown
Installed package size
🔴 eve [########################] 31.17 MB 40.1%
🔴 @rolldown/binding-linux-x64-gnu [###############.........] 19.31 MB 24.9%
🔴 ai [#####...................] 7.01 MB 9.0%
🔴 zod [#####...................] 6.41 MB 8.3%
🟠 undici [###.....................] 3.51 MB 4.5%
🟠 nitro [#.......................] 1.89 MB 2.4%
🔴 Other installed packages [######..................] 8.36 MB 10.8%
Runtime dependencies (2)
| Package | Range | Notes |
|---|---|---|
nitro |
3.0.260903-beta |
|
undici |
8.9.0 |
Peer dependencies (5)
| Package | Range | Notes |
|---|---|---|
@opentelemetry/api |
^1.0.0 |
optional peer |
ai |
catalog: |
|
braintrust |
^3.0.0 |
optional peer |
just-bash |
^3.1.0 |
optional peer |
microsandbox |
^0.5.0 |
optional peer |
eve init install drill-down
eve init install details
- Command:
eve init my-agent - Package manager:
npm - Installed footprint: 116.12 MB across 9639 installed files
- Installed packages: 113 total (107 transitive-only)
- dependencies: 4 direct packages totaling 48.73 MB
- devDependencies: 2 direct packages totaling 5.04 MB
- Other transitive package files: 62.35 MB
Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.
Heavy installed dependencies
eve: 31.17 MB (26.8%)@typescript/typescript-linux-x64: 27.95 MB (24.1%)@rolldown/binding-linux-x64-gnu: 19.31 MB (16.6%)zod: 10.37 MB (8.9%)ai: 7.01 MB (6.0%)
Installed footprint breakdown
Installed package size
🔴 eve [########################] 31.17 MB 26.8%
🔴 @typescript/typescript-linux-x64 [######################..] 27.95 MB 24.1%
🔴 @rolldown/binding-linux-x64-gnu [###############.........] 19.31 MB 16.6%
🔴 zod [########................] 10.37 MB 8.9%
🔴 ai [#####...................] 7.01 MB 6.0%
🟠 undici [###.....................] 3.51 MB 3.0%
🔴 Other installed packages [#############...........] 16.79 MB 14.5%
dependencies (4)
| Package | Range | Installed size | Share |
|---|---|---|---|
@vercel/connect |
1.0.0 |
167.9 kB | 0.1% |
ai |
^7.0.82 |
7.01 MB | 6.0% |
eve |
file:eve-0.52.2.tgz |
31.17 MB | 26.8% |
zod |
4.5.4 |
10.37 MB | 8.9% |
devDependencies (2)
| Package | Range | Installed size | Share |
|---|---|---|---|
@types/node |
24.x |
2.54 MB | 2.2% |
typescript |
7.0.2 |
2.50 MB | 2.2% |
Function Drill-Down
Payload Size Graph
Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func [########################] 10.55 MB 50.0%
🔴 functions/__server.func [########################] 10.55 MB 50.0%
Top Function Payloads
🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 10.55 MB
| Metric | Value |
|---|---|
| Public routes | /.well-known/workflow/v1/flow |
| Runtime | nodejs24.x |
| Handler | index.mjs |
| Payload | 10.55 MB |
| Function files | 10.55 MB across 116 files |
| Traced dependencies | 0 B |
| Signal | 🟠 Bundled file index.mjs is 2.02 MB (19.2%) |
🟠 🔎 Dependency Analysis
📦 Bundled files:
Bundled file size
🟠 index.mjs [#########...............] 2.02 MB 19.2%
🟡 _libs/undici.mjs [####....................] 980.8 kB 9.3%
🟡 _chunks/esm-Fqlolk7e.mjs [###.....................] 723.3 kB 6.9%
🟡 _chunks/chatgpt-model.mjs [###.....................] 693.8 kB 6.6%
🟡 _chunks/signal-exit-Dsy-TT0V.mjs [###.....................] 616.2 kB 5.8%
🔴 Other bundled files [########################] 5.51 MB 52.3%
🧾 Vercel Config
{
"handler": "index.mjs",
"launcherType": "Nodejs",
"shouldAddHelpers": false,
"supportsResponseStreaming": true,
"runtime": "nodejs24.x",
"maxDuration": "max",
"experimentalTriggers": [
{
"type": "queue/v2beta",
"topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
"consumer": "default",
"retryAfterSeconds": 5,
"initialDelaySeconds": 0
}
],
"environment": {
"WORKFLOW_PRECONDITION_GUARD": "1"
}
}🟠 functions/__server.func • 17 public routes, 1 internal alias • 10.55 MB
| Metric | Value |
|---|---|
| Public routes | //.well-known/workflow/v1/webhook/[token]/eve/v1/activity/[token]/eve/v1/callback/[token]/eve/v1/connections/[name]/callback/[attemptId]/[token]/eve/v1/connections/[name]/callback/[token]/eve/v1/health/eve/v1/info/eve/v1/session/eve/v1/session/[parentSessionId]/subagents/[callId]/[childSessionId]/stream/eve/v1/session/[sessionId]/eve/v1/session/[sessionId]/cancel/eve/v1/session/[sessionId]/clear/eve/v1/session/[sessionId]/compact/eve/v1/session/[sessionId]/reset/eve/v1/session/[sessionId]/stream/eve/v1/task-input/[token] |
| Internal aliases | /__server |
| Runtime | nodejs24.x |
| Handler | index.mjs |
| Payload | 10.55 MB |
| Function files | 10.55 MB across 116 files |
| Traced dependencies | 0 B |
| Signal | 🟠 Bundled file index.mjs is 2.02 MB (19.2%) |
🟠 🔎 Dependency Analysis
📦 Bundled files:
Bundled file size
🟠 index.mjs [#########...............] 2.02 MB 19.2%
🟡 _libs/undici.mjs [####....................] 980.8 kB 9.3%
🟡 _chunks/esm-Fqlolk7e.mjs [###.....................] 723.3 kB 6.9%
🟡 _chunks/chatgpt-model.mjs [###.....................] 693.8 kB 6.6%
🟡 _chunks/signal-exit-Dsy-TT0V.mjs [###.....................] 616.2 kB 5.8%
🔴 Other bundled files [########################] 5.51 MB 52.2%
🧾 Vercel Config
{
"handler": "index.mjs",
"launcherType": "Nodejs",
"shouldAddHelpers": false,
"supportsResponseStreaming": true,
"runtime": "nodejs24.x"
}Build Timing: e2e/fixtures/agent-tools-sandbox
This is an informational timing measurement inside eve build, from preflight through publication. Output-size measurement and profile writing are excluded.
Build mode: deployable Vercel build with sandbox template prewarm included.
- Build pipeline: 6.19 s -> 6.13 s (-59.0 ms) vs
main (4b5fad4). - Timing is informational: shared GitHub runners are too variable for a hard timing budget.
Detailed phase timings vs `main (4b5fad4)`
| Phase | Baseline | Current | Delta |
|---|---|---|---|
extension.check |
12.4 ms | 0.9 ms | -11.5 ms |
project.resolve |
0.4 ms | 0.4 ms | 0.0 ms |
workspace.create |
0.6 ms | 0.6 ms | 0.0 ms |
host.prepare |
855.0 ms | 671.9 ms | -183.1 ms |
vercel.service-prefix.resolve |
2.6 ms | 2.7 ms | +0.1 ms |
nitro.create |
572.7 ms | 618.8 ms | +46.1 ms |
sandbox.prewarm |
250.6 ms | 293.7 ms | +43.1 ms |
nitro.cache.prepare |
0.3 ms | 0.3 ms | 0.0 ms |
nitro.prepare |
0.9 ms | 0.8 ms | -0.1 ms |
nitro.public-assets |
0.8 ms | 0.8 ms | 0.0 ms |
nitro.prerender |
0.5 ms | 0.5 ms | 0.0 ms |
nitro.bundle |
4.44 s | 4.48 s | +41.5 ms |
nitro.cache.write |
0.3 ms | 0.4 ms | +0.1 ms |
vercel.workflow-function.materialize |
47.3 ms | 51.7 ms | +4.4 ms |
agent-summary.emit |
0.7 ms | 0.7 ms | 0.0 ms |
nitro.close |
0.2 ms | 0.1 ms | -0.1 ms |
output.publish |
3.4 ms | 3.7 ms | +0.3 ms |
workspace.remove |
2.0 ms | 2.3 ms | +0.3 ms |
5a2b9cd to
1e4f5a9
Compare
`experimental.codeMode` moves eligible tools behind one framework `code_mode` tool whose execute body is a workflow. The generated program runs in the code-mode sandbox with every claimed tool replaced by an interrupt stub, so the body parks at each nested call and resumes it on its own step boundary: ordinary tools execute in a child step over the turn's serialized context; subagent calls go through the owner's `agent-invoke` channel like any workflow tool, so a program can launch and await subagents while the parent keeps sole ownership of agent handles and session state. The parent materializes the sandbox before dispatching `code_mode` so the child reconnects to the recorded session instead of creating a second one. The claimed catalog is pinned into the durable input at advertisement time. Adding `codeMode` to `AgentExperimentalDefinition` bumps the subagent extension epoch to 9; epoch 8 stays supported. Signed-off-by: Rui Conti <ruiconti@gmail.com>
A `Promise.all` in a generated program parks every call in one sandbox continuation. The body previously took the first pending call, settled it, and resumed — so three subagents the model meant to run in parallel ran one after another. `runCodeModeProgramStep` now returns the whole pending batch; the body settles them together (owner `agent-invoke` requests and child tool steps in flight at once) and hands all resolutions back in one resume. Resuming feeds each resolution the interrupt produced by the previous `continue`, since the signed ledger advances per resolution; the sandbox only re-runs the program after the last one. Replayed stubs return the settled value from `codeModeInterrupt.resolution` instead of raising again, which is what lets the program advance past the park point. An integration test pins the sandbox contract this relies on: a batch parks together, intermediate continues do not run the program, resolutions are rejected for the wrong interrupt or a drifted tool catalog, and the program observes every value exactly once. Signed-off-by: Rui Conti <ruiconti@gmail.com>
Add the `agent-code-mode` fixture: a deterministic mock script drives one `code_mode` call per directive and echoes its result. Evals cover the model surface (claimed tools leave it, approval-gated ones stay), a single nested call, a chain that threads one call's output into the next, and a `Promise.all` over two subagents and a tool that asserts both children start before `code_mode` resolves and every value comes back. Running it exposed two wiring gaps. Coordination read the pre-advertisement harness map, so the workflow task never carried code mode's pinned catalog and the body rejected its input; the tool loop now hands coordination the map `getAdvertisedTools` returned. Declared subagents are background tools on the direct surface, so the "blocking only" rule excluded every real subagent from programs; any `resultKind: "subagent"` tool is now claimable, since the body awaits it through the owner channel regardless. Child tool steps now receive the turn's message history in their execute options, matching what a direct call sees. Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Signed-off-by: Rui Conti <ruiconti@gmail.com>
0c77a26 to
ed2894a
Compare
Signed-off-by: Rui Conti <ruiconti@gmail.com>
Summary
Tool orchestration needs to await subagents and resume completed work after a crash. Previously, the experimental
Workflowwrapper only orchestrated agents; this replaces it with a root-onlycode_modeworkflow tool that discovers the full advertised catalog and invokes eligible tools and subagents. Enable it withexperimental.codeMode: { mode: "eager" | "lazy", maxSubagents?: number }. Related to #2508.Programs pin their catalog and subagent budget when dispatched. Ordinary calls execute in separate durable steps; subagent calls use the owning session's
agent-invokechannel and await the child result. Calls issued together settle concurrently. Authorization can park and resume a call, and individual failures remain catchable by the program.requiresDirectCall.maxSubagentschanges the limit.experimental_workflow()fromeve/tools/workflow, then enable code mode inagent.ts. The old helper and harness-specific Workflow continuation machinery are removed. Authored"use workflow"tools remain supported.code_modewrapper.Programs previously ran guest
catch/finallyhandlers while parking a tool call, and deterministic program failures consumed workflow retries. The pinned Run patch now treats suspension as host control flow and marks guest-source errors separately from infrastructure errors. eve returns known program failures from the step as data, then reports the tool error to the model so it can revise the source. The one-program orchestration sentence is restored; eager still exposes eligible direct tools. The vendoring cache now fingerprints the installed worker so a same-version patch rebuilds the shipped runtime. Readable patch and reproduction.Scripted Code Mode E2Es cover execution contracts. The real-model eager/lazy evals check complete pagination, overlapping independent reads, handling an unavailable source, and correct saved reports. They log program grouping and repeated page reads for diagnosis; they do not require a single program or benchmark savings against direct tools.
Optional tool arguments now retain their omission semantics at the provider boundary. eve explicitly disables implicit strict-mode normalization and still validates inputs before execution.
Validation
Rebased onto
4b5fad41bated2894a46: all 47 workspace build/typecheck tasks, 141 focused unit tests, 13 compiled-SDK integration tests, and the installed-extension scenario pass. Frozen installation, formatting/lint, docs and invariant checks pass. Published compatibility reports are preserved: new tool epoch 31 and dynamic-tool epoch 30 replace the removed Workflow contracts; subagent epoch 10 retains main's epoch 9. The new steering fixture now imports the cancellation tool from its own fixture after the sleeper fixture moved. GitHub reports no merge conflicts; required CI checks are pending.Suspension/recovery repair at
0c77a2645: 63 focused eve unit tests, 13 compiled-SDK integration tests, and the packaged Code Mode worker-assets scenario pass. The underlying Run repair passes all 336 tests and its typechecks. eve runtime build, package and scripted-fixture typechecks, changed-file formatting/lint, docs checks and invariant checks pass. Independent source review found no blocking lifecycle/classification defect. The same compiled-SDK catch probe that timed out at 30s on the baseline now completes in 16.7ms; catch/finally and the exact 120,418-byte payload with eight distinct leaf calls pass. This is an in-process interruption/resolution probe, not a full workflow-crash or model-performance benchmark. Two scripted E2Es were added; CI is running on the rebased head.Optional-argument fix: 41 tool unit tests and three input-validation integration tests pass; the new provider-boundary regression failed before the fix. Package and fixture typechecks, lint, docs, invariant checks, and the runtime build pass. The four-call Terra probe produced 0/2 valid inputs with
strictomitted and 2/2 withstrict: false. A real-model fixture eval was added; no eve test workflow was recorded for the benchmark baseline143b0e5a7.pnpm --filter eve exec vitest run --config vitest.unit.config.ts --maxWorkers=4: 7,985 passed, one skipped. The final child-visibility test adjustment also passed its focused harness run.Targeted integration runs: 40 passed across code-mode sandbox/connection execution, turn and session-limit cancellation, cancelled-turn settlement, and authored workflow tools.
Targeted packaged scenarios: seven passed, covering local/remote child continuation, cancellation, code-mode compilation and sandbox assets, and installed extension compatibility.
pnpm --filter eve build:js, packagetsc --noEmit, changed-file formatting/lint,pnpm guard:invariants, andpnpm docs:checkpassed.pnpm --filter agent-code-mode --filter agent-cancellation --filter agent-subagents typecheckpassed, including fixture builds. E2E evals were added/migrated and are left to CI.New planning fixtures: both builds and typechecks passed with
EVE_E2E_MODEL=mock; five assertion unit tests and two audit scenarios passed. Matrix discovery includes both fixtures on both registered models. Live-model eval results are pending CI; no E2E suite was run locally.After rebasing onto
eb5949a62: frozen installation, all 46 workspace build/typecheck tasks, invariant checks, changed-file formatting/lint, and docs checks passed. Focused coverage passed: 68 compiler/Code Mode unit tests, 16 Code Mode integration tests, the installed-extension scenario, and seven planning-fixture tests. Published extension history is preserved; the tool and dynamic-tool contracts now use epoch 28.Benchmark diagnosis: GPT-5.6 Terra, 2026-09-06
At baseline
143b0e5a7, optional eager never selected Code Mode in 20 controlled attempts. Restoring the one-program instruction reduced lazy root-turn time by 33%, with 8/8 correct answers in both conditions. Forcing Code Mode was not uniformly beneficial.The tables below are measurements of that baseline and its explicitly described experimental variants. The repair at
0c77a2645has deterministic regression validation above; it has not been rerun through these model benchmarks.The old Fable pilot and the newer Terra run are not a controlled comparison. The task files are unchanged (594 manifest files verified), but the model, eager tool availability, orchestration wording and runtime artifact changed. The pilot used an uncommitted experimental package with a local suspension fix; the benchmark baseline lacked that repair. This does not identify an upstream commit that introduced the bug.
Controlled selection experiment: 80 attempts. Same Terra model with high reasoning, runtime engine, tools and ten frozen cases; two repetitions across five task shapes and two seeds. Each column has 20 attempts. These are attempt success counts, not best-of-two scores. Every attempt and recovery stays in the timing totals.
Direct disables Code Mode. Optional eager is the implementation at baseline
143b0e5a7. Forced eager hides eligible direct tools. Guidance retains both tool surfaces and repeats the current selection/orchestration text in the main instructions; that intervention changes repetition and placement together. The latter two are experimental variants, not changes included in this PR.Root-turn duration by shape; four attempts per condition in each row:
Optional eager used Code Mode zero times and added 3,673 input tokens to the first request in every first-repetition pair. The tool construction exposes both direct schemas and the Code Mode description. Forced Code Mode won three of four inventory comparisons; the fourth recovered from two syntax errors and took 50.63s versus direct’s 38.93s.
The direct large-task traces emit 32 domain calls within one model step, as separate streamed requests. This already avoids 32 separate model round trips; it does not prove simultaneous network execution. Code Mode can cross dependency boundaries and reduce intermediate data. These local synthetic fixtures do not establish production API latency or a universal fanout threshold.
Orchestration experiment: 24 attempts. All conditions use lazy mode, the same engine and discovery tools, and one/32-project summaries with two seeds and two repetitions. One intervention restores only the first orchestration sentence; the other restores the complete old three-sentence block from 287c074.
Restoring only the first sentence reduced work programs from 23 to 10 and model steps from 41 to 26, with nearly unchanged fixture work (148 versus 150 calls). Root-turn duration fell from 12.83s to 8.62s; seven of eight matched pairs were faster and all answers passed. The full old block averaged 12.23s and recovered from one invalid program after four workflow attempts (3.177s of failed-action time, retained). This supports a small orchestration change; it does not solve eager selection. The first-sentence intervention says to complete the task in one execution program and retains the current reuse instruction. Restoring the full block also adds discovery/retry wording and removes that reuse sentence; its individual clauses are not isolated. Two repetitions on fixed cases establish observed behavior, not a universal policy or statistical significance.
Confirmed failures and remaining defects. One direct answer was correct but the grader extracted an array instead of an object; its correction is reported separately. Forced and guidance each failed one summary by aggregating only the first 100 of 725 global domains. Native replay of those programs reproduces the wrong result, so these are agent pagination mistakes.
The two invalid programs in the recovered inventory trial each received four workflow attempts. Their failed-action intervals totaled 6.482s, including the first attempts and retry delays. The repair at
0c77a2645returns known deterministic program failures to the model without repeating unchanged source. The old experimental error classifier did not cover theRUN_ERRORreturned by these syntax errors, so restoring that classifier alone is insufficient.A separate suspension defect reproduced in the baseline packaged SDK: certain try/catch or finally programs time out before a leaf tool executes. The catch reproducer timed out at 30s in the baseline package and completes in 17ms with the pilot’s repaired package; ordinary execution controls pass. None of the 55 Code Mode programs in the 80-trial selection run uses that pattern, and no timeout occurs there. The repair above addresses this defect; it was not the cause of those observed timings.
Measurement and provenance. Agent lifecycle includes startup and cleanup; root turn excludes both. Cleanup averages roughly 5.4s. Environment setup, building and grading are excluded from both durations. Input totals include cached context; output totals include reasoning. Provider caching stays enabled. Trials run sequentially with shuffled case order and rotated condition order; there are no full-trial retries. Internal workflow retries remain measured.
Recorded model + grader spend for these 104 attempts: $3.4635 ($3.0398 selection; $0.4237 orchestration). Combined recorded-spend stop $5; per-session stop $0.25. Stops are checked after charges, not hard billing caps.
Runtime commit:
143b0e5a7dc8a7335063b6d9dc2c3b54845cb1f9. Base archive SHA-256:93a537f044ea2a45808b5e539e812c376b21bebf2db44a72a11045711f327bc8. Dataset SHA-256:107296bae88bfad6b6e26b4aa8f4047f479702e2fe88ce8b54d711d93eafecd5. Harbor:2fe1615503fed39ad82b7ce09b22497996b30f1f. Saved runtime archives, dependency locks, instructions, agent definitions and adapters match each experiment’s manifest. Raw grades remain unchanged.Earlier full 18-case Terra snapshot at the same commit
This snapshot has one attempt per case/mode; its aggregate covers a different case mix from the controlled ten-case diagnostic above.
The lazy false negative was an extracted status label. Total recorded model + grader spend was $1.9670, separate from the 104-attempt diagnostic. The earlier Mini run had an invalid direct control and cannot establish a Code Mode speedup. The optional-argument schema fix at this commit was separately reproduced and validated; no real-model CI E2E result is claimed here.
Checklist
CONTRIBUTING.mdevepackagegit commit --signoff)Diff size
Docs — 11 files ·
+179 / -25Documents Code Mode configuration, migration, selection guidance, and optional tool arguments.
Implementation — 87 files ·
+2033 / -1299Replaces the Workflow wrapper with durable Code Mode execution and its tool/catalog integration; preserves optional arguments at the provider boundary. Includes the generated Run worker patch and its readable upstream source/test patch.
Tests — 124 files ·
+3987 / -1358Covers runtime contracts, cancellation, tool visibility, real-model planning fixtures, optional-argument validation, suspension and program-error recovery. Counts fixture files with tests; the upstream regression tests embedded in the readable dependency patch are counted with that patch.