bug: durable turns 3–6× slower than tab-owned /api/agent (10–20 min → ~1 hour)

[btipling]

Summary

Operator-visible: a coding turn that used to finish in 10–20 minutes on tab-owned POST /api/agent now takes ~1 hour on the durable path (POST /api/turns → turnWorkflow) after the D17 cutover.

Same work (same model, same sandbox tools). The hour is wall-clock tax on the Workflows loop, not slower DeepSeek.

Parent / umbrella: #794
Status: source bug — create-plan is a later turn. This will almost certainly split (≥2 issues) if a plan tries to land reconnect + persist cadence + step-arg size in one PR.

This is not covered by:

• #872 / #880 — per-round tool batch (stopped N writeTurnSse Fluid steps per tool). Live tokens already use one held writer. The remaining hop is still one Function per model round + one per tool batch + one persist per cadence point.
• #884 / #885 — persist oversize killing the turn. Different failure mode.
• #849 — reader EOF folded as Turn ended.

---

Expected

A durable coding turn’s wall clock is dominated by model + tool time, same as /api/agent.

• Workflows step/queue/replay overhead is a small constant, not 3–6× the turn.
• Sandbox / MCP / BYOK / persona / skills are not rebuilt from scratch on every model round and every tool batch.
• Step args stay deltas (architecture lock). The growing transcript does not ride every modelGenerateStep / persistStep call into the event log.
• A 15-minute /api/agent coding turn is still ~15–20 minutes after D17, not ~60.

---

Actual (grounded on main @ 98f1a0f)

D17 (#811 / PR #841) cut runPrompt over to POST /api/turns. /api/agent is tests/JSON only.

Before: one Fluid Function, one sandbox/MCP/BYOK setup, in-process model↔tool loop, whole-turn ceiling maxDuration = 1800.

After: turnLoop drives isolated 'use step' Functions. A 20-round coding turn is ~60 Function hops:

Per round	Step
model	modelGenerateStep
first tools round	persistStep (running) — user-line
tools	toolExecuteStep (batch, #880)
after the batch	persistStep (running) again
end	terminal persist + writeTurnSse (done) + closeTurnSse

#794’s cost example (8 rounds + 20 tools + 1 persist = 29 steps) is already stale. Live runTurnLoop persists after the first tools model and after every successful batch.

Grounded seams:

1. Replay between every step (platform). Vercel Workflows suspends the orchestrator and enqueues a replay on every 'use step' (vercel/workflow#1930). plan: backend-agents RESTART — client-independent turns (umbrella, ≥20 small issues + handoff) #794 already flagged replay slowing after 2k events / 1 GB. Observability → Workflows → time between steps is the tell: if the gap ≫ step CPU, this is the bulk of the hour.

2. Full world rebuild on every model and tool step. assembleDurableToolWorld runs inside both modelGenerateStep and toolExecuteStep: decrypt GH token, load HTTP instance, resolveAgentSandbox + new sandbox client, MCP connect, HTTP runner, persona + skills, and BYOK again in the model step. The turns route even says the POST probe client is closed after start() because the in-step assemble helper opens its OWN client per step VM. /api/agent did this once. A 20-round turn now does it ~40 times.

3. Growing transcript stuffed into step args (O(n²) event log). Architecture lock: step I/O is deltas; messages stay orchestrator-local. Live code passes the full reconstructed messages[] into every modelGenerateStep, and the full deltas + fold.checkpoint into every persistStep. Later rounds serialize a bigger conversation into the Workflows event log, then replay it on the next resume. Short turns stay fine; 10–20 min coding turns get slower as they go.

4. Mid-turn persist is its own step, every round. persistOnce(false) after the first tools model and after every batch — Blob PUT + Redis overlay + another replay. Detach/refresh needs some persist; it does not need a Function hop after every round.

5. modelGenerateStep still has default 3 retries. toolExecuteStep.maxRetries = 0 (correct — don’t replay mutations). modelGenerateStep and persistStep do not set it. SDK default is 3 retries. A long streamText that blips or hits a step timeout can re-run the whole LLM round.

6. No whole-turn wall clock. Old path died at 30 min. Workflows only cap per step at 1800s, loop at 512 steps. A turn that used to get killed at 30 min can now run to 60.

writeTurnSse / closeTurnSse are still 'use step'. Live tokens are correctly held inside the model/tool step (withDefaultStreamWriter); loop done/error still pay an extra hop. That is not the old N-writes-per-token bug.

---

How to confirm (one slow Production run)

Vercel → Observability → Workflows, one hour-long coding turn:

1. Step count — expect ~3× model rounds, not ~1×.
2. Time between steps vs step duration — large gaps = replay/queue.
3. assembleDurableToolWorld / sandbox / MCP at the start of every model and tool step.
4. Step arg size growing each persist/model (messages / deltas / checkpoint).
5. invincible.turn.model logs — duplicate rounds with the same shape ⇒ retries.

If (2)+(3) dominate, it is the step tax + reconnect. If later steps get slower, it is (3) the O(n²) args. If a round’s LLM time appears twice, it is (5).

---

Likely seam (for a later plan — do not implement here)

Direction, not a locked design. Highest leverage first:

• Cache sandbox/MCP/BYOK across steps in the same run, or pass a handle instead of reconnecting every model and tool step.
• Persist on a timer / every N rounds / terminal only — not a 'use step' after every batch.
• Pass deltas into steps (as locked); keep messages[] orchestrator-local. Stop sending full deltas into persist — persist can read Blob.
• Set maxRetries = 0 (or 1) on modelGenerateStep.
• Collapse loop done/close into the terminal persist step so they are not extra 'use step' calls.

Do not wrap runAgentStream / streamText+execute in one step (forbidden). Do not cut back to /api/agent. Do not cancel the run on F5. Do not change Wasm/protocol.

If this is too large for one PR (it is), split along the table above. Reconnect (2) and persist cadence (4) are the wall-clock fire; step-arg size (3) is the “gets worse as the turn goes” fire.

---

Not this

• Per-token / per-tool writeTurnSse Fluid steps (already fixed; held writer inside the step)
• Tool parallelism inside a round (reads still Promise.all; only bind-mutators + FS editors are serial)
• The 1s TURN_START_MIN_INTERVAL_MS 429 (start only)
• Persist oversize as a turn-killer (plan: persist/SSE bookkeeping must not kill a useful durable turn (parent) #884 / plan: persist oversize must not fail the turn — omit deltas, trim to ceiling #885)
• Raising or lowering MAX_WORKFLOW_STEPS (512) or maxDuration (1800)
• ⛔ No application code in this issue — source bug. create-plan is a later turn.

---

DoD

• A representative durable coding turn (the class that used to be 10–20 min on /api/agent) is no longer ~1 hour. Wall clock is dominated by model + tool time, not step hops / reconnect / replay.
• Observability on a long run: step count is not ~3× model rounds from persist+assemble hops; time-between-steps is not the majority of the hour.
• assembleDurableToolWorld is not a cold sandbox+MCP handshake on every model and tool step of the same run.
• Step args stay bounded (deltas), not the growing messages[] / full deltas / checkpoint on every persist.
• modelGenerateStep does not silently 3×-retry a completed LLM round.
• No /api/agent fallback. No wrapping runAgentStream in one step. No Wasm/protocol change. D18 unmount still detach. C15 409 still live-only.
• Cloud ops: N/A for the source issue (no Production mutate here). Diagnosis is Observability on an existing Production run.
• ⛔ No application code here — source bug. Split the later plan if it is >1 turn.

---

Refs

• Operator report: since D17 cutover, 10–20 min coding turns take ~1 hour
• Umbrella / architecture lock: #794 (deltas, no mega-step, replay budget)
• D17 cutover: #811 / PR #841
• Loop / cadence: lib/workflows/turnLoop.ts (persistOnce after first tools model + after every batch; MAX_WORKFLOW_STEPS = 512)
• Assemble: lib/workflows/assembleDurableToolWorld.ts, lib/workflows/modelGenerateStep.ts, lib/workflows/toolExecuteStep.ts
• Persist args: lib/workflows/persistStep.ts (deltas + fold.checkpoint as step args)
• SSE hop: lib/workflows/turnSseStep.ts (writeTurnSse / closeTurnSse still 'use step')
• Retries: toolExecuteStep.maxRetries = 0; modelGenerateStep unset (SDK default 3)
• Start surface: app/api/turns/route.ts (probe client closed after start(); in-step assemble opens its own)
• Platform: vercel/workflow#1930 replay between steps

[btipling]

Grounding run for this issue: wrun_01M1A1G3346EHMCSSKQHK3W8SR (turnWorkflow, production, dpl_6Li3GcQH3me5BwwrmSwoHLkmNWSF).

Pulled with npx workflow inspect / GET /v1/workflow/v2/runs/…/steps (no decrypt). Times are step startedAt→completedAt. PT = UTC−7.

Headline

This turn’s wall clock is model time, not Workflows hop tax.

	UTC	PT
started	2026-08-30 19:14:17.523Z	12:14:17 PM
completed	2026-08-30 20:08:55.895Z	1:08:55 PM
wall	54m 38.4s

Bucket	Time	% of wall
modelGenerateStep (busy)	49m 33.8s	90.7%
inter-step gaps (replay / queue / orchestrator)	3m 18.8s	6.1%
toolExecuteStep (busy)	1m 20.4s	2.5%
persistStep (busy)	21.7s	0.7%
created→started queue (all steps)	28.6s	0.9% (overlaps gaps; not extra)
writeTurnSse + closeTurnSse	0.2s	~0%
lead + tail outside first/last step	~3.5s	~0.1%
sum of step durations	51m 16.1s	93.8%

Blob event log 27.4 MiB (blobStorageBytes=28,729,729). Stream storage 4.65 MiB.

Step count (issue check 1)

138 steps, all completed, attempt=1 on every step (no SDK 3× retry of a finished LLM round on this run).

Step	n
modelGenerateStep	45
toolExecuteStep	44
persistStep	46
writeTurnSse	1
closeTurnSse	2
total	138

138 / 45 = 3.07 steps per model round. Cadence is model → persist → tool → persist → model (plus a terminal persist + SSE). Matches the issue: ~3× hops, persist after first tools model and after every batch.

Time between steps (issue check 2)

Falsified as the hour on this run. Gaps are small and flat:

n	137 gaps
sum	3m 18.8s
avg	1.45s
p50	1.36s
p95	2.30s
max	3.40s (persistStep → modelGenerateStep)
min	489ms

Created→started per step is ~170–250ms typical (sum 28.6s). Replay/queue is a few minutes of overhead, not 3–6× the turn.

Per-type duration

Step	n	sum	avg	p50	p95	max
modelGenerateStep	45	49m 33.8s	66.1s	11.41s	5.59 min	12.79 min
toolExecuteStep	44	1m 20.4s	1.83s	1.22s	1.95s	24.01s (one outlier)
persistStep	46	21.7s	472ms	455ms	676ms	803ms
closeTurnSse	2	143ms	72ms	72ms	85ms	87ms
writeTurnSse	1	60ms	60ms	60ms	60ms	60ms

Persist is cheap (~0.5s × 46). Tool batches are ~1.2s except one 24s tool step at 12:37:43 PM PT. SSE hops are noise.

Slowest 10 steps (all model except one tool)

Rank	Duration	Step	started PT
1	12.79 min	modelGenerateStep	12:40:22
2	7.01 min	modelGenerateStep	12:53:28
3	6.13 min	modelGenerateStep	12:31:33
4	3.42 min	modelGenerateStep	1:00:38
5	3.11 min	modelGenerateStep	1:05:36
6	2.92 min	modelGenerateStep	12:24:51
7	2.15 min	modelGenerateStep	12:15:02
8	1.72 min	modelGenerateStep	12:38:22
9	1.48 min	modelGenerateStep	12:29:15
10	1.45 min	modelGenerateStep	12:22:50

Those five longest model steps alone are 32.5 min (59% of wall). Mean ≫ p50 because of this long tail, not because every round is slow.

Arg size (issue check 4)

Did not decrypt (audit-logged). From refs:

• Every modelGenerateStep input is an s3rf: remote ref (too big for the inline event row). 45/45.
• Every persistStep input is also s3rf:. 46/46.
• toolExecuteStep inputs stay inline dbrf: except 1/44 spilled to S3. Encrypted inline _data length grows ~600 → ~2900 chars over the turn (not plaintext size).
• 21 step outputs spilled to S3.

So the growing-transcript-in-step-args fire is real as storage (27.4 MiB blob for one turn) but not what ate the hour: persist stayed ~400–800ms even late, and gaps did not grow with index (max gap 3.4s).

Retries (issue check 5)

No. attempt is 1 on all 138 steps. Whatever made those 5–13 minute model steps, it was a single attempt each.

Assemble / sandbox reconnect (issue check 3)

Not visible as a separate step. assembleDurableToolWorld is inside modelGenerateStep and toolExecuteStep.

If tool-step p50 (1.22s) is a rough upper bound on “cold assemble + one tool batch”, then 44× ~1.2s ≈ 1m 20s total — including actual tools. Even if assemble also sits inside each model step at that same ~1s, that’s another ~45s, still not the hour.

To split LLM vs assemble vs held-writer SSE inside modelGenerateStep, need invincible.turn.model / function logs for this run. Step timestamps cannot.

Mapping to the six seams

Seam	This run
(1) ~3× steps	Yes. 3.07×. Persist after model and after every batch.
(2) replay/queue gaps ≫ step CPU	No. Gaps 6% of wall, p50 1.36s.
(3) reconnect every model and tool step	Possible inside steps; not the hour. Tool steps are ~1.2s.
(4) growing args / O(n²) event log	Yes as bytes (27.4 MiB, all model+persist args on S3). No as wall clock (persist stays sub-second).
(5) modelGenerateStep default 3 retries	No on this run (attempt=1).
(6) no whole-turn cap	Turn ran 55 min and completed. Unrelated to why it was 55.

What this run actually says

Operator-visible ~1 hour is real (54m 38s). The 3× Function hops are real. They are not where the minutes went.

The minutes are 45 modelGenerateStep calls totaling 49.5 min, with a heavy tail (five rounds ≥ 3 min). Next measurement: runtime logs on those five step ids to see whether that is DeepSeek/streamText or work in the step wrapper (assemble, BYOK, held SSE writer). I did not pull those logs here.

[btipling]

Follow-up on the same run — decrypted a sample of step I/O (sizes and usage only, not message content).

Model on every decrypted modelGenerateStep sample is zai/glm-5.3-flash, not DeepSeek. Decrypt did not cover all 45 rounds, so I am not asserting the unsampled ones.

Args really do grow (and still are not the hour)

	early	late
model input JSON	8.1 KB (1 msg, 7.8k prompt tok)	450 KB (142 msgs, 118k prompt tok)
persist input JSON	9.1 KB (1 delta)	925 KB (142 deltas, 423 KB checkpoint)
persist duration	398ms	559ms

Tool-step args stay 0.5–2.1 KB. The conversation is riding model + persist args, not tool hops.

Long rounds track completion / reasoning, not input KB

step	input	prompt tok	completion tok	duration
…07Z	446 KB	117k	53	2.7s
…07K	410 KB	109k	12,040 (+ 50,380 chars delta.reasoning)	12m 48s
…07S	433 KB	114k	6,698	7m 01s
…077	376 KB	100k	5,695	6m 08s

Cached tokens, when the field is present, are ~97–99% of prompt. The 12m 48s window (12:40–12:53 PT) is the model step, not idle between steps.

So: hop count 3.07× is real; hop time is ~3 minutes. The ~55 min wall is 45 GLM-5.3-flash rounds totaling 49m 34s, including one 12k-completion reasoning call. Intra-step split (gateway vs reasoning vs assemble) still needs function logs.