Skip to content

bug(plan-phase): the stall watch never observes real completion: it matches markers in the agent's own prompt and definition, and a revision can only end as stalled #5182

Open

Description

@davesienkowski

Pre-submission checklist

  • I have searched existing issues and this bug has not already been reported

GSD Version

1.15.0 (next @ 62549bf)

Runtime

Multiple (specify in description)

Operating System

WSL

Node.js Version

v24.13.0

Shell

/bin/bash

Installation Method

Updated from a previous version

What happened?

User impact. With the default planner.stall_detection_enabled: true, /gsd:plan-phase's wait for its background planner and plan-checker does not observe when they actually finish:

  • Claude Code (reproduced): the wait reports the planner and the checker as finished at its first check, whether or not they are. A planner still working at the 5-minute mark can be treated as done, and a checker that hangs is never reported as stalled. Separately, every plan revision round, on every runtime, can only end in the "Accept as revised / Retry / Stop" stall prompt, about 10 minutes after dispatch, instead of continuing when the revision is done.
  • Other runtimes (code-level, not reproduced live): the wait only sees completion through the {outputFile} of a Claude-Code-style Agent() result. On a host whose background dispatch returns no such file, the marker check can never succeed, so the planner, checker and revision spawns can end only as stalled (the helper's own docs already note the checker has no other working completion signal).

Where: gsd_stall_watch in gsd-core/workflows/plan-phase/steps/stall-detection-helpers.md, used for the planner, plan-checker and revision spawns (plan-phase.md steps 8, 10, 12) on all runtimes.

What did you expect?

The wait returns when the spawned agent has actually returned (with its real marker), returns stalled only when the agent stopped making progress, and behaves the same on every runtime that dispatches background agents.

The helper's design text sets this expectation: for the checker the marker path "is the ONLY working completion signal", and declaring a succeeded agent stalled is "worse than the original unbounded wait" (stall-detection-helpers.md, "Plan-checker's artifact glob needs the marker").

Steps to reproduce

Root cause (next @ 62549bf):

  1. stall-detection-helpers.md:156-159 decides completion with grep -qF "$m" "$output_file" over the entire output file, and only when [ -f "$output_file" ].
  2. On Claude Code that file is the subagent's JSONL transcript (~/.claude/projects/{project}/{sessionId}/subagents/agent-{agentId}.jsonl, per the Claude Code sub-agents docs). The transcript holds the agent's inputs as well as its output:
    • record 1 is the prompt. The checker prompt lists its own markers (plan-phase.md:1074-1075, - ## VERIFICATION PASSED — all checks pass);
    • a prompt_snapshot attachment record (records 9-12 in every transcript below) carries the agent definition, and both definitions quote their markers: agents/gsd-planner.md:929-934 ("## Return Markers ... Your orchestrator dispatches on exact marker strings in your final output. Emit exactly one of: ... ## PLANNING COMPLETE") and agents/gsd-plan-checker.md:957-960.
  3. plan-phase.md:1189: the revision watch passes no markers and repeats while waiting/active; gsd_stall_should_recover returns marker_received only when a marker matched, so stalled is its only exit. The helper header (lines 12-13) states the watch runs "regardless of whether the backgrounded agent's own completion notification ever arrives".
  4. The {outputFile} the helper depends on is not part of Claude Code's documented interface (the sub-agents docs describe only that "a background subagent's results reach Claude as a completion notification in a later turn"), and other hosts dispatch background agents through their own primitives (for Codex, GSD's conversion rules map Agent() to spawn_agent, with completion observed through collaboration.wait_agent, src/runtime-artifact-conversion.cts).

Reproduction 1, deterministic (runs the shipped functions with sleep stubbed; run from a gsd-core checkout):

#!/usr/bin/env bash
set -euo pipefail
REPO="${1:-.}"; REF="${2:-origin/next}"
T="$(mktemp -d)"; trap 'rm -rf "$T"' EXIT
git -C "$REPO" show "$REF:gsd-core/workflows/plan-phase/steps/stall-detection-helpers.md" \
  | awk '/^gsd_stall_should_recover\(\) \{/{f=1} f{print} f&&/^}$/{n++; if(n==2) exit}' > "$T/helpers.sh"
PLANNER_STALL_INTERVAL_MINUTES=0; PLANNER_STALL_THRESHOLD_MINUTES=10
source "$T/helpers.sh"

# 1: checker transcript holding only the prompt record (nothing returned yet)
mkdir -p "$T/phase"
printf '%s\n' '{"type":"user","message":{"role":"user","content":"... Return:\n- ## VERIFICATION PASSED — all checks pass\n- ## ISSUES FOUND — structured issue list\n"}}' > "$T/transcript.jsonl"
echo "1  -> $(gsd_stall_watch "$(date +%s)" "$T/transcript.jsonl" "$T/phase/*-PLAN.md" "## VERIFICATION PASSED" "## ISSUES FOUND")"

# 2: revision spawn (no markers); the revision finished and rewrote a PLAN.md
PLANNER_STALL_INTERVAL_MINUTES=5
echo '# plan' > "$T/phase/01-PLAN.md"
echo "2a -> $(sleep() { :; }; gsd_stall_watch "$(( $(date +%s) - 120 ))" "$T/transcript.jsonl" "$T/phase/*-PLAN.md")"
touch -d '6 minutes ago' "$T/phase/01-PLAN.md"
echo "2b -> $(sleep() { :; }; gsd_stall_watch "$(( $(date +%s) - 360 ))" "$T/transcript.jsonl" "$T/phase/*-PLAN.md")"
echo "2c -> $(sleep() { :; }; gsd_stall_watch "$(( $(date +%s) - 660 ))" "$T/transcript.jsonl" "$T/phase/*-PLAN.md")"

# 3: a host that returned no output file; the checker finished at 2 min (nothing on disk changes for a PASS)
echo "3  -> $(sleep() { :; }; gsd_stall_watch "$(( $(date +%s) - 660 ))" "$T/no-such-file" "$T/phase/none-*-PLAN.md" "## VERIFICATION PASSED" "## ISSUES FOUND")"

Observed:

1  -> marker_received
2a -> active
2b -> waiting
2c -> stalled
3  -> stalled

Case 1 should be waiting. In case 2 the revision finished at 2a and the loop can only leave through 2c. Case 3 is a finished checker on a host without an output file.

Reproduction 2, live (Claude Code 2.1.288): a run_in_background=true Agent() whose prompt listed the two checker markers and whose task took about 3 minutes. Its outputFile resolved to .../subagents/agent-<id>.jsonl; line 1 was the prompt and the only line containing ## VERIFICATION PASSED. The shipped gsd_stall_watch (interval 0) returned marker_received 8 seconds after dispatch; the agent later returned ## ISSUES FOUND.

Reproduction 3, real transcripts (one machine, Claude Code 2.1.267-2.1.287, 2026-09-09 to 2026-10-02): in every stored background gsd-planner run (71) and gsd-plan-checker run (67), the agent's own marker (## PLANNING COMPLETE / ## VERIFICATION PASSED) was already in the transcript before the agent's final reply, at the latest in the prompt_snapshot record (records 9-12). 67 of the 71 planner runs lasted longer than 5 minutes (median 22.6 min) and 9 of the 67 checker runs did (median 1.1 min), so with the default 5-minute interval the first check would match while those agents were still working. (Durations and marker positions, not observed misroutes: that machine ran a locally patched watch.)

Error output / logs

None; the wrong result is a normal return value of gsd_stall_watch.

How often does this happen?

  • Every time (100% reproducible)

Impact

Moderate — Feature is broken but I have a workaround

Workaround (if any)

planner.stall_detection_enabled: false (#4570): plan-phase waits through the runtime's own Agent() completion and never calls gsd_stall_watch, at the cost of #2650's automatic stall recovery.

Additional context

Context: this is the first of a few issues I am filing from my GSD Core speed pack, a set of local speed and friction changes I run on top of next (https://github.com/davesienkowski/gsd-core-speed-pack). It carries a local workaround for this bug; the fix proposed here is a different, runtime-neutral design. I will file the others one at a time, after this one is triaged.

Recommended fix: a GSD-owned return receipt, the same on every runtime.

  1. The orchestrator gives each watched spawn (planner, checker, revision) a receipt path under the phase dir, e.g. ${PHASE_DIR}/.gsd-returns/<spawn>-<TS>.md, in its prompt.
  2. The agent's last action is to write its return marker line to that path. gsd-planner has Write; gsd-plan-checker has Bash but no Write, so the checker's contract gains one small write (a maintainer call on its read-only role).
  3. gsd_stall_watch takes the receipt path instead of {outputFile} and matches the markers only in that file. Prompts, agent definitions and transcripts can never match it, and no host output format is read.
  4. The revision spawn gets a receipt too (## REVISION COMPLETE / ## REVISION_CONFLICT), giving it a success exit.
  5. The runtime's own completion (Claude Code's completion notification, Codex wait_agent terminal status) stays the primary way the orchestrator reads the agent's full return; the receipt is what the bounded watch observes. A missing receipt degrades to today's bounded stalled path, never a hang.

Why this over the alternatives:

  • It is runtime-neutral: no runtime === … branch and no reliance on an undocumented host file, matching ADR-1239 (per-host behavior "explicit and testable rather than scattered runtime === '…' checks"; per-host values "documentation-sourced ... never inferred").
  • There is in-repo precedent for moving a return contract from a marker string to an artifact: fix(#3440): retire GAP CLOSURE PLANS CREATED marker, document artifact return contract (fix(#3440): retire GAP CLOSURE PLANS CREATED marker, document artifact return contract #3443); agents/gsd-planner.md:912 "gap-closure returns are artifact-based (fix(#1891 F9): retire the GAP CLOSURE PLANS CREATED marker — artifact route is canonical #3440)".
  • Considered and not recommended: (a) parsing the Claude Code transcript for the final assistant record (stop_reason: end_turn). It fixes Claude Code only and depends on an undocumented record schema. (b) Removing the literal markers from prompts. The agent definitions are snapshotted into the transcript, so this cannot work.

Acceptance criteria:

  • Markers present in a prompt, agent definition or transcript, but not in the receipt, yield waiting.
  • A checker whose receipt says ## ISSUES FOUND routes to step 12, never step 13.
  • A finished revision yields a success result within one poll interval, not stalled.
  • A spawn that never writes its receipt still yields stalled after the threshold.
  • The same results on Claude Code and on a host with no {outputFile}.
  • Regression tests that fail on next first, in tests/plan-phase-stall-detection.test.cjs.

Behavior change to disclose: the checker and planner waits no longer return at their first check, and the "Accept as revised" menu no longer follows every successful revision.

Related: origin #2650 and PR #3015 (introduced gsd_stall_watch); #4570 / PR #4585 (the disable switch); #4079 / PR #4299 (no wake-up calls between cycles); #3923 (approved: make the threshold configurable; a longer threshold only delays the revision menu); #2699 (general subagent watchdog, wontfix; this fix stays inside the existing helper and contracts).

Activity

  1. added
    bugSomething isn't working
    area: workflowPhase execution, ROADMAP, STATE.md
    needs-triageNew issue awaiting maintainer review
    on Oct 2, 2026
  2. trek-e commented on Oct 2, 2026

    @trek-e
    Collaborator

    This was generated by AI during triage.

    Diagnosis

    Reproduced: Yes. Running the shipped gsd_stall_watch against a transcript containing only the prompt text ## VERIFICATION PASSED returns marker_received (should be waiting); with no output file and 11 minutes elapsed it returns stalled. The marker-less revision watch has no success exit.
    Root cause: gsd-core/workflows/plan-phase/steps/stall-detection-helpers.md:156-159 (gsd_stall_watch) greps the entire output file for markers, which includes the agent's prompt and definition snapshot; :124-126 (gsd_stall_should_recover) returns success only on a marker match, so the revision watch at plan-phase.md:1189 can only end stalled. Call sites: plan-phase.md:958, 1092, 1189.
    Introduced by: 067a4d1, fix(#2650) (#3015).
    Blast radius: plan-phase steps 8, 10, 12 (planner, checker, revision) on all runtimes with the default planner.stall_detection_enabled: true; checker PASS/ISSUES routing and every revision round.
    Coupled (must not miss): tests/plan-phase-stall-detection.test.cjs; return-marker contracts in agents/gsd-planner.md and agents/gsd-plan-checker.md (checker has no Write tool); the spawn prompts in plan-phase.md; docs for planner.stall_detection_enabled; related open #3923 (threshold).
    Regression test: markers appearing only in prompt/definition content of the watched output yield waiting; a checker return of ## ISSUES FOUND routes to step 12; a finished revision yields a success result within one poll interval; a spawn that never signals completion yields stalled after the threshold; identical results when the host provides no output file.

    Agent Brief

    Category: bug

    Summary: The plan-phase stall watch decides completion from text that includes the agent's own prompt and definition, so it reports completion too early, and revisions can never report success.

    Current behavior: The watch returns "marker received" at its first check whenever the watched output contains the marker strings from the prompt or agent definition. A revision watch with no markers can only end as stalled. On hosts with no output file, every watched spawn ends as stalled.

    Desired behavior: The watch returns success only when the spawned agent has actually returned its marker, returns stalled only when the agent stops making progress, and behaves the same on every runtime that dispatches background agents.

    Key interfaces: the stall watch helper and its decision function; the planner, checker and revision spawn contracts and their return markers; the orchestrator routing after the checker result.

    Acceptance criteria:

    • Markers present only in a prompt, agent definition or transcript yield waiting.
    • A checker returning issues routes to the revision step, never to the pass step.
    • A finished revision yields a success result within one poll interval.
    • A spawn that never signals completion yields stalled after the threshold.
    • Results are identical on Claude Code and on a host with no output file.
    • Regression tests fail on current next first.

    Out of scope: a general subagent watchdog (#2699); changing the default threshold (#3923).

  3. added and removed
    needs-triageNew issue awaiting maintainer review
    on Oct 2, 2026
  4. added a commit that references this issue on Oct 3, 2026
  5. davesienkowski commented on Oct 3, 2026

    @davesienkowski
    CollaboratorAuthor

    Fix PR: #5192. It follows the Agent Brief above, and each of the six acceptance criteria maps to a regression row in tests/plan-phase-stall-detection.test.cjs. The new rows fail on next first; the PR body has the counts.

    Two things worth your eye:

    • Scope is five sites, not three. The chunked outline and per-plan watches in steps/chunked-planning-mode.md had the same first-check match (their prompts say Return: ## ... COMPLETE). The existing test also pins the helper contract at exactly five sites.
    • Maintainer call on the checker. gsd-plan-checker has no Write tool (enh(agents): add disallowedTools to read-only verifier/auditor agents (Claude Code) #767 Group A), and Codex derives a read-only sandbox for it. The PR keeps that: the checker writes its receipt with one Bash printf, and a refused write is non-fatal. On a host that runs the checker read-only, its watch ends stalled at the threshold, as it does today, unless the runtime's completion result arrives first. The PR says plainly that AC5 is therefore met for the checker only where it can write one file. The alternative that closes the gap is Group B for the checker (Write allowed, which flips its Codex sandbox to workspace-write). That is your call; I can add it to the PR or as a follow-up.

    Generated by AI during contribution; reviewed and posted by a human maintainer (@davesienkowski).

  6. added 6 commits that reference this issue on Oct 3, 2026
  7. 154 remaining items

  8. added 15 commits that reference this issue on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area: workflowPhase execution, ROADMAP, STATE.mdbugSomething isn't workingconfirmed-bugVerified reproducible bug

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions