codex adoption caches transient CLI failures as completed (exit!=0) and replays them forever — no retry semantics for failed runs

[jianwei-su]

Summary

The codex adoption layer caches a transient CLI failure as a terminal, adoptable result. Once a run is recorded as status=completed with a non-zero exit_code, every subsequent spawn for the same prompt hash adopts (replays) the stored failure instead of re-running codex — there is no retry/invalidation semantics for failed runs. A short-lived environment fault therefore becomes a permanent judgment failure until the operator manually rotates the runtime dir.

What happened (host evidence)

• 2026-07-27 ~02:19Z: OpenAI began rejecting our pinned model with 400 invalid_request_error: "The 'gpt-5.6-sol' model requires a newer version of Codex" (host CLI was codex-cli 0.139.0). Every github-devloop-intake* judgment failed; each failure was recorded under logs/codex-adoption/prompt-<hash>/ as:
  • status.json: {"status":"completed", "exit_code":1, ...}
  • result.json: {"exit_code":1, "error_kind":null, "error":null}
• Host CLI upgraded to codex-cli 0.145.0; a manual codex exec with the same model succeeds.
• 48h after the fix, zero recovery: child logs show the departments re-raising codex-failed with the original 07-27 stderr (including the old OpenAI Codex v0.139.0 banner) and no fresh codex invocation — the failure snapshot is being replayed from the adoption record. 33 intake candidates were stuck this way.
• Only remediation found: restart the supervise unit so a fresh runtime dir (new adoption root) is minted; the very first re-runs then complete with exit_code=0.

Where

crates/fkst-framework/src/sdk_codex.rs — adoption/recovery paths treat CODEX_ADOPTION_STATUS_COMPLETED as adoptable without inspecting exit_code (e.g. the record.status == CODEX_ADOPTION_STATUS_COMPLETED branches around recover_completed_adoption_effect_by_key / the adoption-hit paths near lines ~950–970 and ~1540–1660 at engine 1e03c4f). judgment_codex_opts callers (e.g. github-devloop-intake-default/departments/intake_judge/main.lua:234 at platform cee722da) then surface the replayed stderr as intake-codex-failed, terminal=false, forever.

Why it matters

• Convergence-by-adoption is the right default for successful runs; extending it to failures conflates "this judgment converged" with "this spawn once failed".
• Transient faults (CLI version gates, network, auth hiccups) are common; today each one permanently poisons every prompt hash it touches.
• The failure is invisible at the digest layer (write intents keep flowing), so it silently disables the judgment stage.

Suggested direction (any of)

1. Do not adopt completed records with exit_code != 0 — treat them as absent (re-run), or
2. Give failed records a TTL / bounded retry budget, or
3. Provide an explicit invalidation hook (per-key or per-run) so hosts can clear failure snapshots without rotating the whole runtime dir.

Happy to provide full sanitized logs on request.