The defect
A dead-letter record's error_excerpt is truncated to the head of the child's stderr. The head of a
department child's stderr is always framework boilerplate (EVENT=code_provenance, then
EVENT=external_command lines), so the excerpt reliably captures everything except the failure.
The actual error — the Lua error() message and stack traceback — is emitted at the end of stderr
and is always past the cutoff.
Measured, on a live durable store
fkst-framework observe --durable-root … --json, 461 dead-letter records:
| fact |
value |
| excerpt length |
min 500, p50 500, max 500 bytes — a hard head cut |
| records whose excerpt contains the actual cause |
0 / 461 |
permanent |
461 / 461 |
replayable |
0 / 461 |
Every record begins with the same boilerplate:
exit=1 stderr=TIMESTAMP=… LEVEL=info EVENT=code_provenance ENGINE_VER=… PKG_VERS=…
LEVEL=info EVENT=external_command CMD=/…
…and stops before ever reaching the failure.
The real cause, for the same event, from <RT>/logs/framework-child/<dept>-<ts>.log:
[framework] pipeline failed: pipeline(event) call: runtime error:
…/libraries/devloop/liveness_scan.lua:166: liveness-scan-pr-list-failed:
gh: API rate limit exceeded … (HTTP 403)
stack traceback: …
Why this matters more than it looks
The dead-letter record is designed to be the durable, self-contained account of a failure — the L2
triage layer consumes exactly these facts to draft issues. As it stands, the record is
diagnostically empty: it proves that something failed and hides what.
The only surviving copy of the truth is the per-child log under the runtime scratch root, and that root
is routinely pruned on restart (the operator tooling does this deliberately — the runtime root is
scratch by contract, the durable root is the persistent one). So after any restart, the cause of a
dead-letter storm is unrecoverable. In practice this cost real diagnosis time: a 461-record storm
could only be root-caused because a child log happened to survive; the dead-letter records themselves
were unusable.
Suggested direction
Prefer the tail over the head, or capture both ends. The failure is at the end by construction —
Lua's error() + traceback is the last thing written before the non-zero exit — while the head is
predictable boilerplate that carries no diagnostic value. Options, roughly in order of preference:
- Truncate from the tail (keep the last N bytes) — smallest change, recovers the error and traceback.
- Keep a head slice and a tail slice with an elision marker, if the leading
code_provenance is
wanted for provenance.
- Extract the structured failure (the
pipeline failed: line + traceback) into its own field rather
than relying on a byte window at all.
Filed from the fkst-packages dogfood; the package-side consequence of this blindness is tracked at
ChronoAIProject/fkst-packages#2930.
⟦AI:FKST⟧
The defect
A dead-letter record's
error_excerptis truncated to the head of the child's stderr. The head of adepartment child's stderr is always framework boilerplate (
EVENT=code_provenance, thenEVENT=external_commandlines), so the excerpt reliably captures everything except the failure.The actual error — the Lua
error()message and stack traceback — is emitted at the end of stderrand is always past the cutoff.
Measured, on a live durable store
fkst-framework observe --durable-root … --json, 461 dead-letter records:permanentreplayableEvery record begins with the same boilerplate:
…and stops before ever reaching the failure.
The real cause, for the same event, from
<RT>/logs/framework-child/<dept>-<ts>.log:Why this matters more than it looks
The dead-letter record is designed to be the durable, self-contained account of a failure — the L2
triage layer consumes exactly these facts to draft issues. As it stands, the record is
diagnostically empty: it proves that something failed and hides what.
The only surviving copy of the truth is the per-child log under the runtime scratch root, and that root
is routinely pruned on restart (the operator tooling does this deliberately — the runtime root is
scratch by contract, the durable root is the persistent one). So after any restart, the cause of a
dead-letter storm is unrecoverable. In practice this cost real diagnosis time: a 461-record storm
could only be root-caused because a child log happened to survive; the dead-letter records themselves
were unusable.
Suggested direction
Prefer the tail over the head, or capture both ends. The failure is at the end by construction —
Lua's
error()+ traceback is the last thing written before the non-zero exit — while the head ispredictable boilerplate that carries no diagnostic value. Options, roughly in order of preference:
code_provenanceiswanted for provenance.
pipeline failed:line + traceback) into its own field ratherthan relying on a byte window at all.
Filed from the fkst-packages dogfood; the package-side consequence of this blindness is tracked at
ChronoAIProject/fkst-packages#2930.
⟦AI:FKST⟧