Skip to content

dead-letter error_excerpt is head-truncated at 500B, so it captures code_provenance boilerplate and never the failure (0/461 records diagnosable) #296

Description

@loning

The defect

A dead-letter record's error_excerpt is truncated to the head of the child's stderr. The head of a
department child's stderr is always framework boilerplate (EVENT=code_provenance, then
EVENT=external_command lines), so the excerpt reliably captures everything except the failure.
The actual error — the Lua error() message and stack traceback — is emitted at the end of stderr
and is always past the cutoff.

Measured, on a live durable store

fkst-framework observe --durable-root … --json, 461 dead-letter records:

fact value
excerpt length min 500, p50 500, max 500 bytes — a hard head cut
records whose excerpt contains the actual cause 0 / 461
permanent 461 / 461
replayable 0 / 461

Every record begins with the same boilerplate:

exit=1 stderr=TIMESTAMP=… LEVEL=info EVENT=code_provenance ENGINE_VER=… PKG_VERS=…
LEVEL=info EVENT=external_command CMD=/…

…and stops before ever reaching the failure.

The real cause, for the same event, from <RT>/logs/framework-child/<dept>-<ts>.log:

[framework] pipeline failed: pipeline(event) call: runtime error:
  …/libraries/devloop/liveness_scan.lua:166: liveness-scan-pr-list-failed:
  gh: API rate limit exceeded … (HTTP 403)
stack traceback: …

Why this matters more than it looks

The dead-letter record is designed to be the durable, self-contained account of a failure — the L2
triage layer consumes exactly these facts to draft issues. As it stands, the record is
diagnostically empty: it proves that something failed and hides what.

The only surviving copy of the truth is the per-child log under the runtime scratch root, and that root
is routinely pruned on restart (the operator tooling does this deliberately — the runtime root is
scratch by contract, the durable root is the persistent one). So after any restart, the cause of a
dead-letter storm is unrecoverable.
In practice this cost real diagnosis time: a 461-record storm
could only be root-caused because a child log happened to survive; the dead-letter records themselves
were unusable.

Suggested direction

Prefer the tail over the head, or capture both ends. The failure is at the end by construction —
Lua's error() + traceback is the last thing written before the non-zero exit — while the head is
predictable boilerplate that carries no diagnostic value. Options, roughly in order of preference:

  1. Truncate from the tail (keep the last N bytes) — smallest change, recovers the error and traceback.
  2. Keep a head slice and a tail slice with an elision marker, if the leading code_provenance is
    wanted for provenance.
  3. Extract the structured failure (the pipeline failed: line + traceback) into its own field rather
    than relying on a byte window at all.

Filed from the fkst-packages dogfood; the package-side consequence of this blindness is tracked at
ChronoAIProject/fkst-packages#2930.

⟦AI:FKST⟧

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions