Skip to content

cockpit v3: blocker paths and status lines, roll-ups and the base env, alarms closed with a cause (S5, S6, S7) - #210

Merged
LittleChimera merged 2 commits into
stagingfrom
factory/v3-status-rollups
Nov 3, 2026
Merged

LittleChimera merged 2 commits into
stagingfrom
factory/v3-status-rollups

Conversation

@LittleChimera

@LittleChimera LittleChimera commented Nov 3, 2026 •

Copy link
Copy Markdown

Related issue

Closes # (cockpit v3 plan items S5, S6, S7; notes/factory-v3/final/plan.md)

Summary

Every node of the v3 node API (GET …/n/{ref}) now says in one line why it is or isn't moving, ending at the person who must act. Roll-ups carry money, time and what is in the base env. Alarms the server closed show with their cause. v3 still ships behind ?cockpit=v3; v1 and v2 are untouched.

ELI5: each card used to show only its state ("Needs you"). Now it follows the chain of waits down and across ("T-16 → held for C-16 → needs you: Apply C-16 (D-45)"), so you can see who has to act.

product ›(step down, the child that holds the most) Agent assistant › assistant-core › T-16
          →(hold) held for C-16 →(person) needs you: Apply C-16 (D-45)   · also holds assistant-panel-ux
assistant-panel-ux →(dependency hop) waits on assistant-core › T-16 → held for C-16 → needs you: …

Plan bullet → what this PR does

S5. Blocker path, status lines, why (node_status.py, new)

  • blocker_path per api.md §3.3, reusing waits.py. A node's own waits come first: the snapshot's waiting_on for a project (waits.project_waits, already cut to the viewer), waits.task_waits for a task, and a ready PR's merge for a change. They are ranked stall > decision > proposal > hold > merge > dependency > capacity.
    • An ask a person answers ends the path.
    • A dependency, another project's merge, or a hold on another node hops there (→).
  • Otherwise the path steps down (›) into the child that blocks most. Person-ending paths rank first, by held weight (Σ estimate of the tasks the ask holds, plus the weight of projects that depend on where it sits), then by dependents, then by earliest column.
  • The node's own open decision ends the path when no child's path ends at a person. A decision on a proposed (ghost) child counts as the parent's own, which is how the feature ends at D-49 once D-45 is answered. A path that ends at a person always wins over one that ends at the server, at moving work or elsewhere, so a node's own wait never hides its own ask.
  • A project that has not started and that nothing holds ends at who starts it, from the snapshot's start: "the server starts it within a minute" (ends: server), "waits for Atlas to start it", "starting: pulling the image", or the capacity wait "(all 12 machines in use, 3 ahead)". Without a start, the charter's spawn owner decides.
  • Limits: a line shows at most 6 segments. A longer path keeps its first 4 steps, an elision segment (…) and its end, so ends: person and the decision stay on the line. A path that comes back to a node still being computed ends cycle (cycle results are not memoized, so the answer does not depend on which node was asked first).
  • Ready PRs that a person merges into one base become one ask, merge:<base> (grouped, external_url, "Review and merge").
  • status (full line):
    • Labels: "Not started for 7 h:", "Held on purpose:", "Top blocker:" (other work moves), "Not moving:".
    • decision is a full DecisionRef: verb, actions from the decision's options, asker, age, expires from the 72 h proposal expiry, subject, title, text.
    • also_holds, numbered why and why_title.
    • note names met links that had no reason.
  • Waits that end elsewhere keep who must act: "held for C-16, waiting on the other lead", "waits for a free machine (all 12 machines in use, 3 ahead)", "waits on the server to merge PR docs(changelog): record v0.18.0 omnigent-ai/omnigent#9856".
  • status_compact shows only the end of the path ("Waits on X · held for C-16", "Needs you: … · via a › b", "Held for C-16: …"). A moving node shows moving {ref, activity}. With no path, the line falls back to the state line, which now has done notes ("fix(sessions): refuse a fork that needs a copy of a missing agent bundle omnigent-ai/omnigent#9881 merged · in base env") and "queued 7 h".

Dependencies are met only when the work is finished (gating.py, used by the waits, the gate and every view of it)

  • A merged-mode dependency was met as soon as the target's PRs were merged with none open. A project that merged its first task's PR and still had open tasks (assistant-core after [Bug] A message cancelled while queued blocks every later message in its session omnigent-ai/omnigent#9876, T-16 still open) released its dependents, so the panel's "waits on assistant-core" came only from a hand-written fixture and the real snapshot had no wait for it.
  • Now a project with an open task (planned, ready, running, review, blocked) does not meet a merged dependency until those close or it is done. ready mode is unchanged. "Start now" (S8) is how a person relaxes a dependency to the change it needs (PR-9876), per api.md §4.1.
  • Every caller passes the open tasks: the release pass, Start, the create guard, agent state, the snapshot's overlay, the graph's project edges, the agent projects listing, a task's project:<slug> dependency and waits.py.

S6. Roll-ups, results, base env fact

  • Cost: a project's cost is still the snapshot's session spend. Burn is the last hour of turn spans (cost at open and close, prorated; read newest first). A turn still open counts up to its session's cost now (the snapshot's member). A project shows its own burn; a feature or need shows the sum over its projects. The root shows the snapshot's factory rate, which also pays for leads and agents that work for no project, so it is more than the features' sum; for a viewer from whom part of the factory is hidden, it is the sum of what they may read.
  • Budget is the node's own cap, else the sum of its children's caps only when every child has one (otherwise no budget: the cost would be compared with only some of the caps).
  • ETA appears only when every unfinished task's kind has ≥ 5 observed durations (tasks.observed_weights → graph.critical_path). A feature's ETA is the slowest project per column, summed. Otherwise there is no ETA field at all.
  • Feature progress stays acceptance on the base env first, build % second (from cockpit v3: project → feature pointer and the read-only node projection (S2, S4) #208, now tested here). try_it lists passing acceptance steps on the feature and on the project that owns each step, marked "API only: not built yet" while a plan.user_facing project is unfinished. try_ready and rollup.in_base_env.try_it are set from it.
  • Deployed sha stored and served: base.py records the commit the base env runs on the root (omnigent.factory.base_env.<key>: sha, first-report time, and when the watcher saw it become the head).
    • Written from env.reported (integration project, ready).
    • Placed from the current head label or the base.moved history, and placed later when the move lands. base.moved now carries at.
    • Served as factory.base.deployed in the snapshot, and as node.env on the root ({name, branch, sha, deployed_at, state, smoke?, app_url, api_url, note, ref}). smoke shows only when the server's probe ran.
  • in_base_env per change by ancestry (base.contains): the deployed commit is the merge, or the watcher saw it become the head after the merge landed. A deployed commit the watcher never saw become the head (first sighting, two moves between polls, a report that raced the watcher) holds only itself until its next move or report places it; the env's first-report time is no longer used as a bound. A PR into another base, or into the same branch of another repo, is never in the base env.

S7. Alarms closed with a cause

  • server_decisions.close_by_server(…, real_ask, real_ask_verb, about) records {cause, real_ask?, real_ask_verb?, about?, was} on the closing event (closed_data). expire_decision records a cause only when given one (why); the generic "the situation cleared" is now only the summary's wording and no cause. expire_decision(…, superseded_by=D-n) records that a newer alarm replaced it.
  • loop_escalation.settle: a stall that ran again because its task is now held on purpose is answered by the server with "T-1 is held on purpose for C-n; a held task is not stalled. The real ask is D-m (Apply C-n)." The PL gets no RESUMED notice because the hold stands. A loop that runs again expires its alarm with "T-1 runs again: progress resumed" (settled_cause). Loops that ended (merged, closed) record the node they were about. A refuted claim's "Keep waiting?" is expired with superseded_by: it was escalated, not closed.
  • alarms.py (new) reads today's server-closed loop and done_check alarms back from the events: answers by the server, and expiries that carry a cause and no superseded_by. The read grows (500 → 2000 → 8000) until it covers the 24 h window, so other decisions' events cannot push today's alarms out.
    • fx:health lists them as D-n items with closed_alarms on its page.
    • The node each alarm was about gets an "alarm closed" chip.
    • The root roll-up has alarms_closed {count, ref: "fx:health"}.
    • An open no-progress stall alarm on a task held on purpose is no ask on the node pages, even before the tick closes it, so there is no D-48 on assistant-core. Only those are hidden (loops.resumes_when_held: an idle stall within its wall budget, every loop riding on the decision): a wall stall, an exhausted or a thrashing loop stays an ask, since the server will not close it. The snapshot's stall wait for such an alarm is dropped too.
    • A closed alarm's real ask reads with its verb: "The real ask is D-45 (Apply C-16)".

Web (types and contract only; rendering comes in W8/W9)

  • nodeTypes.ts gains optional NodeFull.env and closed_alarms (with real_ask_verb). factoryApi.ts gains FactoryBaseView.deployed.
  • nodeWire.server.test.tsx now reads the server's lines through statusText instead of expecting empty segments.
  • The fixture is regenerated from the server test and adds fx:health.

Deviations

  • Project-level own waits come from the snapshot's waiting_on (the server's own waits.project_waits, viewer-filtered). They are recomputed only for a project the snapshot has no view of. The test fixture now builds every view's waiting_on with waits.project_waits the way the snapshot does (node_fixtures.snapshot_waits), so the headline case is what production computes.
  • The dependency rule change above is server behaviour shared by every cockpit: a dependent now starts after the work it depends on is finished, not after that work's first merge.
  • A building task (running or in review) does not "wait" on its project's dependencies; its builder works regardless.
  • Project cost stays the snapshot's session spend (what budgets and the money line already show). Only burn comes from spans, because the cost ledger is durable and turn spans are pruned after 7 days.
  • Feature budget is the sum of its children's caps when every child has one (critique §4), not the prototype's illustrative $400 feature cap. In the 13:40 fixture two finished projects have no cap, so the feature shows no budget. Initiatives have no budget field yet.
  • Alarms the server expired with a cause (reconcile's closed-project path, a loop that ran again) also count as closed. Reconcile still expires them rather than answering them, so the existing test_reconcile_expires_* contract is unchanged.
  • ETA, the per-pair code-host compare for ancestry, and a separate "committed caps" field are not added: without a placed head only an exact sha counts, and placement retries on the next move or report.
  • No migration: the deployed sha is a root label next to the base head label. Closed alarms are read from the decision events.
  • A task with runs and no PR still has no run page (noted in S4; not part of S5–S7).
  • The notes' api.md NodeFull gained env and closed_alarms lines to match.

Test Plan

  • tests/server/factory/test_node_status.py (31 tests) runs the prototype's cases on the 13:40 fixture. Every view's waiting_on is now built by waits.project_waits as the snapshot does (node_fixtures.snapshot_waits), not written by hand:
    • assistant-core reads "Not moving: T-16 → held for C-16 → needs you: Apply C-16 (D-45)", with no D-48 anywhere on the page.
    • The panel reads "Not started for 7 h: waits on assistant-core › T-16 → held for C-16 → needs you: Apply C-16 (D-45)", and its note names the 3 met links. Its wait comes from the gate: dependency_met(assistant-core) is false while T-16 is open and would be true without it.
    • The root reads "Not moving: Agent assistant (sample) › assistant-core › T-16 → … · also holds assistant-panel-ux". Tiles read "Needs you: Apply C-16 · via T-16 · the ask above" and "Waits on assistant-core · held for C-16".
    • After D-45: the panel reads "Waiting: waits on assistant-core · moving (T-16: Luz building)", the feature ends at D-49, the root reads "Top blocker: build-tier-drafts-3 → needs you: Merge docs(changelog): record v0.18.0 omnigent-ai/omnigent#9856 and perf(web): overlap linked diff requests and editor initialization omnigent-ai/omnigent#9857". With a blocking ask on the panel too, the panel reads "Needs you: Answer D-60" (state needs, needs 1/1).
    • Projects nothing holds: "Not started for 7 h: waits for Atlas to start it" (ends: other), "… the server starts it within a minute" (ends: server), "… starting: pulling the image", and "… waits for a free machine (all 12 machines in use, 3 ahead)".
    • A chain of 7 queued projects ahead of assistant-core: 6 segments (4 hops, …, the person), ends: person, decision D-45, label "Not started for 7 h:"; the tile keeps D-45.
    • Two holds tied on dependents and column: the 240-min ask beats the 60-min one (the test fails with weight() forced to 0).
    • A ready PR on a factory branch into an auto-merge target: the change reads "Waiting: waits for the server to merge it", the project "Waiting: waits on the server to merge PR docs(changelog): record v0.18.0 omnigent-ai/omnigent#9856", both ends: server.
    • A hold on a proposal being co-signed reads "held for C-16, waiting on the other lead". A true dependency cycle ends cycle from both sides.
    • Property at scale (13/45/360): every node with a person-ending child path ends at a person.
    • S6: assistant-core $25.13 of $150 at $2.50/h; the feature shows $51.13 and no budget (two uncapped projects), $400 when each project has a cap. An open 40-min turn adds its session's spend so far ($3.84/h). ETA only with observed samples. Results "7 merged · in base env"; no smoke unless the probe ran. A deployed commit not yet placed holds only itself (a merge while it built is "not yet in base env"; the merge that is the deployed commit is in it); a PR into the same branch of another repo is never in it.
    • S7: D-46, D-47, D-48 under Health, D-48 with "The real ask is D-45 (Apply C-16)". An open idle-stall alarm on a held task is no ask; a wall stall on the same held task stays an ask (needs 2/2). Events read back: an expiry with a cause counts, one without a cause or superseded_by does not.
  • test_dependency_gating.py: a project with an open task does not meet a merged dependency; a done one does; ready mode unchanged.
  • test_loops.py: a refuted claim's "Keep waiting?" is expired with superseded_by and no cause, and Health does not list it.
  • test_loop_stall.py: a stall then held on purpose is answered with its cause, the real ask and its verb ("Apply C-n"), and Health reads it back.
  • test_base_watcher.py: an unplaced deployed head contains only itself; once placed on its move, the earlier merge is in it.
  • Commands:
    • uv run --frozen pytest -q -n 8 tests/server/factory tests/examples/test_factory_sample_tokens.py: 4321 passed, 2 failed. test_discovery_beats::test_the_budget_line_then_the_stop_and_the_queued_ask fails the same way on a clean origin/staging checkout (checked). test_nodes::test_every_page_of_a_13_45_360_factory_is_fast hit 0.76 s while pre-commit and the web tests ran alongside; alone it passes, and the first build takes 0.17 s with and without this change.
    • Touched suites after the last edit (node status, nodes, base watcher, loop stall, loops, gating, waits, sample tokens): 284 passed. Other dependency-touching suites (test_sessions_factory_labels, runner factory tools, bundles): 336 passed.
    • Web: vitest run src/components/factory/v3 (114 passed), tsc -b, oxlint, prettier --check.
    • pre-commit on the changed files: ruff and web hooks pass; pyrefly reports only the 28 errors already on staging (telemetry, gensee, reconcile), none in touched files.
    • No migration.
  • Manual check: OMNIGENT_FACTORY_NODE_API=1, open /factory/<id>/n/assistant-panel-ux?cockpit=v3 and read the header line, then "why? ▾". Open n/product to see the top blocker and "3 alarms closed as noise". Open n/fx:health to see the closed alarms with their real asks.

Demo

N/A here: this PR is server data plus types. The web already renders these lines (W2), and the server's pages go through the live source in nodeWire.server.test.tsx.

Type of change

  • Bug fix
  • Feature
  • UI / frontend change
  • Refactor / chore
  • Docs
  • Test / CI
  • Breaking change

Test coverage

  • Unit tests added / updated
  • Integration tests added / updated
  • E2E tests added / updated
  • Manual verification completed
  • Existing tests cover this change
  • Not applicable

Coverage notes

The contract cases run on the shared 13:40 fixture. The same JSON drives the web's live-source contract test, so server and web cannot drift.

Changelog

Cockpit v3 node pages say why each node is or isn't moving, ending at the person who must act, and show burn, ETA, Try it, what is in the base env and the alarms the server closed. A project that depends on another now starts when that project's work is finished, not after its first merge.

🤖 Generated with Claude Code

…, alarms closed with a cause (S5, S6, S7)

S5. node_status.py computes each node's blocker path over the built tree
(pure, memoized): own waits from waits.py first (the snapshot's waiting_on
for a project, task_waits for a task, a ready PR's merge), ranked stall >
decision > proposal > hold > merge > dependency > capacity; an ask ends the
path, a dependency, another project's merge or a hold on another node hops
there; otherwise it steps into the child that blocks the most (person-ending
paths first, by the weight they hold, then dependents, then column); a node's
own decision (or one on a proposed child) ends it when no child's does; 6
shown segments, cycle guard. Ready PRs a person merges into one base are one
ask (merge:<base>). The header's line carries its label (Not started for 7 h,
Held on purpose, Top blocker, Not moving), also_holds, a numbered why and a
note on met links with no reason; status_compact is the end of the path.

S6. Roll-ups add burn from turn spans, budget sums, an ETA only from
observed task durations (graph.critical_path; a feature's slowest project
per column), Try it from passing acceptance steps, and the base env on the
root. The base watcher records the commit the base env runs
(omnigent.factory.base_env.<key>, served as factory.base.deployed), placed in
the base's history; in_base_env per change is by ancestry (base.contains).

S7. A stall that ran again because its task is now held on purpose is
answered by the server with the cause and the real ask; server closings
record cause, real_ask, about and was on their event. Health lists today's
closed alarms (closed_alarms), the node each was about gets a chip, and the
root roll-up counts them (alarms_closed). An open alarm on a held task is no
ask on the node pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the size/XL Pull request size: XL label Nov 3, 2026
…closed only with a real cause

Review fixes for S5-S7:
- Dependencies in merged mode are met only when the target has no open task
  (gating, waits, snapshot, start, guards, graph): the panel really waits on
  assistant-core while T-16 is open; the test fixture builds waiting_on with
  waits.project_waits like the snapshot does.
- A path that ends at a person wins over own waits that end elsewhere; a
  not-started project nothing holds ends at who starts it (view.start); a
  long path keeps its first steps, an elision and the person; cycle paths are
  not memoized; waits that end elsewhere keep who must act.
- base.contains counts only an exact sha until the deployed head is placed;
  a merge into another repo is never in the base env; smoke only from a probe.
- Expiries count as closed alarms only with an explicit cause; a refuted
  claim's Keep waiting is superseded, not closed; settle records a concrete
  cause; only idle stalls of held tasks are hidden; the real ask shows its verb;
  the alarm read pages until it covers the window.
- Burn counts open turns and reads spans newest first; a parent's budget is
  its children's caps only when each has one; auto-merge targets are read only
  on a tree rebuild.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@LittleChimera
LittleChimera merged commit be787a7 into staging Nov 3, 2026
53 of 66 checks passed
@LittleChimera
LittleChimera deleted the factory/v3-status-rollups branch November 3, 2026 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL Pull request size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant