Skip to content

feat(agents): evaluation & authoring assistant - #1111

Draft
OchnikBartek wants to merge 14 commits into
mainfrom
feat/52-authoring-assistant
Draft

feat(agents): evaluation & authoring assistant#1111
OchnikBartek wants to merge 14 commits into
mainfrom
feat/52-authoring-assistant

Conversation

@OchnikBartek

Copy link
Copy Markdown
Member

Summary

Design-first deliverable for #52, as the issue requires ("a written plan, reviewed by @DEENUU1 before implementation"). This PR carries no platform code — two artifacts in docs/design/ (excluded from the published site):

  • authoring-assistant-plan.md — the plan: ten numbered decisions with their rejected alternatives, a phased work breakdown, an implementation contract distilled from auditing the demo, and nine open questions for review.
  • authoring-assistant-demo.html — a standalone, clickable demo of the review UI on the real Builder's design tokens. Open it in a browser: suggestion diffs with evidence popovers (hover the ✦ icons), Accept / Edit / Dismiss with a draft bar, the async verify cycle, and the replay comparison dialog (second example on the Toolbox tab).

How this PR works

One PR, two stages, same branch:

  1. Now — design review. @DEENUU1 reviews the plan (and clicks the demo); outcomes land in the plan's "Resolved in review" section.
  2. After approval — the full implementation is pushed to this same branch and this description is rewritten to cover it.

What the plan decides (short form)

  1. Ratings and the run record lead; traces enrich. message_ratings (+comments), agent_runs, run_manifests (what the model was actually handed) and the transcript cover mode 2 on every deployment; Logfire is optional with honest degradation.
  2. Propose, never write. Suggestions are anchored hunks with rationale + evidence; accepting lands in the draft; publish stays a human action. Propose-only is also the prompt-injection boundary.
  3. The improver is a seeded, org-owned agent with one runner-assembled selectable=False capability (the channel_tools shape). Its runs are budgeted, metered and visible in Activity for free.
  4. Org-scoped everything, no new permission — existing gates compose (runs:view, agents:edit/publish, skills:edit); cross-tenant refusal tests owed.
  5. Mode 1 (describe → draft) reuses the promote-a-specialist shape — a draft, never a published version.
  6. Synthetic evaluation stays out (phase 2, own milestone); replay on real prompts is in.
  7. Three surfaces, one review grammar — instructions, skill bodies (via the existing skill_proposals), tool descriptions (per-binding tool_overrides).
  8. The assistant also verifies manual edits; the verdict informs and can never block a publish.
  9. Replay picks past prompts for relevance to the edit (on-demand embeddings through the org's RAG credential, honest fallbacks), serves tool calls from the recordings, never executes a divergent call, and stamps RunSurface.REPLAY.
  10. An org-level switch, enforced in the service — off means the endpoints refuse, not just that the UI hides.

Verification

The demo was exercised click-by-click in both themes: every suggestion action (accept / edit / dismiss / undo, both surfaces), the three instruction views, the async verify cycle (spinner → results → re-run under a new run id), the replay dialog with verdict overrides recomputing the summary, and responsive layout. A functional audit then reconciled every number, state and flow; four inconsistencies were fixed and the rest is recorded in the plan's "The demo, audited" section — fifteen contract items the implementation must keep, plus the list of what the demo deliberately fakes.

Refs #52

Issue #52 asks for a plan first: an assistant that helps people write
agents - instructions, skills, tool descriptions - reviewed by DEENUU1
before implementation. The plan corrects three premises that went stale
since the issue was filed (ratings are surfaced in five places now,
run_manifests records what the model actually saw, skill_proposals
already implements propose-not-write) and decides seven questions with
their rejected alternatives: ratings and the run record lead with traces
as optional enrichment; every suggestion is a proposal whose acceptance
lands in the draft, never a published version; the improver is a seeded
org-owned agent with one runner-assembled selectable=False capability;
no new permission - existing gates compose; mode 1 reuses the
promote-a-specialist shape; evaluation is scoped out to its own
milestone; three surfaces share one review grammar.

The demo is a standalone HTML file on the real builder's design tokens:
the instructions diff with evidence popovers (rationale tied to specific
ratings and runs), Accept/Edit/Dismiss with a draft bar, and the same
grammar on a Toolbox tool-description override. Verified interactively
in both themes; nothing in it is wired to a backend.

Both files live in docs/design/, which mkdocs excludes from the
published site.

Refs #52
Three additions from Bartek's review of the first cut, plus one
correction the feasibility pass found:

- Decision 2 reshaped: an instructions proposal is a base snapshot plus
  anchored hunks decided independently, not a whole body - the per-change
  Accept/Edit/Dismiss the demo shows cannot be built on skill_proposals'
  whole-body shape without proposals clobbering each other. Staleness is
  shown, never silently applied. Also names propose-only as the
  prompt-injection boundary and the Builder-autosave race.
- Decision 8: the assistant also verifies manual edits - same improver
  run with the human's diff injected, verdict informs and cannot block a
  publish, actionable findings land as ordinary hunks.
- Decision 9: replay a draft against prompts users already sent - tool
  calls served from tool_calls/manifest recordings so nothing
  side-effecting fires twice, divergent calls marked not executed, runs
  metered with a new RunSurface.REPLAY, draft-vs-shadow-env left open.
- Decision 10: an org-level switch gated on org:settings; the service
  refusal is the boundary, the hidden strip a courtesy.

The demo grew both surfaces: a verification report card (two supported
findings, one regression risk with restore/ignore) and the replay
comparison dialog (five prompt pairs, judge suggestion with human
override recomputing the summary, a divergence badge, spend shown).
Verified interactively.

Refs #52
Two additions to Decision 9 from review:

- A skill edit replays over the runs that loaded it - queryable today
  (tool_calls rows with tool_name='load_skill' and the name in args) -
  each pair under its own recorded agent version, the candidate body
  injected as a resource override, so no publish question arises. The
  replay toolset serves the candidate body for the edited skill and the
  recording for everything else.
- The replay set (five pairs by default) is ranked by similarity to the
  edited hunks, embedded on demand through the org's existing RAG
  embedding credential - nearest-neighbour gives the topical fallback
  for free - degrading to lexical match and rated-down-first where no
  credential exists, with each pair labelled by why it was picked.
  Deliberately the embedding client only, never the RAG pipeline:
  conversations are not ingested into collections.

The demo's replay dialog shows the picked-because label per pair and
the matched-to-the-edit summary chip. Open question 9 records the
deferred standing-index alternative.

Refs #52
The manual-edit path (edit by hand, verify, then test) had no way into
the replay dialog - the draft bar's button only appears after accepting
an assistant suggestion. The verification card now carries the same
button, closing the second of the two journeys the plan describes
(Decision 9: after editing, either direction).

Refs #52
The verify button now behaves the way the real run will: click turns it
into a disabled Verifying spinner, completion swaps in See results plus
a fresh Verify my edits beside it (the prompt may have changed again),
and a re-run keeps the old results reachable while it spins, then
regenerates the report under a new improver-run id. The strip wraps its
action row so four buttons no longer crush the summary column at narrow
widths.

Refs #52
A full functional pass over the demo against itself, the plan and the
product's mechanics. Four inconsistencies found and fixed in the demo:

- one popover counted comments where its chip counted ratings (3 of 5
  vs 3 of 6) - unified on ratings, and the rule recorded;
- Dismiss all cleared only the Build tab while the chip counted both
  surfaces - bulk actions now share the chip's scope;
- the divergent replay pair showed a finished answer, impossible when
  the divergent call is never executed - it now shows the attempted
  call, its arguments and the not-executed notice, and stays judgeable;
- a static tab-count fallback disagreed with the computed one, and
  'rated answers' now says rated-down.

The plan gains 'The demo, audited' - fifteen contract items the
implementation owes (entry conditions, view semantics, verify lifecycle
as run status, re-run over the current draft, undo returning the
original proposal, replay selection over the accepted diff only, spend
preview, linked run references, reconciling rating arithmetic,
supersede-on-rerun, ICU copy, recorded costs) and the explicit list of
demo fakes that must not be copied.

Refs #52
CodeQL flagged js/xss-through-dom on the demo's edit flow: diff-row
textContent was interpolated into an insertAdjacentHTML template, so
DOM text was reinterpreted as HTML. Harmless with the demo's fixed
content, but exactly the sink the real Builder must never have - the
box, textarea and buttons are now created with DOM APIs and the text
travels through textarea.value only. Edit -> save & accept re-verified
in the browser.

Refs #52
@OchnikBartek OchnikBartek added the enhancement New feature or request label Aug 25, 2026
@OchnikBartek OchnikBartek self-assigned this Aug 25, 2026
@OchnikBartek
OchnikBartek requested a review from DEENUU1 August 25, 2026 10:30
@OchnikBartek

Copy link
Copy Markdown
Member Author

@DEENUU1 — the review packet in one place. The plan is the document under review; the demo is its visual argument (open the HTML in a browser and click around — everything works offline). Below: what is settled and what needs your call.

Settled — push back on any of these

  • Issue premises corrected before designing. "Ratings surfaced nowhere" went stale (rated=down filter, transcript comments, /stats/ratings/summary, per-version counts all shipped since); run_manifests — which the issue never mentions — beats traces as the mode-2 source because it records what the model was actually handed, on every deployment.
  • The proposal shape diverges from skill_proposals on purpose. Instructions proposals are a base snapshot + anchored hunks (old_text/new_text/rationale/evidence) decided independently — per-change Accept/Edit/Dismiss cannot be built on a whole-body row without proposals clobbering each other. A hunk whose anchor no longer matches renders stale with side-by-side, never applies silently.
  • The improver is an ordinary platform agent (seeded per org, editable, its own prompt improvable), and its capability is selectable=False, assembled by the runner per run like channel_tools — so no ordinary agent can be handed org-wide run/rating reads from the Toolbox, and the capability never touches the DB (resources are resolved org-scoped and injected).
  • No new permission. Starting an improve run = runs:view + agents:edit on the target via resolve_access; accept = agents:edit / skills:edit; publish = agents:publish. A new authoring:* perm would be a second name for the same authority.
  • Replay never re-fires a side effect. Tool calls are answered from tool_calls/manifest recordings; a divergent call (a tool the recorded run never used) is not executed — the pair is marked divergent and stays judgeable. Skills replay over the runs that loaded them (tool_name='load_skill'), each pair under its own recorded agent version, candidate body injected as a resource override.
  • Replay pairs are picked for relevance to the accepted diff — on-demand embeddings through the org's existing RAG credential (nearest-neighbour gives the topical fallback for free), degrading to lexical then rated-down-first, each pair labelled with why it was picked. The embedding client is shared; conversations are never ingested into collections.
  • The verify verdict informs, never blocks a publish.
  • The org switch is a service refusal, not a hidden button.
  • The demo was audited and the plan's "The demo, audited" section is binding: fifteen contract items (entry conditions, view semantics, verify-state = run status, re-run over the current draft, undo restores the original proposal, spend preview before replay, linked run refs, reconciling rating arithmetic, supersede-on-rerun, ICU copy, recorded costs) plus the explicit list of demo fakes not to copy.

Questions for you — answers land in the plan's "Resolved in review"

  1. Seeded improver visibility — ordinary row in the Agents list, editable (my lean), or a hidden system row?
  2. Approval gate on Accept? My lean: no — the draft is the review stage and publish is the second gate.
  3. Where mode 1 lives — a "Describe it" panel in the create dialog, or a chat-first flow ending in a draft? Cheap to decide late.
  4. Default budget cap on the seeded improver — ship the seed with budget.monthly_usd set?
  5. Proposal retention — decided proposals kept forever (audit trail) or swept after N days?
  6. How replay runs a draft — explicit-spec escape hatch in prepare (my lean: version history stays clean) vs publish-to-shadow-environment (reuses more, pollutes versions).
  7. Switch granularity — one org toggle (my lean) or per-surface (proposals / verify / replay)?
  8. Who judges a replay pair — human-only in v1 with the judge off by default, or judge on from the start? (Judge calls cost money and add a second model opinion to govern.)
  9. Standing embedding index over opening prompts — defer until on-demand embedding measurably hurts (my lean), or build now?

Process note: after your approval the full implementation lands on this same branch and the PR description is rewritten to cover it.

The automated review on #1109 (closed by the branch rename) raised ten
findings; all ten hold and the plan absorbs them:

- decision state moves onto the hunk (per-hunk status + decided_by/at);
  the row keeps lifecycle only, and undo is legal while the proposal is
  open - a row-level status cannot represent one hunk accepted and one
  dismissed, and terminal-once-decided contradicted undo;
- draft writes take optimistic concurrency (a revision echoed back),
  because an in-flight whole-spec autosave landing after Accept would
  silently drop the accepted hunk;
- replayed tool calls match by tool id, normalized args and occurrence
  order - an args mismatch is divergent, never served an unrelated
  recording;
- the draft-first guarantee is narrowed honestly: applying a skill
  proposal is a live change (skills have no draft stage), stated in the
  one-sentence design, Decision 2 and the Apply control's copy;
- skill_proposals is extended additively (rationale, evidence, base
  version) so skills can carry the unified review grammar the plan
  promises - the existing table stores only the replacement body;
- tool-override proposals get base snapshots and staleness like hunks;
- the org switch covers mode 1's describe-to-draft run, the fourth
  AI-authoring entry point the refusal list missed;
- v1 has no Logfire read path at all - both tokens are write
  credentials - so trace enrichment is future work, not a fallback;
- the embedding credential for replay selection is an explicit org
  setting (resolution is per collection; there is no org-wide RAG
  credential to borrow);
- consent moves before the selection embedding, which already spends -
  confirming the preview is what creates the replay session.

Work breakdown and contract items updated to match.

Refs #52

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fe75a994dd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md
Comment thread docs/design/authoring-assistant-plan.md
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
@OchnikBartek

Copy link
Copy Markdown
Member Author

Codex reviewed the superseded #1109 and left ten findings on the plan. All ten held up against the code and are adopted in 8475a080:

  1. Per-hunk decision state — status + decided_by/decided_at move onto the hunk; the row keeps lifecycle (open/superseded/closed). Undo is legal while the proposal is open; decided-once applies to the closed proposal. (Fixes the row-status ↔ independent-hunks contradiction.)
  2. Optimistic concurrency on the draft — an in-flight whole-spec autosave landing after Accept would silently drop the hunk; draft_spec gains a revision the client echoes back, stale writes are refused and re-merged.
  3. Replay matches recordings by tool id + normalized args + occurrence order — an args mismatch is divergent, never served an unrelated result.
  4. The draft-first guarantee is narrowed honestly: applying a skill proposal is a live change (skills have no draft stage). Stated in the one-sentence design; replay-before-apply is the test stage a draft would otherwise provide.
  5. skill_proposals extended additively (rationale, evidence, base version) so skills can carry the unified review grammar; the current table stores only the replacement body.
  6. Tool-override proposals get base snapshots + staleness like hunks — never re-add a removed binding, never overwrite a newer manual edit.
  7. The org switch covers mode 1 (describe→draft) — the fourth AI-authoring entry point the refusal list missed.
  8. v1 has no Logfire read path — both tokens are write credentials — so trace enrichment is explicitly future work, not a fallback.
  9. The replay-selection embedding credential is an explicit org setting — embedding resolution is per collection, there is no org-wide credential to borrow.
  10. Consent moves before the selection embedding — the preview shows selection + run cost together and confirming creates the session the embedding is metered into.

The CodeQL js/xss-through-dom finding on the demo was fixed earlier (fe75a994).

Codex re-reviewed on the pre-adoption commit; four of its twelve threads
were already answered by the previous commit. The eight new findings all
hold and the plan absorbs them:

- a hunk's anchor is a base-range position with context, not bare
  old_text - duplicate lines are ambiguous and an empty old_text
  insertion matches everywhere; ambiguity goes stale, never applies at
  the wrong place;
- one open batch per agent, tool-override entries keyed by
  (capability_id, tool_id) - one-pending-per-target could not hold two
  tool descriptions from one run;
- the improver stages skill changes through an explicit propose_skill
  tool carrying rationale and evidence - the workspace-write seam only
  sees filesystem changes, so the promised evidence popover had no data
  source on the skill path;
- pending skill proposals key on (org, skill, proposing agent) - the
  existing recorder dedups by skill alone, so an improver proposal would
  replace another agent's pending edit and corrupt its attribution;
- skill authoring gates on skills:edit for the target skill via
  resolve_access - skills carry their own ownership and grants;
- starting any authoring run also resolves agents:run on the seeded
  improver - grants are independent and starting it spends money;
- load_skill lookups resolve each run's frozen version stable-id
  mapping - tool_calls records the effective, possibly renamed, name;
- attachment-bearing prompts are excluded from the replay pool and the
  exclusion is reported - replaying their text alone manufactures
  regressions.

Refs #52
@OchnikBartek

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for security reviews. Please try again later.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: babae32d36

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread docs/design/authoring-assistant-plan.md
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md
Comment thread docs/design/authoring-assistant-plan.md
@OchnikBartek

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for security reviews. Please try again later.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: babae32d36

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
@OchnikBartek

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for security reviews. Please try again later.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: babae32d36

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
@OchnikBartek

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for security reviews. Please try again later.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: babae32d36

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread docs/design/authoring-assistant-plan.md
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
@OchnikBartek

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for security reviews. Please try again later.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: babae32d36

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-demo.html Outdated
Comment thread docs/design/authoring-assistant-plan.md
Comment thread docs/design/authoring-assistant-demo.html Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Comment thread docs/design/authoring-assistant-plan.md Outdated
Thirty-one findings across three review passes on the design PR; all
hold and the plan and demo absorb them. The themes:

Proposal mechanics: batch lifecycle is open-until-publish-or-supersede
(deciding the last hunk no longer strands Undo); supersession orders by
the source run's start time, not finish order; a hunk decision and its
draft mutation are one conditional transaction (concurrent
Accept/Dismiss resolve to one outcome); Undo is anchored and goes stale
after a manual edit of the accepted text; skill Apply enforces its
recorded base version; evidence references bind to the target, not the
tenant; mode-1 turns update one conversation-scoped draft instead of
minting siblings; the authoring capability builds narrowed for
targetless mode-1 runs; read_agent_spec carries the resolved capability
catalog so an unoverridden tool has a nameable (capability_id, tool_id).

Replay integrity: the explicit-spec path validates the draft
caller-scoped before assembly, carries no agent_version_id (a candidate
must not book onto a published version), and freezes the candidate
snapshot for the whole session; the shadow-environment alternative is
marked unsafe as-built (publish moves tracks-latest environments); each
pair holds the recorded model and recorded resources constant, replacing
only the edited variable; recorded calls match by resolved stable id so
a candidate rename is not divergence; the skill candidate is served
whole (body and resources, across all three skill tools); later-turn
skill pairs replay with the recorded conversation prefix; the candidate
pool is capped deterministically before embedding, and unanswered or
attachment-bearing prompts never enter it; relevance text is defined per
surface, not only from instruction hunks.

Governance: replay resolves agents:run on every agent it executes, and
skill authoring on skills:edit for the skill plus agents:run on the
improver; verdict writes take the session creator or the target's edit
permission; skill Verify is defined against a pending proposal (skills
keep no previous body); sessions key on target kind + id since skill
pairs span agents; rating_counts_by_version is read as rates with a
minimum-sample caveat; manifests are best-available evidence (absent or
truncated ones are skipped or surfaced); verification findings merge
into the open batch instead of superseding pending suggestions.

Demo: the Draft updated badge now shows after a tool-only accept, and
human replay verdicts start unset - the judge suggests beside the
control, the summary counts undecided pairs as undecided.

Refs #52
@vstorm-co vstorm-co deleted a comment from chatgpt-codex-connector Bot Aug 26, 2026
The engine writeup needs a visible governance control: the assistant
strip now carries a Manual / Canary / Auto segmented toggle with a note
that changes per mode - manual (nothing live without a person), canary
(accepted change ships to a traffic slice, replay must pass, one-click
rollback), auto (eval-gated graduation behind replay + canary + org
switch + budget cap, off by default). Verified in the browser.

Refs #52
Five headless-Chrome captures of the demo states referenced by the
engine writeup: the assistant strip with the apply-mode toggle, the
instructions diff with an evidence popover, the Toolbox proposed
tool description, the replay comparison dialog, and the verification
card.
Screens 3 and 5 had a large empty band and clipped content. Recaptured:
the Toolbox shot now shows the full two-column layout with the proposed
search_documents override, and the verification card sits at the top of
the frame with all findings, the regression warning and the footer
visible.
All five captures now use the 1300px viewport so the sidebar and app
chrome are visible and it is clear where in the app each screen sits -
previously screens 1 and 2 were captured narrow, hiding the sidebar.
@DEENUU1
DEENUU1 marked this pull request as draft September 5, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant