feat(agents): evaluation & authoring assistant - #1111
Conversation
Issue #52 asks for a plan first: an assistant that helps people write agents - instructions, skills, tool descriptions - reviewed by DEENUU1 before implementation. The plan corrects three premises that went stale since the issue was filed (ratings are surfaced in five places now, run_manifests records what the model actually saw, skill_proposals already implements propose-not-write) and decides seven questions with their rejected alternatives: ratings and the run record lead with traces as optional enrichment; every suggestion is a proposal whose acceptance lands in the draft, never a published version; the improver is a seeded org-owned agent with one runner-assembled selectable=False capability; no new permission - existing gates compose; mode 1 reuses the promote-a-specialist shape; evaluation is scoped out to its own milestone; three surfaces share one review grammar. The demo is a standalone HTML file on the real builder's design tokens: the instructions diff with evidence popovers (rationale tied to specific ratings and runs), Accept/Edit/Dismiss with a draft bar, and the same grammar on a Toolbox tool-description override. Verified interactively in both themes; nothing in it is wired to a backend. Both files live in docs/design/, which mkdocs excludes from the published site. Refs #52
Three additions from Bartek's review of the first cut, plus one correction the feasibility pass found: - Decision 2 reshaped: an instructions proposal is a base snapshot plus anchored hunks decided independently, not a whole body - the per-change Accept/Edit/Dismiss the demo shows cannot be built on skill_proposals' whole-body shape without proposals clobbering each other. Staleness is shown, never silently applied. Also names propose-only as the prompt-injection boundary and the Builder-autosave race. - Decision 8: the assistant also verifies manual edits - same improver run with the human's diff injected, verdict informs and cannot block a publish, actionable findings land as ordinary hunks. - Decision 9: replay a draft against prompts users already sent - tool calls served from tool_calls/manifest recordings so nothing side-effecting fires twice, divergent calls marked not executed, runs metered with a new RunSurface.REPLAY, draft-vs-shadow-env left open. - Decision 10: an org-level switch gated on org:settings; the service refusal is the boundary, the hidden strip a courtesy. The demo grew both surfaces: a verification report card (two supported findings, one regression risk with restore/ignore) and the replay comparison dialog (five prompt pairs, judge suggestion with human override recomputing the summary, a divergence badge, spend shown). Verified interactively. Refs #52
Two additions to Decision 9 from review: - A skill edit replays over the runs that loaded it - queryable today (tool_calls rows with tool_name='load_skill' and the name in args) - each pair under its own recorded agent version, the candidate body injected as a resource override, so no publish question arises. The replay toolset serves the candidate body for the edited skill and the recording for everything else. - The replay set (five pairs by default) is ranked by similarity to the edited hunks, embedded on demand through the org's existing RAG embedding credential - nearest-neighbour gives the topical fallback for free - degrading to lexical match and rated-down-first where no credential exists, with each pair labelled by why it was picked. Deliberately the embedding client only, never the RAG pipeline: conversations are not ingested into collections. The demo's replay dialog shows the picked-because label per pair and the matched-to-the-edit summary chip. Open question 9 records the deferred standing-index alternative. Refs #52
The manual-edit path (edit by hand, verify, then test) had no way into the replay dialog - the draft bar's button only appears after accepting an assistant suggestion. The verification card now carries the same button, closing the second of the two journeys the plan describes (Decision 9: after editing, either direction). Refs #52
The verify button now behaves the way the real run will: click turns it into a disabled Verifying spinner, completion swaps in See results plus a fresh Verify my edits beside it (the prompt may have changed again), and a re-run keeps the old results reachable while it spins, then regenerates the report under a new improver-run id. The strip wraps its action row so four buttons no longer crush the summary column at narrow widths. Refs #52
A full functional pass over the demo against itself, the plan and the product's mechanics. Four inconsistencies found and fixed in the demo: - one popover counted comments where its chip counted ratings (3 of 5 vs 3 of 6) - unified on ratings, and the rule recorded; - Dismiss all cleared only the Build tab while the chip counted both surfaces - bulk actions now share the chip's scope; - the divergent replay pair showed a finished answer, impossible when the divergent call is never executed - it now shows the attempted call, its arguments and the not-executed notice, and stays judgeable; - a static tab-count fallback disagreed with the computed one, and 'rated answers' now says rated-down. The plan gains 'The demo, audited' - fifteen contract items the implementation owes (entry conditions, view semantics, verify lifecycle as run status, re-run over the current draft, undo returning the original proposal, replay selection over the accepted diff only, spend preview, linked run references, reconciling rating arithmetic, supersede-on-rerun, ICU copy, recorded costs) and the explicit list of demo fakes that must not be copied. Refs #52
CodeQL flagged js/xss-through-dom on the demo's edit flow: diff-row textContent was interpolated into an insertAdjacentHTML template, so DOM text was reinterpreted as HTML. Harmless with the demo's fixed content, but exactly the sink the real Builder must never have - the box, textarea and buttons are now created with DOM APIs and the text travels through textarea.value only. Edit -> save & accept re-verified in the browser. Refs #52
|
@DEENUU1 — the review packet in one place. The plan is the document under review; the demo is its visual argument (open the HTML in a browser and click around — everything works offline). Below: what is settled and what needs your call. Settled — push back on any of these
Questions for you — answers land in the plan's "Resolved in review"
Process note: after your approval the full implementation lands on this same branch and the PR description is rewritten to cover it. |
The automated review on #1109 (closed by the branch rename) raised ten findings; all ten hold and the plan absorbs them: - decision state moves onto the hunk (per-hunk status + decided_by/at); the row keeps lifecycle only, and undo is legal while the proposal is open - a row-level status cannot represent one hunk accepted and one dismissed, and terminal-once-decided contradicted undo; - draft writes take optimistic concurrency (a revision echoed back), because an in-flight whole-spec autosave landing after Accept would silently drop the accepted hunk; - replayed tool calls match by tool id, normalized args and occurrence order - an args mismatch is divergent, never served an unrelated recording; - the draft-first guarantee is narrowed honestly: applying a skill proposal is a live change (skills have no draft stage), stated in the one-sentence design, Decision 2 and the Apply control's copy; - skill_proposals is extended additively (rationale, evidence, base version) so skills can carry the unified review grammar the plan promises - the existing table stores only the replacement body; - tool-override proposals get base snapshots and staleness like hunks; - the org switch covers mode 1's describe-to-draft run, the fourth AI-authoring entry point the refusal list missed; - v1 has no Logfire read path at all - both tokens are write credentials - so trace enrichment is future work, not a fallback; - the embedding credential for replay selection is an explicit org setting (resolution is per collection; there is no org-wide RAG credential to borrow); - consent moves before the selection embedding, which already spends - confirming the preview is what creates the replay session. Work breakdown and contract items updated to match. Refs #52
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fe75a994dd
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
Codex reviewed the superseded #1109 and left ten findings on the plan. All ten held up against the code and are adopted in
The CodeQL |
Codex re-reviewed on the pre-adoption commit; four of its twelve threads were already answered by the previous commit. The eight new findings all hold and the plan absorbs them: - a hunk's anchor is a base-range position with context, not bare old_text - duplicate lines are ambiguous and an empty old_text insertion matches everywhere; ambiguity goes stale, never applies at the wrong place; - one open batch per agent, tool-override entries keyed by (capability_id, tool_id) - one-pending-per-target could not hold two tool descriptions from one run; - the improver stages skill changes through an explicit propose_skill tool carrying rationale and evidence - the workspace-write seam only sees filesystem changes, so the promised evidence popover had no data source on the skill path; - pending skill proposals key on (org, skill, proposing agent) - the existing recorder dedups by skill alone, so an improver proposal would replace another agent's pending edit and corrupt its attribution; - skill authoring gates on skills:edit for the target skill via resolve_access - skills carry their own ownership and grants; - starting any authoring run also resolves agents:run on the seeded improver - grants are independent and starting it spends money; - load_skill lookups resolve each run's frozen version stable-id mapping - tool_calls records the effective, possibly renamed, name; - attachment-bearing prompts are excluded from the replay pool and the exclusion is reported - replaying their text alone manufactures regressions. Refs #52
|
@codex review |
|
You have reached your Codex usage limits for security reviews. Please try again later. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: babae32d36
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
|
You have reached your Codex usage limits for security reviews. Please try again later. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: babae32d36
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
|
You have reached your Codex usage limits for security reviews. Please try again later. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: babae32d36
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
|
You have reached your Codex usage limits for security reviews. Please try again later. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: babae32d36
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
|
You have reached your Codex usage limits for security reviews. Please try again later. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: babae32d36
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Thirty-one findings across three review passes on the design PR; all hold and the plan and demo absorb them. The themes: Proposal mechanics: batch lifecycle is open-until-publish-or-supersede (deciding the last hunk no longer strands Undo); supersession orders by the source run's start time, not finish order; a hunk decision and its draft mutation are one conditional transaction (concurrent Accept/Dismiss resolve to one outcome); Undo is anchored and goes stale after a manual edit of the accepted text; skill Apply enforces its recorded base version; evidence references bind to the target, not the tenant; mode-1 turns update one conversation-scoped draft instead of minting siblings; the authoring capability builds narrowed for targetless mode-1 runs; read_agent_spec carries the resolved capability catalog so an unoverridden tool has a nameable (capability_id, tool_id). Replay integrity: the explicit-spec path validates the draft caller-scoped before assembly, carries no agent_version_id (a candidate must not book onto a published version), and freezes the candidate snapshot for the whole session; the shadow-environment alternative is marked unsafe as-built (publish moves tracks-latest environments); each pair holds the recorded model and recorded resources constant, replacing only the edited variable; recorded calls match by resolved stable id so a candidate rename is not divergence; the skill candidate is served whole (body and resources, across all three skill tools); later-turn skill pairs replay with the recorded conversation prefix; the candidate pool is capped deterministically before embedding, and unanswered or attachment-bearing prompts never enter it; relevance text is defined per surface, not only from instruction hunks. Governance: replay resolves agents:run on every agent it executes, and skill authoring on skills:edit for the skill plus agents:run on the improver; verdict writes take the session creator or the target's edit permission; skill Verify is defined against a pending proposal (skills keep no previous body); sessions key on target kind + id since skill pairs span agents; rating_counts_by_version is read as rates with a minimum-sample caveat; manifests are best-available evidence (absent or truncated ones are skipped or surfaced); verification findings merge into the open batch instead of superseding pending suggestions. Demo: the Draft updated badge now shows after a tool-only accept, and human replay verdicts start unset - the judge suggests beside the control, the summary counts undecided pairs as undecided. Refs #52
The engine writeup needs a visible governance control: the assistant strip now carries a Manual / Canary / Auto segmented toggle with a note that changes per mode - manual (nothing live without a person), canary (accepted change ships to a traffic slice, replay must pass, one-click rollback), auto (eval-gated graduation behind replay + canary + org switch + budget cap, off by default). Verified in the browser. Refs #52
Five headless-Chrome captures of the demo states referenced by the engine writeup: the assistant strip with the apply-mode toggle, the instructions diff with an evidence popover, the Toolbox proposed tool description, the replay comparison dialog, and the verification card.
Screens 3 and 5 had a large empty band and clipped content. Recaptured: the Toolbox shot now shows the full two-column layout with the proposed search_documents override, and the verification card sits at the top of the frame with all findings, the regression warning and the footer visible.
All five captures now use the 1300px viewport so the sidebar and app chrome are visible and it is clear where in the app each screen sits - previously screens 1 and 2 were captured narrow, hiding the sidebar.
Summary
Design-first deliverable for #52, as the issue requires ("a written plan, reviewed by @DEENUU1 before implementation"). This PR carries no platform code — two artifacts in
docs/design/(excluded from the published site):authoring-assistant-plan.md— the plan: ten numbered decisions with their rejected alternatives, a phased work breakdown, an implementation contract distilled from auditing the demo, and nine open questions for review.authoring-assistant-demo.html— a standalone, clickable demo of the review UI on the real Builder's design tokens. Open it in a browser: suggestion diffs with evidence popovers (hover the ✦ icons), Accept / Edit / Dismiss with a draft bar, the async verify cycle, and the replay comparison dialog (second example on the Toolbox tab).How this PR works
One PR, two stages, same branch:
What the plan decides (short form)
message_ratings(+comments),agent_runs,run_manifests(what the model was actually handed) and the transcript cover mode 2 on every deployment; Logfire is optional with honest degradation.selectable=Falsecapability (thechannel_toolsshape). Its runs are budgeted, metered and visible in Activity for free.runs:view,agents:edit/publish,skills:edit); cross-tenant refusal tests owed.skill_proposals), tool descriptions (per-bindingtool_overrides).RunSurface.REPLAY.Verification
The demo was exercised click-by-click in both themes: every suggestion action (accept / edit / dismiss / undo, both surfaces), the three instruction views, the async verify cycle (spinner → results → re-run under a new run id), the replay dialog with verdict overrides recomputing the summary, and responsive layout. A functional audit then reconciled every number, state and flow; four inconsistencies were fixed and the rest is recorded in the plan's "The demo, audited" section — fifteen contract items the implementation must keep, plus the list of what the demo deliberately fakes.
Refs #52