feat(bench): execute restart-safe runtime candidates through Pier

[drewstone]

Summary

• extend the existing runtime candidate preparation, atomic execution, protected trace, and grader primitives into the official Pier custom-agent path
• bind task, repository, candidate, profile, OCI image, grader, instruction, workspace, and result identities before launch
• persist only non-secret process and Docker ownership so a fresh evaluator process can stop and clean an interrupted attempt
• export the Pier controller from @tangle-network/agent-bench and update the bench package to the published runtime/eval/interface/sandbox releases

Grep-before-build result: this reuses prepareAgentCandidateExecution, executeAgentCandidateAtomically, protected TraceStore capture, and the existing benchmark adapters; it does not add a second candidate executor or grader path.

Verification

• pnpm typecheck:public
• pnpm test — 21/21 TypeScript test files plus the Python Pier bridge
• pnpm verify:package — packed consumer installed @tangle-network/agent-runtime@0.92.1 and prepared/disposed a real candidate
• PIER_REPO=/home/drew/code/pier pnpm verify:pier — failure control 0/1, fresh-process recovery removed process and container, success control 1/1; both controls recorded 0 model calls, 0 input/output tokens, and $0
• git merge-tree --write-tree origin/main HEAD

No model inference was used by the verification fixtures.

[tangletools]

✅ Auto-approved drewstone PR — 23bf4991

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-07-11T15:56:31Z}

[tangletools]

🟢 Value Audit — sound

Verdict	sound
Concerns	2 (2 low)
Heuristic	0.2s
Duplication	0.0s
Interrogation	179.1s (2 bridge agents)
Total	179.3s

💰 Value — sound

Adds Pier (Datacurve's harness) as a restart-safe transport for the existing immutable candidate-execution spine, correctly reusing the runtime's executor port and atomic execution path — no equivalent existed and no better approach found.

• What it does: Extends the runtime's frozen candidate-execution pipeline (prepare → execute → finalize → grade) to run through Pier's official task container and verifier. A new Pier executor implements the runtime's existing AgentCandidateExecutorPort: it stages the runtime's exact canonical execution-plan and materialization-receipt bytes to disk, a Python custom agent (TangleCandidateAgent) rechecks those sig
• Goals it achieves: (1) Open Pier's official verifier as a deployable grading path for frozen Tangle candidates, widening the benchmark set the eval flywheel can score against. (2) Preserve the tamper-evidence invariant: the runtime stays the sole authority for verification, model access, trace capture, and settlement; Pier only replays signed bytes and cannot self-author a score. (3) Make an interrupted candidate at
• Assessment: Architecturally sound and in the grain of the codebase. The change implements AgentCandidateExecutorPort (src/candidate-execution/types.ts:479) — the runtime's documented extension seam — and delegates to executePreparedAgentCandidate (src/candidate-execution/execute.ts:75), so it rides the single existing atomic execution path rather than forking a second executor. The Python bridge (candidate_co
• Better / existing approach: none — this is the right approach. Searched for an existing container/docker executor implementing AgentCandidateExecutorPort (grep'd src/ and bench/src/): none exists beyond test mocks and this Pier implementation, so there is nothing to reuse or extend. Checked whether pier-workspace-archive.ts duplicates runtime workspace logic: it does not — the runtime's artifacts.ts scans live directories ag
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

A Pier transport adapter for the runtime's existing candidate-execution port, with a serious restart-safe recovery story; reuses the atomic execution path rather than duplicating it, and is exercised end-to-end against a real Pier checkout.

• Integration: Cleanly wired. bench/src/pier-agent.ts:538 constructs an AgentCandidateExecutorPort whose execute/stopAndCapture delegate to a Pier trial controller, and pier-agent.ts:616 calls the runtime's existing executePreparedAgentCandidate (the same atomic path the improvement-cycle uses at src/intelligence/improvement-cycle.ts:253). Everything is exported from bench/src/index.ts:57-81. The Python `Tan
• Fit with existing patterns: Matches the codebase's adapter-implements-port grain exactly. The runtime owns AgentCandidateExecutorPort (src/candidate-execution/types.ts:479) and stays domain-clean; the bench package owns the Pier-specific transport — consistent with CLAUDE.md's layering rule that domain behavior enters by injection. The recovery executor (pier-agent.ts:128) implements the same stopAndCapture-only contract
• Real-world viability: Designed for realistic crash/restart conditions. FilePierCandidateTrialController (pier-trial-controller.ts:411) persists only non-secret process/container identity (pid, process-group, session, start-tick, Docker compose project labels) via atomic fsync'd writes, so a fresh evaluator process can locate and kill an abandoned trial. Process-identity verification via /proc/pid/stat start-tick matchi
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

🔎 Heuristic Signals

🟡 Cruft: console debug added bench/scripts/run-package-tests.mjs

+console.log(package tests passed: ${tests.length}/${tests.length} TypeScript files + Pier bridge)

🟡 Cruft: magic number added bench/pier_agents/tangle_candidate.py

•    timeout_seconds = self._contract.timeout_ms / 1000
  

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260711T160116Z}

[tangletools]

✅ No Blockers — 23bf4991

Review health 100/100 · Reviewer score 45/100 · Confidence 95/100 · 12 findings (2 medium, 10 low)

deepseek: Correctness 45 · Security 45 · Testing 45 · Architecture 45

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 8/8 planned shots over 37 changed files. Global verifier still owns final merge decision.

🟠 MEDIUM Unsound 'as' cast on incomplete StagedPierCandidateExecution in recovery test — bench/scripts/verify-pier-recovery.mts

Line 61 casts { executionId, evaluatorEnv: {} } as StagedPierCandidateExecution, hiding that 10+ required fields are missing (directory, taskDirectory, profileDirectory, planPath, receiptPath, agentArgs, attemptArgs, etc.). The test works because FilePierCandidateTrialController.start() only accesses staged.executionId at line 447 before delegating to the launch callback, and this test's launch callback ignores staged entirely. However, if start() is ever extended to read additional staged fields d

🟠 MEDIUM waitForResult grace period can extend past deadline by up to 15s — bench/src/pier-trial-controller.ts

waitMs = Math.max(1, deadlineAtMs - Date.now() + 15_000) adds an unconditional 15-second grace period to the deadline. When deadlineAtMs is already past, this computes to a positive wait (e.g., if the deadline expired 10s ago, it waits 5s). The runtime enforces deadline via AbortSignal separately, but the controller's own wait can exceed the advertised deadline by up to 15s, wasting evaluator time on an already-timed-out trial.

🟡 LOW Docs sample omits second launch callback parameter (context) — bench/HARNESS.md

The code sample shows launch: (staged) => ({ ... }) but the actual type signature (pier-trial-controller.ts:91-99) is launch: (staged, context) => PierCandidateProcessSpec. The context object carries request (AgentCandidateExecutorRequest), traceStore, signal (AbortSignal), and deadlineAtMs — all material for production callers. The verify-pier-agent.mts production script at line 522 correctly destructures the second parameter: (staged, { request }). Fix: add the second parameter to the docs sample to prevent implementers from missing deadline/signal handling.

🟡 LOW Inconsistent delivery schema validation: utf8-file allows extra fields — bench/pier_agents/candidate_contract.py

argv-append and stdin-utf8 delivery kinds enforce strict field-set (set(delivery) != {'kind'} raises error). The utf8-file block (line 324) only checks env and path values but does not reject extra keys. Extra fields are silently ignored, which is inconsistent with other delivery kinds and could mask plan construction bugs. Fix: add set(delivery) != {'kind', 'env', 'path'} check in the utf8-file branch.

🟡 LOW Tests not discoverable by standard pytest collection — bench/pier_agents/tangle_candidate_test.py

bench/pier_agents/__init__.py imports TangleCandidateAgent which imports pier at module level. Pytest collection triggers __init__.py before the test module's top-level stubs run, causing ModuleNotFoundError: No module named 'pier'. Tests run correctly via python test.py with PYTHONPATH set and pass all 13 cases, but they cannot be collected by pytest. This means CI pipelines cannot discover them through standard pytest bench/ or test runners that rely on collection. Fix: either make __init__.py import lazy (import inside a function) or add a conftest.py fixture that installs pier stubs before collection.

🟡 LOW agent-eval version split: agent-knowledge uses 0.113.0, project uses 0.114.0 — bench/pnpm-lock.yaml

agent-knowledge@1.11.2 resolves @tangle-network/agent-eval@0.113.0 while the project directly depends on 0.114.0. Previously (before this PR) both used 0.108.1 — a single version. Two eval copies now coexist in node_modules. If agent-knowledge exports types parameterized on agent-eval schemas, TypeScript may reject them when consumed by project code using the 0.114.0 types. Verify agent-knowledge@1.11.2 can be upgraded to declare ^0.114.0, or confirm no eval-typed exports cross the boundary.

🟡 LOW Python venv setup installs zero packages, relying on test-internal stubs — bench/scripts/run-package-tests.mjs

Line 42 runs python3 -m venv .venv but never installs any packages into it. The Python tests currently pass because tangle_candidate_test.py (lines 17-68) injects stub pier.* modules into sys.modules before importing the bridge, and datacurve_pier is only accessed at runtime via importlib.metadata.version(), not at module import time. If any future pier_agents module adds a hard top-level import of a third-party package, these tests will fail with ModuleNotFoundError with no obvious cause. Adding a `r

🟡 LOW verify_pier_import.py embedded in packed consumer duplicates test stubs without sharing — bench/scripts/verify-packed-consumer.mjs

The verify_pier_import.py inline content (lines 100-132) re-implements the same pier.* module stubs that exist in tangle_candidate_test.py lines 17-68. These two copies will inevitably drift. The packed consumer's goal is to prove pier_agents.tangle_candidate.TangleCandidateAgent is importable from the installed package — this could be achieved by running the existing test file itself (python3 -m pytest pier_agents/ or python3 -m unittest pier_agents.tangle_candidate_test) instead of maint

🟡 LOW prepareOnly mode hides proofArm default that changes output digest identity — bench/scripts/verify-pier-agent.mts

When PIER_PREPARE_ONLY=1 and PIER_PROOF_ARM is unset, proofArm defaults to 'failure' (line 46). This default flows into executionId = pier-no-model-${proofArm}-${contextDigest}`` (line 410), which feeds into prepareAgentCandidateExecution and produces a plan digest that differs from what a full `PIER_PROOF_ARM=success` run would produce. The `verify-packed-consumer.mjs` caller only validates digest format (`/^sha256:[a-f0-9]{64}$/`) not content, so no current bug. But if anyone extends the packed consumer to

🟡 LOW Complete evaluator env (including secrets) travels over fd 3 pipe — bench/src/pier-trial-controller.ts

controlPipe.end(JSON.stringify({ ... env: spec.env })) serializes the complete evaluator-owned environment — including protected model keys, trace IDs, and other secrets — into JSON over process fd 3. The interface comment correctly notes 'never persisted', but on Linux /proc/<pid>/fd/3 is readable by the same user, so a same-user process snapshot could capture the pipe contents. This is inherent to inter-process secret transport but worth noting as a non-persistent exposure vector.

🟡 LOW Supervisor process inherits full evaluator environment — bench/src/pier-trial-controller.ts

spawn(process.execPath, [this.supervisorPath, controlDirectory], { detached: true, stdio: [...] }) omits an env option, so the supervisor inherits process.env — the full evaluator environment including all secrets. While the supervisor script does not log or exfiltrate env vars, and it is a trusted in-package script, this is not fail-closed. If the supervisor script were ever compromised (e.g., via dependency hijack), it would have direct access to model API keys, trace credentials, and other evaluator secrets.

🟡 LOW parseTerminal uses loose type cast accepting extra fields silently — bench/src/pier-trial-controller.ts

return record as unknown as PersistedTrialTerminal casts after only validating schemaVersion, kind, status, processExited, and containersRemoved. Extra fields like recoveredByPid (written by forceRecovery at line 698) and removedContainers pass through silently. In contrast, parseArchive in pier-workspace-archive.ts validates the exact key set (Object.keys(record).sort().join(',') !== 'files,kind,schemaVersion'). This inconsistency means typos in terminal fields would go undetected at parse time.

---

_{tangletools · 2026-07-11T16:25:31Z · trace}

[tangletools]

✅ Approved — 12 non-blocking findings — 23bf4991

Full multi-shot audit completed 8/8 planned shots over 37 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

---

_{tangletools · 2026-07-11T16:25:31Z · immutable trace}

[tangletools]

✅ Auto-approved drewstone PR — 3cc6622e

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-07-11T16:57:30Z}