Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 66 additions & 1 deletion bench/HARNESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
If you're an agent picking this up: read this page, then run `pnpm help` + `pnpm gate` —
do NOT re-derive the harness from source. This map is SHORT on purpose; if it disagrees
with the code, the code wins — fix this page in the same turn (the anti-rediscovery law).
Verified against source 2026-07-07 · agent-eval pinned `^0.106.1`. The CANONICAL surface is now
Verified against source 2026-07-11 · agent-eval pinned `^0.114.0`. The CANONICAL surface is now
the published optimization suite (`@tangle-network/agent-runtime/loops`): `Environment` +
`Strategy`/`defineStrategy` + `runBenchmark` — see the section below FIRST. The recursive
diverse-vs-blind gate runs through the keystone (`gate-cli.mts` → `runGate`);
Expand Down Expand Up @@ -134,6 +134,7 @@ the gate + measurement tools:
run-benchmarks-cli.mts runBenchmarks: any subset of the ADAPTERS registry × model/harness cells, one combined ranked report (#420)
commit0-env-run.mts the HARD domain through `runBenchmark` (the optimization suite)
terminal-compare.ts Terminal-Bench compare (own main)
pnpm verify:pier zero-model failure/pass Pier controls through a separate verifier
unit tests (the only fully-green, cred-free runnable surface besides offline replay):
node --test --import tsx src/{selector,refine-loop}.test.mts
tsx src/gate.test.mts # offline plumbing test (no creds)
Expand Down Expand Up @@ -215,6 +216,70 @@ stdin-piping runner (`runVenvScriptStdin`).
- **Absent (not built):** swe-gym, swe-bench-multimodal, and the rest of the survey set.
Every unbuilt/scaffold adapter fails LOUD (throws with the integration step) rather than faking a score — no silent zeros in any corpus. Offline fixture tests: `benchmarks/{aec-bench,commit0,programbench,appworld,rag-benchmarks}.test.mts` (`tsx --test`).

## Pier candidate bridge

`pier_agents.tangle_candidate:TangleCandidateAgent` is the reusable Pier custom-agent path for frozen Tangle candidates.
`executePreparedPierCandidate()` is the only public entry point; its private staging step writes the runtime's execution-plan and materialization-receipt bytes verbatim.
The Python bridge rechecks those bytes and their signed task, candidate, profile, repository, instruction, and workspace identities before launch.
It rejects a raw candidate bundle, never projects an `AgentProfile`, and leaves task isolation, patch transfer, verification, retries, and result storage to Pier.

The adapter fails the trial when any prepared identity drifts, when the task checkout or immutable OCI image differs from the signed identity, or when the candidate exits nonzero or exceeds its signed deadline.
Candidate code runs as an unprivileged numeric user, while evaluator inputs and timeout evidence remain root-owned.
It never accepts candidate-authored token, cost, or trace receipts; `executePreparedPierCandidate()` uses the runtime's atomic execution path and reconciles the protected `TraceStore` with the model-gateway ledger before returning a gradable receipt.
The prepared object contains no credentials.
The runtime passes model and trace bindings only to the trusted executor request.
The Pier launcher inherits their values through its protected process environment and passes only `${NAME}` references on the command line, so credentials never enter prepared bytes, CLI arguments, or job files.
The executor builds fresh task, candidate, and profile trees from the request's exact verified file bytes; prepared staging directories are never launch authority.
Isolated memory and knowledge-bearing candidates currently fail closed until Pier has executor-owned mount and after-state capture.
The signed wall deadline is a hard stop: the runtime aborts, Pier kills the process tree, and the executor acknowledges process and container death.
The signed tool-step count is a post-run validity check over protected traces, not a pre-tool stop; generic black-box Pier processes cannot honestly prevent step N+1.
The executable zero-model fixture is `fixtures/pier-agent/`; run it against the R360 Pier checkout with `PIER_REPO=/path/to/pier pnpm verify:pier`.
That command runs a no-change candidate that must score 0/1, proves a fresh evaluator process can kill a persisted child and remove its real Docker container, and runs a known-good candidate that must score 1/1.
It then checks that each official result and exact task patch is bound into its own runtime receipt with zero model usage.

For a real frozen candidate, use `FilePierCandidateTrialController`, append `agentArgs` and `attemptArgs`, and pass each executor-only `evaluatorEnv` entry through Pier's evaluator-owned environment mechanism.
The launch callback's second argument carries the signed runtime `request`, protected `traceStore`, cancellation `signal`, and absolute `deadlineAtMs`; production launchers must use that context rather than reconstructing it.
The controller sends secrets to its supervisor over a pipe, while its durable files contain only process and Docker-project identities:
The supervisor never inherits `process.env`; non-default Docker connections use a stable `dockerConnection.id` plus the exact environment injected into both Pier and cleanup.
Fresh recovery workers must reconstruct that same named connection; `terminate-pier-trial.mts` selects only the comma-delimited variables named by `PIER_DOCKER_ENV_NAMES` when `PIER_DOCKER_CONNECTION_ID` is set.
`jobName` must be unique per prepared execution; the controller atomically reserves that job directory so recovery can remove only containers owned by that execution.

```ts
const controller = new FilePierCandidateTrialController({
directory: '/var/lib/tangle/pier-control',
launch: (staged, { request }) => {
const jobName = request.executionId
return {
command: 'uv',
args: ['run', 'pier', 'run', ...staged.agentArgs, ...staged.attemptArgs],
cwd: pierCheckout,
env: { ...evaluatorEnvironment, ...staged.evaluatorEnv },
jobsDirectory,
jobName,
readResult: () => readOfficialPierResult(jobsDirectory, jobName),
}
},
})

const result = await executePreparedPierCandidate({
prepared,
directory: '/sealed/candidate',
pierVersion: '0.3.0',
traceStore,
claimStore,
outputArtifacts,
grader,
controller,
})
```

The controller's result resolves only after normal process/container cleanup.
On a signed deadline, external abort, or recovery by another evaluator process, the adapter waits for `terminateAndWait()` to acknowledge both process exit and container removal before it returns control to the runtime.

One prepared execution always maps to one Pier attempt (`--n-attempts 1 --max-retries 0`).
Production callers pass a long-lived `FileAgentCandidateExecutionClaimStore`; an in-memory claim store is test-only and cannot prevent a second process from replaying the same attempt.
Any allowed pre-model infrastructure retry is a new prepared execution with its own counted attempt identity.

## Is it runnable RIGHT NOW? (verify the map, don't trust it blindly)
```
ls src/*.mts src/*.ts # the real tool list (each its own main — source of truth)
Expand Down
11 changes: 10 additions & 1 deletion bench/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# agent-runtime-bench

Private experiment workspace nested in agent-runtime; decoupled from its build/lint/release (the package builds `src/`, lints `src tests examples` — `bench/` is none of those).
Published as `@tangle-network/agent-bench`, with independent CI and release checks for its TypeScript and Python surfaces.

**Read [`bench/HARNESS.md`](./HARNESS.md) FIRST.** It is the one maintained map: the commands, the `rollout → corpus → selector → CI → gate` data flow, the canonical-suite table, the wired/needs-creds/scaffolded matrix, and the gate one-liners — kept verified against source.

Expand All @@ -13,3 +13,12 @@ pnpm install # tsx + link parent
```

The judge needs only Docker; workers need a model key (Tangle router `TANGLE_API_KEY`, or a direct provider).

## Pier custom candidates

The package executes a branded `PreparedAgentCandidateExecution` from `@tangle-network/agent-runtime` through one atomic API and ships `pier_agents.tangle_candidate:TangleCandidateAgent` as its thin Pier transport.
The executor recreates every input from runtime-verified file bytes and reveals model credentials only inside the claimed execution callback.
Pier owns the task container and verifier; protected model usage and traces stay in `@tangle-network/agent-eval` and are finalized by the shared runtime.
`FilePierCandidateTrialController` atomically reserves a unique Pier job, then persists the supervisor PID, process-session identity, and that job's exact Docker projects so a fresh evaluator process can stop and remove an abandoned trial.
Run `PIER_REPO=/path/to/pier pnpm verify:pier` for the zero-model failure/pass and fresh-process recovery proof, and see `HARNESS.md` for the exact invocation and failure contract.
From an installed npm package, expose the shipped Python module with `export PYTHONPATH="$(npm root)/@tangle-network/agent-bench${PYTHONPATH:+:$PYTHONPATH}"` before invoking Pier.
16 changes: 16 additions & 0 deletions bench/fixtures/pier-agent/no-model-task/environment/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
FROM ubuntu:24.04

RUN apt-get update \
&& apt-get install -y --no-install-recommends git ca-certificates \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY seed/ /app/
RUN git init \
&& git config user.email fixture@tangle.tools \
&& git config user.name "Tangle Fixture" \
&& git add -A \
&& GIT_AUTHOR_DATE=2000-01-01T00:00:00Z \
GIT_COMMITTER_DATE=2000-01-01T00:00:00Z \
git commit -m baseline \
&& git tag benchmark-base
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
not-ready
6 changes: 6 additions & 0 deletions bench/fixtures/pier-agent/no-model-task/instruction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Change src/status.txt so it contains exactly:

ready
owner=tangle

Commit the change to git when finished.
6 changes: 6 additions & 0 deletions bench/fixtures/pier-agent/no-model-task/pre_artifacts.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -euo pipefail

mkdir -p /logs/artifacts
cd /app
git -c safe.directory=/app diff --binary benchmark-base..HEAD > /logs/artifacts/model.patch
35 changes: 35 additions & 0 deletions bench/fixtures/pier-agent/no-model-task/task.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
version = "1.0"

[task]
name = "agent-bench/pier-candidate-no-model"
authors = []
keywords = ["pier", "candidate", "no-model"]

[metadata]
author_name = "Tangle Research"
author_email = "research@tangle.tools"
difficulty = "synthetic"
category = "software-engineering"
tags = ["no-model", "separate-verifier"]

[agent]
timeout_sec = 60.0

[verifier]
timeout_sec = 60.0
environment_mode = "separate"

[verifier.environment]

[environment]
build_timeout_sec = 300.0
cpus = 1
memory_mb = 1024
storage_mb = 2048
gpus = 0
allow_internet = false
mcp_servers = []

[verifier.env]

[solution.env]
17 changes: 17 additions & 0 deletions bench/fixtures/pier-agent/no-model-task/tests/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
FROM ubuntu:24.04

RUN apt-get update \
&& apt-get install -y --no-install-recommends git ca-certificates \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY seed/ /app/
RUN git init \
&& git config user.email fixture@tangle.tools \
&& git config user.name "Tangle Fixture" \
&& git add -A \
&& GIT_AUTHOR_DATE=2000-01-01T00:00:00Z \
GIT_COMMITTER_DATE=2000-01-01T00:00:00Z \
git commit -m baseline \
&& git tag benchmark-base
COPY test.sh /tests/test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
not-ready
19 changes: 19 additions & 0 deletions bench/fixtures/pier-agent/no-model-task/tests/test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/usr/bin/env bash
set -euo pipefail

reward=0
patch_applied=0
patch=/logs/artifacts/model.patch

if [ -s "$patch" ] \
&& git -c safe.directory=/app apply --check "$patch" \
&& git -c safe.directory=/app apply "$patch"; then
patch_applied=1
fi
if [ "$(cat /app/src/status.txt 2>/dev/null || true)" = $'ready\nowner=tangle' ] \
&& [ ! -e /app/AGENTS.md ]; then
reward=1
fi

printf '{"reward":%d,"patch_applied":%d}\n' "$reward" "$patch_applied" \
> /logs/verifier/reward.json
25 changes: 19 additions & 6 deletions bench/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@tangle-network/agent-bench",
"version": "0.1.1",
"version": "0.2.0",
"type": "module",
"description": "The unified benchmark suite for agent-runtime agents: 31 adapters (commit0, enterpriseops-gym, ragbench, crag, nomiracl, open-rag-bench, t2-ragbench, tau3-banking, bfcl, finresearchbench, …) behind one resolveAdapter registry, each with a real judge or fail-loud unsupported scorer. Score any profile/skill/prompt change against them. Map: bench/HARNESS.md.",
"main": "src/index.ts",
Expand All @@ -18,25 +18,38 @@
"terminal-compare": "tsx src/terminal-compare.ts",
"test": "node scripts/run-package-tests.mjs",
"typecheck:public": "tsc -p tsconfig.public.json",
"verify:package": "node scripts/verify-packed-consumer.mjs"
"verify:package": "node scripts/verify-packed-consumer.mjs",
"verify:pier": "tsx scripts/verify-pier-pair.mts"
},
"dependencies": {
"@tangle-network/agent-eval": "^0.108.1",
"@tangle-network/agent-runtime": "^0.90.1",
"@tangle-network/sandbox": "^0.9.7"
"@tangle-network/agent-eval": "^0.114.0",
"@tangle-network/agent-interface": "^0.25.0",
"@tangle-network/agent-runtime": "^0.92.0",
"@tangle-network/sandbox": "^0.10.3"
},
"devDependencies": {
"@types/node": "^25.0.0",
"tsx": "^4.19.0",
"typescript": "^6.0.3"
},
"pnpm": {
"overrides": {
"@tangle-network/agent-eval": "0.114.0"
}
},
"files": [
"src",
"fixtures",
"scripts",
"pier_agents/__init__.py",
"pier_agents/candidate_contract.py",
"pier_agents/process_boundary.py",
"pier_agents/tangle_candidate.py",
"pier_agents/workspace_boundary.py",
"tb_agents/*.py",
"steerers",
"README.md"
"README.md",
"HARNESS.md"
],
"publishConfig": {
"access": "public"
Expand Down
18 changes: 18 additions & 0 deletions bench/pier_agents/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
"""Pier custom agents shipped by ``@tangle-network/agent-bench``."""

from typing import TYPE_CHECKING, Any


if TYPE_CHECKING:
from .tangle_candidate import TangleCandidateAgent

__all__ = ["TangleCandidateAgent"]


def __getattr__(name: str) -> Any:
"""Load Pier only when callers request the custom agent."""
if name != "TangleCandidateAgent":
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
from .tangle_candidate import TangleCandidateAgent

return TangleCandidateAgent
Loading
Loading