@@ -65,7 +65,7 @@ flowchart LR
6565- ** Agent loop and tool calling:** bounded iterations, calls, output, deadlines, cancellation, and recovery.
6666- ** Workspace isolation:** per-run Git worktrees plus constrained file, search, shell, and Git tools.
6767- ** Verification and assessment:** public verification, hidden Oracle evaluation, risk checks, approval, and guarded apply.
68- - ** Evidence and trace:** structured trajectories, execution results, patches, EvidenceBundles , assessment reports, and benchmark reports with sensitive-text redaction.
68+ - ** Evidence and trace:** structured trajectories, execution results, patches, structured evidence artifacts , assessment reports, and benchmark reports with sensitive-text redaction.
6969- ** Evaluation orchestration:** sequential benchmark execution, prediction generation, evaluator bridging, and independent execution/grading validity tracking.
7070
7171## APC Console
@@ -87,21 +87,14 @@ It supports:
8787- ** Runs** browsing and Run Detail with verification facts.
8888- An ** Artifact explorer and preview** for imported runtime and evaluator
8989 outputs.
90- - ** Benchmark Evaluation** with the canonical P2 failure taxonomy and P3
90+ - ** Benchmark Evaluation** with the canonical failure taxonomy and
9191 termination × correctness analysis.
9292
9393The Console is the observation and evaluation layer after controlled execution
9494and independent evaluation; it is not an execution control plane. It never
9595writes back to the workspace, launches a Candidate, executes a Provider or
9696benchmark, or exposes Apply, Retry, or mutation controls.
9797
98- ### Runs
99-
100- The Runs browser presents imported execution records, outcomes, termination,
101- model, runtime, attempts, and mutation facts from the local workspace.
102-
103- ![ APC Console Runs browser] ( docs/assets/apc-console/runs.png )
104-
10598### Run inspection
10699
107100Run Detail exposes the imported verification facts for an individual real run.
@@ -111,36 +104,12 @@ Run Detail exposes the imported verification facts for an individual real run.
111104### Benchmark evaluation
112105
113106The frozen aggregate view makes the official HAL result, execution validity,
114- P2 failure taxonomy, and P3 termination × correctness analysis directly
107+ failure taxonomy, and termination × correctness analysis directly
115108inspectable.
116109
117110![ APC Console benchmark evaluation] ( docs/assets/apc-console/evaluation.png )
118111
119- ### Artifact explorer
120-
121- The artifact explorer previews the imported evidence and evaluator outputs
122- without altering them.
123-
124- ![ APC Console artifact explorer] ( docs/assets/apc-console/artifacts.png )
125-
126- ## 📊 Benchmark
127-
128- ### HAL SWE-bench Verified Mini fixed-50
129-
130- | Metric | Result |
131- | --- | ---: |
132- | Official resolved | ** 35 / 50** |
133- | Resolution rate | ** 70.0%** |
134- | Valid executions | ** 50 / 50** |
135- | Harness-invalid | ** 0** |
136- | Grading-invalid | ** 0** |
137-
138- #### Notes
139-
140- This fixed-50 result is one frozen end-to-end configuration:
141- ` deepseek-v4-flash ` with APC Runtime/tooling and the declared execution
142- policy. It is neither a complete SWE-bench Verified score nor evidence that APC
143- independently causes a given uplift.
112+ ## Benchmark Analysis
144113
145114#### Observed non-resolved categories
146115
@@ -197,12 +166,9 @@ The script invokes `tsx src/agentpatchcheck/cli.ts run`. Read the
197166[ Headless Core guide] ( docs/agentpatchcheck-headless-core.md ) before preparing a
198167TaskSpec or applying a patch.
199168
200- ### Full repository / release build
169+ ### Repository build
201170
202- The independent local APC Console lives in [ ` apc-console/ ` ] ( apc-console/ ) .
203- Existing root web and desktop build paths are separate from APC Console and
204- are not part of the APC Runtime & Evaluation workflow described here.
205- For the repository and release build:
171+ For the repository build:
206172
207173``` bash
208174npm run install:all
@@ -212,16 +178,6 @@ node dist/agentpatchcheck.js --help
212178
213179## How It Works
214180
215- ### Headless repair
216-
217- ``` text
218- TaskSpec → Runtime → Tool Calls → Verification → Evidence → Assessment → Guarded Apply
219- ```
220-
221- 1 . A TaskSpec becomes a policy with repository, provider, verification, and risk boundaries.
222- 2 . APC creates an isolated worktree and executes the selected adapter through its bounded runtime.
223- 3 . Verification, trace, evidence, and patch outputs support assessment and any guarded apply decision.
224-
225181### Benchmark evaluation
226182
227183``` text
@@ -232,8 +188,8 @@ Frozen Manifest → APC Runtime / Tooling → Prediction → Official SWE-bench
2321882 . The official SWE-bench evaluator is used only for this benchmark path and produces the BenchmarkReport.
2331893 . APC Console reads existing runtime artifacts and benchmark outputs locally; it does not participate in execution.
234190
235- See [ Headless Core] ( docs/agentpatchcheck-headless-core.md ) for CLI, evidence,
236- retention, and guarded-apply contracts.
191+ See [ Headless Core] ( docs/agentpatchcheck-headless-core.md ) for the headless
192+ repair lifecycle, CLI, evidence, retention, and guarded-apply contracts.
237193
238194## Implementation Boundaries
239195
@@ -248,47 +204,17 @@ Third-party components and retained upstream attribution are documented in
248204[ THIRD_PARTY_NOTICES.md] ( THIRD_PARTY_NOTICES.md ) . APC does not claim authorship
249205of the official evaluator or complete benchmark datasets.
250206
251- ## Frozen Benchmark Reproduction Contract
252-
253- The repository preserves the portable reproduction contract, not a one-command
254- replica of the published run. Complete datasets, evaluator checkout and Python
255- environment, repositories, Docker images, predictions, logs, and scored
256- artifacts are external to this repository.
207+ ## Reproducibility
257208
258- | Contract item | Frozen value / requirement |
259- | --- | --- |
260- | Dataset source | ` MariusHobbhahn/swe-bench-verified-mini ` , ` test ` split, revision ` b316c349947c29963fce3f4a65967c9807a4b673 ` |
261- | Selected subset | ` HAL-Verified-Mini-v1.full.jsonl ` , 50 rows, SHA-256 in the tracked manifest |
262- | Evaluator | official SWE-bench checkout at ` 7d92bde324b9b96d41fb3e5e1023c8476f17b0bf ` |
263- | Model profile | ` deepseek-v4-flash ` |
264- | Entry point | ` node_modules/.bin/tsx src/agentpatchcheck/swebench-cli.ts ` |
265-
266- Set these existing bootstrap variables in the operator environment; do not put
267- credential values in manifests, source, or evidence:
268-
269- ``` text
270- DEEPSEEK_API_KEY
271- AGENTPATCHCHECK_SWEBENCH_MANIFEST
272- AGENTPATCHCHECK_SWEBENCH_EVALUATOR_ROOT
273- AGENTPATCHCHECK_SWEBENCH_EVALUATOR_PYTHON
274- ```
275-
276- For one manifest-selected instance and a fresh run identity:
277-
278- ``` powershell
279- node_modules/.bin/tsx src/agentpatchcheck/swebench-cli.ts `
280- --instance <instance-id-from-the-manifest> `
281- --run-id <fresh-run-id>
282- ```
209+ The fixed-50 benchmark is backed by a frozen dataset revision, selected
210+ manifest, evaluator revision, model profile, and execution policy.
283211
284- The manifest owns dataset identity, evaluator revision, model, timeout, and
285- classification. Operator arguments cannot override them; this is not an
286- instruction to rerun the published fixed-50 result.
212+ [ View the frozen benchmark reproduction contract] ( docs/benchmark-reproduction.md )
287213
288214## Documentation
289215
290216- [ Headless Core] ( docs/agentpatchcheck-headless-core.md ) — TaskSpec, CLI, evidence, assessment, approval, cleanup, and apply.
291- - [ Historical Formal Runtime Contract ] ( docs/formal-runtime-contract .md ) — frozen lifecycle semantics for the formal pilot .
217+ - [ Benchmark reproduction contract ] ( docs/benchmark-reproduction .md ) — frozen fixed-50 dataset, evaluator, model, environment, and invocation details .
292218- [ Harness-native Benchmark Suite] ( docs/harness-native-benchmark-suite.md ) — deterministic suite boundaries and reports.
293219- [ Third-party notices] ( THIRD_PARTY_NOTICES.md ) — retained attribution and license notices.
294220
0 commit comments