Skip to content

Commit c741dd2

Browse files
committed
docs: streamline project README
1 parent 7eb5cb6 commit c741dd2

2 files changed

Lines changed: 55 additions & 87 deletions

File tree

‎README.md‎

Lines changed: 13 additions & 87 deletions
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ flowchart LR
6565
- **Agent loop and tool calling:** bounded iterations, calls, output, deadlines, cancellation, and recovery.
6666
- **Workspace isolation:** per-run Git worktrees plus constrained file, search, shell, and Git tools.
6767
- **Verification and assessment:** public verification, hidden Oracle evaluation, risk checks, approval, and guarded apply.
68-
- **Evidence and trace:** structured trajectories, execution results, patches, EvidenceBundles, assessment reports, and benchmark reports with sensitive-text redaction.
68+
- **Evidence and trace:** structured trajectories, execution results, patches, structured evidence artifacts, assessment reports, and benchmark reports with sensitive-text redaction.
6969
- **Evaluation orchestration:** sequential benchmark execution, prediction generation, evaluator bridging, and independent execution/grading validity tracking.
7070

7171
## APC Console
@@ -87,21 +87,14 @@ It supports:
8787
- **Runs** browsing and Run Detail with verification facts.
8888
- An **Artifact explorer and preview** for imported runtime and evaluator
8989
outputs.
90-
- **Benchmark Evaluation** with the canonical P2 failure taxonomy and P3
90+
- **Benchmark Evaluation** with the canonical failure taxonomy and
9191
termination × correctness analysis.
9292

9393
The Console is the observation and evaluation layer after controlled execution
9494
and independent evaluation; it is not an execution control plane. It never
9595
writes back to the workspace, launches a Candidate, executes a Provider or
9696
benchmark, or exposes Apply, Retry, or mutation controls.
9797

98-
### Runs
99-
100-
The Runs browser presents imported execution records, outcomes, termination,
101-
model, runtime, attempts, and mutation facts from the local workspace.
102-
103-
![APC Console Runs browser](docs/assets/apc-console/runs.png)
104-
10598
### Run inspection
10699

107100
Run Detail exposes the imported verification facts for an individual real run.
@@ -111,36 +104,12 @@ Run Detail exposes the imported verification facts for an individual real run.
111104
### Benchmark evaluation
112105

113106
The frozen aggregate view makes the official HAL result, execution validity,
114-
P2 failure taxonomy, and P3 termination × correctness analysis directly
107+
failure taxonomy, and termination × correctness analysis directly
115108
inspectable.
116109

117110
![APC Console benchmark evaluation](docs/assets/apc-console/evaluation.png)
118111

119-
### Artifact explorer
120-
121-
The artifact explorer previews the imported evidence and evaluator outputs
122-
without altering them.
123-
124-
![APC Console artifact explorer](docs/assets/apc-console/artifacts.png)
125-
126-
## 📊 Benchmark
127-
128-
### HAL SWE-bench Verified Mini fixed-50
129-
130-
| Metric | Result |
131-
| --- | ---: |
132-
| Official resolved | **35 / 50** |
133-
| Resolution rate | **70.0%** |
134-
| Valid executions | **50 / 50** |
135-
| Harness-invalid | **0** |
136-
| Grading-invalid | **0** |
137-
138-
#### Notes
139-
140-
This fixed-50 result is one frozen end-to-end configuration:
141-
`deepseek-v4-flash` with APC Runtime/tooling and the declared execution
142-
policy. It is neither a complete SWE-bench Verified score nor evidence that APC
143-
independently causes a given uplift.
112+
## Benchmark Analysis
144113

145114
#### Observed non-resolved categories
146115

@@ -197,12 +166,9 @@ The script invokes `tsx src/agentpatchcheck/cli.ts run`. Read the
197166
[Headless Core guide](docs/agentpatchcheck-headless-core.md) before preparing a
198167
TaskSpec or applying a patch.
199168

200-
### Full repository / release build
169+
### Repository build
201170

202-
The independent local APC Console lives in [`apc-console/`](apc-console/).
203-
Existing root web and desktop build paths are separate from APC Console and
204-
are not part of the APC Runtime & Evaluation workflow described here.
205-
For the repository and release build:
171+
For the repository build:
206172

207173
```bash
208174
npm run install:all
@@ -212,16 +178,6 @@ node dist/agentpatchcheck.js --help
212178

213179
## How It Works
214180

215-
### Headless repair
216-
217-
```text
218-
TaskSpec → Runtime → Tool Calls → Verification → Evidence → Assessment → Guarded Apply
219-
```
220-
221-
1. A TaskSpec becomes a policy with repository, provider, verification, and risk boundaries.
222-
2. APC creates an isolated worktree and executes the selected adapter through its bounded runtime.
223-
3. Verification, trace, evidence, and patch outputs support assessment and any guarded apply decision.
224-
225181
### Benchmark evaluation
226182

227183
```text
@@ -232,8 +188,8 @@ Frozen Manifest → APC Runtime / Tooling → Prediction → Official SWE-bench
232188
2. The official SWE-bench evaluator is used only for this benchmark path and produces the BenchmarkReport.
233189
3. APC Console reads existing runtime artifacts and benchmark outputs locally; it does not participate in execution.
234190

235-
See [Headless Core](docs/agentpatchcheck-headless-core.md) for CLI, evidence,
236-
retention, and guarded-apply contracts.
191+
See [Headless Core](docs/agentpatchcheck-headless-core.md) for the headless
192+
repair lifecycle, CLI, evidence, retention, and guarded-apply contracts.
237193

238194
## Implementation Boundaries
239195

@@ -248,47 +204,17 @@ Third-party components and retained upstream attribution are documented in
248204
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md). APC does not claim authorship
249205
of the official evaluator or complete benchmark datasets.
250206

251-
## Frozen Benchmark Reproduction Contract
252-
253-
The repository preserves the portable reproduction contract, not a one-command
254-
replica of the published run. Complete datasets, evaluator checkout and Python
255-
environment, repositories, Docker images, predictions, logs, and scored
256-
artifacts are external to this repository.
207+
## Reproducibility
257208

258-
| Contract item | Frozen value / requirement |
259-
| --- | --- |
260-
| Dataset source | `MariusHobbhahn/swe-bench-verified-mini`, `test` split, revision `b316c349947c29963fce3f4a65967c9807a4b673` |
261-
| Selected subset | `HAL-Verified-Mini-v1.full.jsonl`, 50 rows, SHA-256 in the tracked manifest |
262-
| Evaluator | official SWE-bench checkout at `7d92bde324b9b96d41fb3e5e1023c8476f17b0bf` |
263-
| Model profile | `deepseek-v4-flash` |
264-
| Entry point | `node_modules/.bin/tsx src/agentpatchcheck/swebench-cli.ts` |
265-
266-
Set these existing bootstrap variables in the operator environment; do not put
267-
credential values in manifests, source, or evidence:
268-
269-
```text
270-
DEEPSEEK_API_KEY
271-
AGENTPATCHCHECK_SWEBENCH_MANIFEST
272-
AGENTPATCHCHECK_SWEBENCH_EVALUATOR_ROOT
273-
AGENTPATCHCHECK_SWEBENCH_EVALUATOR_PYTHON
274-
```
275-
276-
For one manifest-selected instance and a fresh run identity:
277-
278-
```powershell
279-
node_modules/.bin/tsx src/agentpatchcheck/swebench-cli.ts `
280-
--instance <instance-id-from-the-manifest> `
281-
--run-id <fresh-run-id>
282-
```
209+
The fixed-50 benchmark is backed by a frozen dataset revision, selected
210+
manifest, evaluator revision, model profile, and execution policy.
283211

284-
The manifest owns dataset identity, evaluator revision, model, timeout, and
285-
classification. Operator arguments cannot override them; this is not an
286-
instruction to rerun the published fixed-50 result.
212+
[View the frozen benchmark reproduction contract](docs/benchmark-reproduction.md)
287213

288214
## Documentation
289215

290216
- [Headless Core](docs/agentpatchcheck-headless-core.md) — TaskSpec, CLI, evidence, assessment, approval, cleanup, and apply.
291-
- [Historical Formal Runtime Contract](docs/formal-runtime-contract.md) — frozen lifecycle semantics for the formal pilot.
217+
- [Benchmark reproduction contract](docs/benchmark-reproduction.md) — frozen fixed-50 dataset, evaluator, model, environment, and invocation details.
292218
- [Harness-native Benchmark Suite](docs/harness-native-benchmark-suite.md) — deterministic suite boundaries and reports.
293219
- [Third-party notices](THIRD_PARTY_NOTICES.md) — retained attribution and license notices.
294220

‎docs/benchmark-reproduction.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Fixed-50 Benchmark Reproduction Contract
2+
3+
The repository preserves the portable reproduction contract, not a one-command
4+
replica of the published run. Complete datasets, evaluator checkout and Python
5+
environment, repositories, Docker images, predictions, logs, and scored
6+
artifacts are external to this repository.
7+
8+
## Frozen contract
9+
10+
| Contract item | Frozen value / requirement |
11+
| --- | --- |
12+
| Dataset source | `MariusHobbhahn/swe-bench-verified-mini`, `test` split, revision `b316c349947c29963fce3f4a65967c9807a4b673` |
13+
| Selected subset | `HAL-Verified-Mini-v1.full.jsonl`, 50 rows, SHA-256 in the tracked manifest |
14+
| Evaluator | official SWE-bench checkout at `7d92bde324b9b96d41fb3e5e1023c8476f17b0bf` |
15+
| Model profile | `deepseek-v4-flash` |
16+
| Entry point | `node_modules/.bin/tsx src/agentpatchcheck/swebench-cli.ts` |
17+
18+
## Bootstrap environment
19+
20+
Set these existing bootstrap variables in the operator environment; do not put
21+
credential values in manifests, source, or evidence:
22+
23+
```text
24+
DEEPSEEK_API_KEY
25+
AGENTPATCHCHECK_SWEBENCH_MANIFEST
26+
AGENTPATCHCHECK_SWEBENCH_EVALUATOR_ROOT
27+
AGENTPATCHCHECK_SWEBENCH_EVALUATOR_PYTHON
28+
```
29+
30+
## Example invocation
31+
32+
For one manifest-selected instance and a fresh run identity:
33+
34+
```powershell
35+
node_modules/.bin/tsx src/agentpatchcheck/swebench-cli.ts `
36+
--instance <instance-id-from-the-manifest> `
37+
--run-id <fresh-run-id>
38+
```
39+
40+
The manifest owns dataset identity, evaluator revision, model, timeout, and
41+
classification. Operator arguments cannot override them; this is not an
42+
instruction to rerun the published fixed-50 result.

0 commit comments

Comments
 (0)