Repository navigation
docs: ground NakedAgent paper claims in the Python reference #45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # NakedAgent paper | ||
|
|
||
| The **unreleased living manuscript** is [nakedagent_paper.tex](nakedagent_paper.tex), revised October 10, 2026, with [references.bib](references.bib). Its title is *NakedAgent: Standard-Library Coding Agents with Explicit Effect Boundaries and Offline Replay*. It describes the Python reference at commit `0c96f1a8adac035b9d204ebc78c5eb4175c13685`, not every port or deployment. | ||
|
|
||
| The repository-recorded historical DOI is [10.5281/zenodo.23001340](https://doi.org/10.5281/zenodo.23001340). This revision does not create a DOI or claim the remote deposit was independently checked. | ||
|
|
||
| ## Artifact status | ||
|
|
||
| | File | Status | | ||
| |---|---| | ||
| | `nakedagent_paper.tex`, `references.bib` | Current review source; changes include substantive corrections to earlier claims | | ||
| | `check_claims.py` | New offline source inventory and eight synthetic checks; no model or real tool execution | | ||
| | [nakedagent.notebook.md](nakedagent.notebook.md) | Post-hoc audit record and prospective evaluation plan | | ||
| | `nakedagent_paper.pdf`, `nakedagent_paper.html`, `nakedagent_paper.bbl` | Preserved historical renderings; **do not describe the revised manuscript** | | ||
| | `NakedAgent_arxiv_bundle.tar.gz` | Preserved historical source bundle; not regenerated | | ||
| | `.zenodo.json`, `TechRxiv_SSRN_Submission_Dossier.md` | Historical deposit/submission material; not approved metadata for this revision | | ||
| | `ARCANA_PEER_REVIEW_SYNTHESIS.md` | Historical review record; its verdict does not apply to the new source | | ||
|
|
||
| Historical metadata and renderings contain claims narrowed in the living manuscript, including non-repudiation, supply-chain elimination and execution proof. Do not use them to submit the new version. A future release needs consistent new renderings, reviewed metadata and a complete source/check archive. The original artifacts remain unchanged so the correction is traceable. | ||
|
|
||
| ## Source-to-claim map | ||
|
|
||
| All paths below refer to the pinned commit. The claim checker rejects runtime source that differs from that baseline, after normalizing Git/worktree line endings. | ||
|
|
||
| | Claim | Source | What it does not establish | | ||
| |---|---|---| | ||
| | No declared third-party Python runtime packages | `pyproject.toml`; static imports in `nakedagent/*.py` | No build requirements, no interpreter/provider risk, or dependency-free plugins | | ||
| | Ordered, nesting-aware fenced-call parsing | `nakedagent/toolcall.py`; `spec/SPEC.md` | Superior model compliance or injection resistance | | ||
| | File and shell tool behavior | `nakedagent/tools.py` | Atomic writes, universal path isolation, or safe semantics for every allowed program | | ||
| | Registry substitution updates prompt examples | `nakedagent/plugins.py`, `nakedagent/loop.py` | Plugin sandboxing or correctness of `.usage` text | | ||
| | Optional functional driver | `nakedagent/cli.py`, `nakedagent/driver.py` | All CLI paths use the reducer or install Sidekick policy | | ||
| | Hash-linked events and replay | `nakedagent/functional.py`, `eventlog.py`, `replay.py` | Authenticated origin, full effect causality, genuine model identity or actual tool effects | | ||
| | Tri-state extension point | `nakedagent/sidekick.py` | A calibrated risk estimate or cryptographic approval protocol | | ||
| | Historical pilot observations | `bench/RESULTS.md`, `bench/run_nakedagent_swebench.py`, `bench/score_preds.py` | Reproduced raw trajectories, controlled comparison or general task-success rate | | ||
|
|
||
| The current foundation conformance specification excludes replay, the functional driver, Sidekick and MCP. Foundation port conformance therefore cannot be cited as evidence for those features. | ||
|
|
||
| ## Offline checks | ||
|
|
||
| From the repository root, on Python 3.10+: | ||
|
|
||
| ```text | ||
| python paper/check_claims.py | ||
| python -m examples.replay_demo | ||
| python -m unittest discover -s tests | ||
| git diff --check -- paper | ||
| ``` | ||
|
|
||
| The claim checker writes JSON to stdout and creates only temporary synthetic logs. It imports pinned runtime functions directly without loading plugins. Expected cases include stale-hash edit rejection, coherent rewritten-history acceptance, acceptance of a reported tool result without a prior request, missing-final and post-terminal rejection, and event-budget dispatch boundaries. These are small falsifiable examples of the paper's scope, not exhaustive verification or a benchmark rerun. | ||
|
|
||
| ## Build the living paper | ||
|
|
||
| Use an already installed TeX distribution with `pdflatex`, `bibtex` and the packages listed in the preamble. Run **all build commands from a separate build directory**, giving the TeX source's absolute path and setting the bibliography search path. Compiling from `paper/` can accidentally load the archived BBL instead of the new bibliography. | ||
|
|
||
| PowerShell example; substitute the actual paths and create an empty build directory first: | ||
|
|
||
| ```powershell | ||
| $env:BIBINPUTS = 'C:/absolute/path/to/nakedagent/paper;' | ||
| Push-Location 'C:/absolute/path/to/build' | ||
| pdflatex -no-shell-escape -interaction=nonstopmode -halt-on-error C:/absolute/path/to/nakedagent/paper/nakedagent_paper.tex | ||
| bibtex nakedagent_paper | ||
| pdflatex -no-shell-escape -interaction=nonstopmode -halt-on-error C:/absolute/path/to/nakedagent/paper/nakedagent_paper.tex | ||
| pdflatex -no-shell-escape -interaction=nonstopmode -halt-on-error C:/absolute/path/to/nakedagent/paper/nakedagent_paper.tex | ||
| Pop-Location | ||
| ``` | ||
|
|
||
| On MiKTeX, `--disable-installer` can additionally prevent automatic package installation. Do not overwrite the historical PDF or BBL. Review the generated `nakedagent_paper.pdf` and log in the build directory; a successful compile does not validate scientific claims. The revision notebook records the observed build/test results and unresolved data limitations. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| """Offline paper evidence: source inventory and replay-boundary counterexamples. | ||
|
|
||
| Run from the repository root: python paper/check_claims.py | ||
| Uses synthetic events, a stub model and stub tool; never loads plugins, calls a | ||
| provider, executes a shell tool, or changes runtime source. Temporary logs are | ||
| deleted on exit. Passing is evidence about these cases, not a security proof. | ||
| """ | ||
| from __future__ import annotations | ||
|
|
||
| import ast | ||
| import hashlib | ||
| import json | ||
| import subprocess | ||
| import sys | ||
| from pathlib import Path | ||
| from tempfile import TemporaryDirectory | ||
|
|
||
| ROOT = Path(__file__).resolve().parents[1] | ||
| sys.path.insert(0, str(ROOT)) | ||
|
|
||
| from nakedagent.driver import run_functional | ||
| from nakedagent.eventlog import open_log | ||
| from nakedagent.functional import AgentEvent, AgentState, agent_reducer | ||
| from nakedagent.replay import verify | ||
|
|
||
| BASELINE = "0c96f1a8adac035b9d204ebc78c5eb4175c13685" | ||
|
|
||
|
|
||
| def make_log(path: Path, events: list[AgentEvent]) -> str: | ||
| system = "Synthetic paper claim check; no external effects." | ||
| policy = {"model": "stub", "workspace": "synthetic", "extra": {}} | ||
| state = AgentState.initial(system, policy=policy) | ||
| writer = open_log(path, system_prompt=system, model="stub", workspace="synthetic", extra={}) | ||
| try: | ||
| for event in events: | ||
| state, _ = agent_reducer(state, event) | ||
| writer.write_event(event, state.current_hash()) | ||
| finally: | ||
| writer.close(state.step_count, state.current_hash(), state.is_terminal, | ||
| state.terminal_reason, state.suspended) | ||
| return state.current_hash() | ||
|
|
||
|
|
||
| def main() -> int: | ||
| source_paths = sorted((ROOT / "nakedagent").glob("*.py")) | ||
| source_paths += [ROOT / "pyproject.toml"] | ||
| source_matches = {} | ||
| imports: set[str] = set() | ||
| for path in source_paths: | ||
| relative = path.relative_to(ROOT).as_posix() | ||
| data = path.read_bytes() | ||
| archived = subprocess.check_output( | ||
| ["git", "show", f"{BASELINE}:{relative}"], cwd=ROOT) | ||
| # Git stores LF while Windows worktrees may use CRLF. | ||
| normalized = data.replace(b"\r\n", b"\n") | ||
| source_matches[relative] = { | ||
| "sha256_lf": hashlib.sha256(normalized).hexdigest(), | ||
| "matches_baseline": normalized == archived.replace(b"\r\n", b"\n"), | ||
| } | ||
| if path.suffix == ".py": | ||
| for node in ast.walk(ast.parse(data)): | ||
| if isinstance(node, ast.Import): | ||
| imports.update(n.name.split(".")[0] for n in node.names) | ||
| elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: | ||
| imports.add(node.module.split(".")[0]) | ||
| external_imports = sorted(imports - sys.stdlib_module_names - {"nakedagent"}) | ||
| assert not external_imports, external_imports | ||
| assert all(item["matches_baseline"] for item in source_matches.values()) | ||
|
|
||
| events = [AgentEvent("USER_INPUT", "Hello"), | ||
| AgentEvent("MODEL_REPLY", "Hello back"), | ||
| AgentEvent("TERMINATION", "DONE")] | ||
| checks = {} | ||
| with TemporaryDirectory(prefix="nakedagent-paper-") as temp: | ||
| work = Path(temp) | ||
| original = work / "original.jsonl" | ||
| first_hash = make_log(original, events) | ||
| checks["original_accepted"] = verify(original)[0] | ||
| duplicate = work / "duplicate.jsonl" | ||
| checks["same_inputs_same_hash"] = make_log(duplicate, events) == first_hash | ||
|
|
||
| records = [json.loads(x) for x in original.read_text(encoding="utf-8").splitlines()] | ||
| records[2]["payload"] = "Different reply" | ||
| stale = work / "stale.jsonl" | ||
| stale.write_text("".join(json.dumps(x) + "\n" for x in records), encoding="utf-8") | ||
| checks["stale_hash_edit_rejected"] = not verify(stale)[0] | ||
|
|
||
| rewritten = work / "rewritten.jsonl" | ||
| new_hash = make_log(rewritten, [events[0], AgentEvent("MODEL_REPLY", "Different reply"), events[2]]) | ||
| checks["coherent_rewrite_accepted"] = verify(rewritten)[0] and new_hash != first_hash | ||
|
|
||
| orphan = work / "orphan.jsonl" | ||
| make_log(orphan, [events[0], AgentEvent("TOOL_RESULT", "Claimed success", {"tool": "write"}), events[2]]) | ||
| checks["unrequested_tool_result_accepted"] = verify(orphan)[0] | ||
|
|
||
| truncated = work / "truncated.jsonl" | ||
| truncated.write_text("\n".join(original.read_text(encoding="utf-8").splitlines()[:-1]) + "\n", encoding="utf-8") | ||
| checks["missing_final_rejected"] = not verify(truncated)[0] | ||
|
|
||
| post_terminal = work / "post-terminal.jsonl" | ||
| make_log(post_terminal, events + [AgentEvent("USER_INPUT", "After termination")]) | ||
| checks["post_terminal_event_rejected"] = not verify(post_terminal)[0] | ||
|
|
||
| dispatch_counts = {} | ||
| for budget in (2, 3): | ||
| calls: list[str] = [] | ||
| def stub_tool(args: str, content: str, workspace: Path, *, _calls=calls) -> str: | ||
| _calls.append(content) | ||
| return "synthetic result" | ||
| def stub_model(*args, **kwargs) -> str: | ||
| return "```probe\nsynthetic request\n```" | ||
| state = run_functional("Hello", "stub", work, | ||
| tools={"probe": stub_tool}, llm_fn=stub_model, | ||
| max_steps=budget) | ||
| dispatch_counts[str(budget)] = len(calls) | ||
| assert state.step_count == budget and state.is_terminal | ||
| checks["budget_2_dispatches_0_budget_3_dispatches_1"] = dispatch_counts == {"2": 0, "3": 1} | ||
|
|
||
| assert all(checks.values()), checks | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win Make failed checks produce a nonzero exit under Python removes 🤖 Prompt for AI Agents |
||
| report = { | ||
| "baseline": BASELINE, | ||
| "python": sys.version.split()[0], | ||
| "platform": sys.platform, | ||
| "scope": "synthetic checks; no provider, plugin, shell tool, or benchmark run", | ||
| "external_static_imports": external_imports, | ||
| "source": source_matches, | ||
| "checks": checks, | ||
| "interpretation": "Expected coherent-rewrite and orphan-result acceptance delimit the verifier; they are not authenticated execution evidence.", | ||
| } | ||
| print(json.dumps(report, indent=2)) | ||
| return 0 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| raise SystemExit(main()) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| # NakedAgent research notebook | ||
|
|
||
| ## Provenance | ||
|
|
||
| This is a **post-hoc source audit**, begun October 10, 2026. It is not the notebook of the September 14 pilot and not a preregistration. The baseline is `0c96f1a8adac035b9d204ebc78c5eb4175c13685`. Runtime code and archived paper artifacts are preserved. | ||
|
|
||
| Question: which architectural and replay claims are supported by the current source, and what can be reproduced without a model service or unavailable pilot logs? | ||
|
|
||
| ## Findings and revision decisions | ||
|
|
||
| 1. Retained the constructive standard-library architecture and Omakase substitution contribution. Scoped zero dependencies to the foundation's third-party Python runtime packages; the build backend, interpreter, providers, commands and plugins are separate dependencies. | ||
| 2. Distinguished the default imperative loop, optional event-log driver and Sidekick harness. Corrected event-budget semantics, terminal budget exhaustion, default gate heuristics and the absence of automatic Sidekick gating in the CLI functional path. | ||
| 3. Replaced a fictitious state tuple and hash formula with the actual v0.4 fields. The state is not finite in the strict automata-theoretic sense. The linear chain is not a Merkle tree with inclusion proofs; the reducer's receipt can record proposed actions suppressed at terminal exhaustion. | ||
| 4. Corrected replay's claim to internal consistency. Rehashing a complete modified transcript is possible; a reported TOOL_RESULT needs no matched pending request. Neither authenticated origin nor tool-effect truth follows from a PASS result. | ||
| 5. Corrected atomic-write, numbered-read, single-regex parser and universal read-only claims. Disabling shell/write leaves patch active. Even disabling all three mutating built-ins does not constrain arbitrary plugin imports. | ||
| 6. Preserved the pilot's useful debugging sequence while identifying C1/C2 as invalid harness configurations. The narrative reports C3 zero of three resolved and seven regressions on one patch; no raw trajectories or test output are tracked at the baseline. The manuscript does not claim those results were reproduced or establish a local-model capability ceiling. | ||
| 7. Preserved historical PDF, HTML, BBL, bundle, submission dossier and Zenodo metadata. Added an explicit distinction from the living TeX. No submission, deposit update or DOI minting occurred. | ||
|
|
||
| ## Primary-source checks | ||
|
|
||
| Opened on October 10, 2026: | ||
|
|
||
| - [SWE-bench paper](https://arxiv.org/abs/2310.06770): task/evaluation context and bibliographic metadata. | ||
| - [SWE-agent paper](https://arxiv.org/abs/2405.15793): interface-design motivation; no numerical comparison with NakedAgent inferred. | ||
| - [gptme project](https://github.com/gptme/gptme): provenance for the acknowledged terminal-tool architecture; exact inheritance is stated by local source comments, not inferred from current upstream behavior. | ||
| - [Aider edit-format documentation](https://aider.chat/docs/more/edit-formats.html): SEARCH/REPLACE precedent. | ||
| - [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162): distinction between local hash chaining and a transparency protocol with signed checkpoints/consistency mechanisms. No conformance claimed. | ||
| - [MCP 2024-11-05 stdio transport](https://modelcontextprotocol.io/specification/2024-11-05/basic/transports): version matches the local client constant. No remote MCP was connected. | ||
|
|
||
| The remote historical Zenodo deposit was not independently verified. The DOI is attributed to repository records. | ||
|
|
||
| ## Executable evidence | ||
|
|
||
| `paper/check_claims.py` uses only synthetic events, a stub model and a stub tool. It checks LF-normalized runtime source hashes against the baseline and eight named cases. Complete command output and build products are kept outside the repository in the operator's run directory; observed results are recorded below after execution. | ||
|
|
||
| The checks are post-hoc examples selected from this audit. They are not a test sample for an estimated security failure rate, and they do not demonstrate externally executed effects. | ||
|
|
||
| ## Prospective study | ||
|
|
||
| Before a new empirical run, freeze source/plugin versions, dataset task selection, model digest/quantization, provider settings, prompt variants, container images and scoring version. Define matched sampling and repeated-run counts before inspecting outcomes. Predetermine invalid-infrastructure classifications and rerun rules. Archive exact patch bytes, predictions, trajectories and full grader outputs. | ||
|
|
||
| Keep parse compliance, successful dispatch, patch applicability, resolved tests, regressions and final task success separate. Use one-factor comparisons for fence/native calls, prompt examples and shell/file tools. This protocol is proposed and has not been executed here. | ||
|
|
||
| ## Observed checks, October 10, 2026 | ||
|
|
||
| Environment: Windows, Python 3.14.6. No model benchmark or provider request was made. | ||
|
|
||
| | Check | Observed result | Limits | | ||
| |---|---|---| | ||
| | `python paper/check_claims.py` | Exit 0; eight expected outcomes; runtime/metadata files match the pinned baseline; no external static imports | Synthetic examples and static import inventory, not complete execution verification | | ||
| | `python -m examples.replay_demo` | Exit 0; original three-event trace accepted; changed payload rejected | Partial-edit detection with hashes held fixed | | ||
| | `python -m unittest discover -s tests` | 498 tests run, OK, 45 skipped, 31.746 s | Windows/environment skips remain; suite emits existing mock HTTP resource warnings | | ||
| | `ruff check paper/check_claims.py` | Passed | Companion script only; runtime unchanged | | ||
| | `git diff --check -- paper` | Passed | Whitespace validation | | ||
| | PDFLaTeX + BibTeX + two PDFLaTeX passes | Five-page review PDF; no undefined citations or overfull boxes | A build check, not scientific validation; ordinary underfull-line warnings remain | | ||
| | Preserved artifact comparison | All seven historical PDF/HTML/BBL/bundle/metadata/dossier/review files byte-identical to baseline | Remote deposit not checked | | ||
| | Visual inspection | Pages 1, 3 and 5 readable, tables/equations/references fit | Sampled layout review | | ||
|
|
||
| The first TeX build inadvertently selected the archived BBL because compilation started in `paper/`. The corrected build runs entirely in a separate build directory with an explicit bibliography search path; the README documents this requirement. No archived output was overwritten. MiKTeX emitted an update-check reminder; no package installation or update was performed. | ||
|
|
||
| Local review outputs are in `C:/Users/Owner/PROJECTS/_runs/coronal-papers-20261010/`: `nakedagent-claims.json`, `nakedagent-demo.txt`, `nakedagent-tests.txt`, `nakedagent-archived-artifacts.json`, `nakedagent-validation-manifest.json`, and `nakedagent-build/`. These local outputs are not a published dataset or release archive. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Compare the baseline file inventory with the current file inventory.
The current glob cannot detect a tracked module that is missing from the worktree. If that module is not needed to import this checker, every remaining file can match the baseline and the source check can pass. Derive the expected Python paths from the pinned commit, then compare the path sets before comparing file contents. Include nested Python modules if they are part of the claimed runtime inventory.
🤖 Prompt for AI Agents