diff --git a/paper/README.md b/paper/README.md new file mode 100644 index 0000000..dc46962 --- /dev/null +++ b/paper/README.md @@ -0,0 +1,67 @@ +# NakedAgent paper + +The **unreleased living manuscript** is [nakedagent_paper.tex](nakedagent_paper.tex), revised October 10, 2026, with [references.bib](references.bib). Its title is *NakedAgent: Standard-Library Coding Agents with Explicit Effect Boundaries and Offline Replay*. It describes the Python reference at commit `0c96f1a8adac035b9d204ebc78c5eb4175c13685`, not every port or deployment. + +The repository-recorded historical DOI is [10.5281/zenodo.23001340](https://doi.org/10.5281/zenodo.23001340). This revision does not create a DOI or claim the remote deposit was independently checked. + +## Artifact status + +| File | Status | +|---|---| +| `nakedagent_paper.tex`, `references.bib` | Current review source; changes include substantive corrections to earlier claims | +| `check_claims.py` | New offline source inventory and eight synthetic checks; no model or real tool execution | +| [nakedagent.notebook.md](nakedagent.notebook.md) | Post-hoc audit record and prospective evaluation plan | +| `nakedagent_paper.pdf`, `nakedagent_paper.html`, `nakedagent_paper.bbl` | Preserved historical renderings; **do not describe the revised manuscript** | +| `NakedAgent_arxiv_bundle.tar.gz` | Preserved historical source bundle; not regenerated | +| `.zenodo.json`, `TechRxiv_SSRN_Submission_Dossier.md` | Historical deposit/submission material; not approved metadata for this revision | +| `ARCANA_PEER_REVIEW_SYNTHESIS.md` | Historical review record; its verdict does not apply to the new source | + +Historical metadata and renderings contain claims narrowed in the living manuscript, including non-repudiation, supply-chain elimination and execution proof. Do not use them to submit the new version. A future release needs consistent new renderings, reviewed metadata and a complete source/check archive. The original artifacts remain unchanged so the correction is traceable. + +## Source-to-claim map + +All paths below refer to the pinned commit. The claim checker rejects runtime source that differs from that baseline, after normalizing Git/worktree line endings. + +| Claim | Source | What it does not establish | +|---|---|---| +| No declared third-party Python runtime packages | `pyproject.toml`; static imports in `nakedagent/*.py` | No build requirements, no interpreter/provider risk, or dependency-free plugins | +| Ordered, nesting-aware fenced-call parsing | `nakedagent/toolcall.py`; `spec/SPEC.md` | Superior model compliance or injection resistance | +| File and shell tool behavior | `nakedagent/tools.py` | Atomic writes, universal path isolation, or safe semantics for every allowed program | +| Registry substitution updates prompt examples | `nakedagent/plugins.py`, `nakedagent/loop.py` | Plugin sandboxing or correctness of `.usage` text | +| Optional functional driver | `nakedagent/cli.py`, `nakedagent/driver.py` | All CLI paths use the reducer or install Sidekick policy | +| Hash-linked events and replay | `nakedagent/functional.py`, `eventlog.py`, `replay.py` | Authenticated origin, full effect causality, genuine model identity or actual tool effects | +| Tri-state extension point | `nakedagent/sidekick.py` | A calibrated risk estimate or cryptographic approval protocol | +| Historical pilot observations | `bench/RESULTS.md`, `bench/run_nakedagent_swebench.py`, `bench/score_preds.py` | Reproduced raw trajectories, controlled comparison or general task-success rate | + +The current foundation conformance specification excludes replay, the functional driver, Sidekick and MCP. Foundation port conformance therefore cannot be cited as evidence for those features. + +## Offline checks + +From the repository root, on Python 3.10+: + +```text +python paper/check_claims.py +python -m examples.replay_demo +python -m unittest discover -s tests +git diff --check -- paper +``` + +The claim checker writes JSON to stdout and creates only temporary synthetic logs. It imports pinned runtime functions directly without loading plugins. Expected cases include stale-hash edit rejection, coherent rewritten-history acceptance, acceptance of a reported tool result without a prior request, missing-final and post-terminal rejection, and event-budget dispatch boundaries. These are small falsifiable examples of the paper's scope, not exhaustive verification or a benchmark rerun. + +## Build the living paper + +Use an already installed TeX distribution with `pdflatex`, `bibtex` and the packages listed in the preamble. Run **all build commands from a separate build directory**, giving the TeX source's absolute path and setting the bibliography search path. Compiling from `paper/` can accidentally load the archived BBL instead of the new bibliography. + +PowerShell example; substitute the actual paths and create an empty build directory first: + +```powershell +$env:BIBINPUTS = 'C:/absolute/path/to/nakedagent/paper;' +Push-Location 'C:/absolute/path/to/build' +pdflatex -no-shell-escape -interaction=nonstopmode -halt-on-error C:/absolute/path/to/nakedagent/paper/nakedagent_paper.tex +bibtex nakedagent_paper +pdflatex -no-shell-escape -interaction=nonstopmode -halt-on-error C:/absolute/path/to/nakedagent/paper/nakedagent_paper.tex +pdflatex -no-shell-escape -interaction=nonstopmode -halt-on-error C:/absolute/path/to/nakedagent/paper/nakedagent_paper.tex +Pop-Location +``` + +On MiKTeX, `--disable-installer` can additionally prevent automatic package installation. Do not overwrite the historical PDF or BBL. Review the generated `nakedagent_paper.pdf` and log in the build directory; a successful compile does not validate scientific claims. The revision notebook records the observed build/test results and unresolved data limitations. diff --git a/paper/check_claims.py b/paper/check_claims.py new file mode 100644 index 0000000..17c4c5d --- /dev/null +++ b/paper/check_claims.py @@ -0,0 +1,135 @@ +"""Offline paper evidence: source inventory and replay-boundary counterexamples. + +Run from the repository root: python paper/check_claims.py +Uses synthetic events, a stub model and stub tool; never loads plugins, calls a +provider, executes a shell tool, or changes runtime source. Temporary logs are +deleted on exit. Passing is evidence about these cases, not a security proof. +""" +from __future__ import annotations + +import ast +import hashlib +import json +import subprocess +import sys +from pathlib import Path +from tempfile import TemporaryDirectory + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from nakedagent.driver import run_functional +from nakedagent.eventlog import open_log +from nakedagent.functional import AgentEvent, AgentState, agent_reducer +from nakedagent.replay import verify + +BASELINE = "0c96f1a8adac035b9d204ebc78c5eb4175c13685" + + +def make_log(path: Path, events: list[AgentEvent]) -> str: + system = "Synthetic paper claim check; no external effects." + policy = {"model": "stub", "workspace": "synthetic", "extra": {}} + state = AgentState.initial(system, policy=policy) + writer = open_log(path, system_prompt=system, model="stub", workspace="synthetic", extra={}) + try: + for event in events: + state, _ = agent_reducer(state, event) + writer.write_event(event, state.current_hash()) + finally: + writer.close(state.step_count, state.current_hash(), state.is_terminal, + state.terminal_reason, state.suspended) + return state.current_hash() + + +def main() -> int: + source_paths = sorted((ROOT / "nakedagent").glob("*.py")) + source_paths += [ROOT / "pyproject.toml"] + source_matches = {} + imports: set[str] = set() + for path in source_paths: + relative = path.relative_to(ROOT).as_posix() + data = path.read_bytes() + archived = subprocess.check_output( + ["git", "show", f"{BASELINE}:{relative}"], cwd=ROOT) + # Git stores LF while Windows worktrees may use CRLF. + normalized = data.replace(b"\r\n", b"\n") + source_matches[relative] = { + "sha256_lf": hashlib.sha256(normalized).hexdigest(), + "matches_baseline": normalized == archived.replace(b"\r\n", b"\n"), + } + if path.suffix == ".py": + for node in ast.walk(ast.parse(data)): + if isinstance(node, ast.Import): + imports.update(n.name.split(".")[0] for n in node.names) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + imports.add(node.module.split(".")[0]) + external_imports = sorted(imports - sys.stdlib_module_names - {"nakedagent"}) + assert not external_imports, external_imports + assert all(item["matches_baseline"] for item in source_matches.values()) + + events = [AgentEvent("USER_INPUT", "Hello"), + AgentEvent("MODEL_REPLY", "Hello back"), + AgentEvent("TERMINATION", "DONE")] + checks = {} + with TemporaryDirectory(prefix="nakedagent-paper-") as temp: + work = Path(temp) + original = work / "original.jsonl" + first_hash = make_log(original, events) + checks["original_accepted"] = verify(original)[0] + duplicate = work / "duplicate.jsonl" + checks["same_inputs_same_hash"] = make_log(duplicate, events) == first_hash + + records = [json.loads(x) for x in original.read_text(encoding="utf-8").splitlines()] + records[2]["payload"] = "Different reply" + stale = work / "stale.jsonl" + stale.write_text("".join(json.dumps(x) + "\n" for x in records), encoding="utf-8") + checks["stale_hash_edit_rejected"] = not verify(stale)[0] + + rewritten = work / "rewritten.jsonl" + new_hash = make_log(rewritten, [events[0], AgentEvent("MODEL_REPLY", "Different reply"), events[2]]) + checks["coherent_rewrite_accepted"] = verify(rewritten)[0] and new_hash != first_hash + + orphan = work / "orphan.jsonl" + make_log(orphan, [events[0], AgentEvent("TOOL_RESULT", "Claimed success", {"tool": "write"}), events[2]]) + checks["unrequested_tool_result_accepted"] = verify(orphan)[0] + + truncated = work / "truncated.jsonl" + truncated.write_text("\n".join(original.read_text(encoding="utf-8").splitlines()[:-1]) + "\n", encoding="utf-8") + checks["missing_final_rejected"] = not verify(truncated)[0] + + post_terminal = work / "post-terminal.jsonl" + make_log(post_terminal, events + [AgentEvent("USER_INPUT", "After termination")]) + checks["post_terminal_event_rejected"] = not verify(post_terminal)[0] + + dispatch_counts = {} + for budget in (2, 3): + calls: list[str] = [] + def stub_tool(args: str, content: str, workspace: Path, *, _calls=calls) -> str: + _calls.append(content) + return "synthetic result" + def stub_model(*args, **kwargs) -> str: + return "```probe\nsynthetic request\n```" + state = run_functional("Hello", "stub", work, + tools={"probe": stub_tool}, llm_fn=stub_model, + max_steps=budget) + dispatch_counts[str(budget)] = len(calls) + assert state.step_count == budget and state.is_terminal + checks["budget_2_dispatches_0_budget_3_dispatches_1"] = dispatch_counts == {"2": 0, "3": 1} + + assert all(checks.values()), checks + report = { + "baseline": BASELINE, + "python": sys.version.split()[0], + "platform": sys.platform, + "scope": "synthetic checks; no provider, plugin, shell tool, or benchmark run", + "external_static_imports": external_imports, + "source": source_matches, + "checks": checks, + "interpretation": "Expected coherent-rewrite and orphan-result acceptance delimit the verifier; they are not authenticated execution evidence.", + } + print(json.dumps(report, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/paper/nakedagent.notebook.md b/paper/nakedagent.notebook.md new file mode 100644 index 0000000..4691255 --- /dev/null +++ b/paper/nakedagent.notebook.md @@ -0,0 +1,61 @@ +# NakedAgent research notebook + +## Provenance + +This is a **post-hoc source audit**, begun October 10, 2026. It is not the notebook of the September 14 pilot and not a preregistration. The baseline is `0c96f1a8adac035b9d204ebc78c5eb4175c13685`. Runtime code and archived paper artifacts are preserved. + +Question: which architectural and replay claims are supported by the current source, and what can be reproduced without a model service or unavailable pilot logs? + +## Findings and revision decisions + +1. Retained the constructive standard-library architecture and Omakase substitution contribution. Scoped zero dependencies to the foundation's third-party Python runtime packages; the build backend, interpreter, providers, commands and plugins are separate dependencies. +2. Distinguished the default imperative loop, optional event-log driver and Sidekick harness. Corrected event-budget semantics, terminal budget exhaustion, default gate heuristics and the absence of automatic Sidekick gating in the CLI functional path. +3. Replaced a fictitious state tuple and hash formula with the actual v0.4 fields. The state is not finite in the strict automata-theoretic sense. The linear chain is not a Merkle tree with inclusion proofs; the reducer's receipt can record proposed actions suppressed at terminal exhaustion. +4. Corrected replay's claim to internal consistency. Rehashing a complete modified transcript is possible; a reported TOOL_RESULT needs no matched pending request. Neither authenticated origin nor tool-effect truth follows from a PASS result. +5. Corrected atomic-write, numbered-read, single-regex parser and universal read-only claims. Disabling shell/write leaves patch active. Even disabling all three mutating built-ins does not constrain arbitrary plugin imports. +6. Preserved the pilot's useful debugging sequence while identifying C1/C2 as invalid harness configurations. The narrative reports C3 zero of three resolved and seven regressions on one patch; no raw trajectories or test output are tracked at the baseline. The manuscript does not claim those results were reproduced or establish a local-model capability ceiling. +7. Preserved historical PDF, HTML, BBL, bundle, submission dossier and Zenodo metadata. Added an explicit distinction from the living TeX. No submission, deposit update or DOI minting occurred. + +## Primary-source checks + +Opened on October 10, 2026: + +- [SWE-bench paper](https://arxiv.org/abs/2310.06770): task/evaluation context and bibliographic metadata. +- [SWE-agent paper](https://arxiv.org/abs/2405.15793): interface-design motivation; no numerical comparison with NakedAgent inferred. +- [gptme project](https://github.com/gptme/gptme): provenance for the acknowledged terminal-tool architecture; exact inheritance is stated by local source comments, not inferred from current upstream behavior. +- [Aider edit-format documentation](https://aider.chat/docs/more/edit-formats.html): SEARCH/REPLACE precedent. +- [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162): distinction between local hash chaining and a transparency protocol with signed checkpoints/consistency mechanisms. No conformance claimed. +- [MCP 2024-11-05 stdio transport](https://modelcontextprotocol.io/specification/2024-11-05/basic/transports): version matches the local client constant. No remote MCP was connected. + +The remote historical Zenodo deposit was not independently verified. The DOI is attributed to repository records. + +## Executable evidence + +`paper/check_claims.py` uses only synthetic events, a stub model and a stub tool. It checks LF-normalized runtime source hashes against the baseline and eight named cases. Complete command output and build products are kept outside the repository in the operator's run directory; observed results are recorded below after execution. + +The checks are post-hoc examples selected from this audit. They are not a test sample for an estimated security failure rate, and they do not demonstrate externally executed effects. + +## Prospective study + +Before a new empirical run, freeze source/plugin versions, dataset task selection, model digest/quantization, provider settings, prompt variants, container images and scoring version. Define matched sampling and repeated-run counts before inspecting outcomes. Predetermine invalid-infrastructure classifications and rerun rules. Archive exact patch bytes, predictions, trajectories and full grader outputs. + +Keep parse compliance, successful dispatch, patch applicability, resolved tests, regressions and final task success separate. Use one-factor comparisons for fence/native calls, prompt examples and shell/file tools. This protocol is proposed and has not been executed here. + +## Observed checks, October 10, 2026 + +Environment: Windows, Python 3.14.6. No model benchmark or provider request was made. + +| Check | Observed result | Limits | +|---|---|---| +| `python paper/check_claims.py` | Exit 0; eight expected outcomes; runtime/metadata files match the pinned baseline; no external static imports | Synthetic examples and static import inventory, not complete execution verification | +| `python -m examples.replay_demo` | Exit 0; original three-event trace accepted; changed payload rejected | Partial-edit detection with hashes held fixed | +| `python -m unittest discover -s tests` | 498 tests run, OK, 45 skipped, 31.746 s | Windows/environment skips remain; suite emits existing mock HTTP resource warnings | +| `ruff check paper/check_claims.py` | Passed | Companion script only; runtime unchanged | +| `git diff --check -- paper` | Passed | Whitespace validation | +| PDFLaTeX + BibTeX + two PDFLaTeX passes | Five-page review PDF; no undefined citations or overfull boxes | A build check, not scientific validation; ordinary underfull-line warnings remain | +| Preserved artifact comparison | All seven historical PDF/HTML/BBL/bundle/metadata/dossier/review files byte-identical to baseline | Remote deposit not checked | +| Visual inspection | Pages 1, 3 and 5 readable, tables/equations/references fit | Sampled layout review | + +The first TeX build inadvertently selected the archived BBL because compilation started in `paper/`. The corrected build runs entirely in a separate build directory with an explicit bibliography search path; the README documents this requirement. No archived output was overwritten. MiKTeX emitted an update-check reminder; no package installation or update was performed. + +Local review outputs are in `C:/Users/Owner/PROJECTS/_runs/coronal-papers-20261010/`: `nakedagent-claims.json`, `nakedagent-demo.txt`, `nakedagent-tests.txt`, `nakedagent-archived-artifacts.json`, `nakedagent-validation-manifest.json`, and `nakedagent-build/`. These local outputs are not a published dataset or release archive. diff --git a/paper/nakedagent_paper.tex b/paper/nakedagent_paper.tex index 06d4aa6..39d6122 100644 --- a/paper/nakedagent_paper.tex +++ b/paper/nakedagent_paper.tex @@ -1,287 +1,201 @@ \documentclass[10pt,twocolumn,letterpaper]{article} - \usepackage[margin=0.75in]{geometry} -\usepackage{times} -\usepackage{amsmath,amssymb,amsfonts} -\usepackage{algorithmic} -\usepackage{graphicx} -\usepackage{textcomp} -\usepackage{xcolor} -\usepackage{booktabs} -\usepackage{listings} -\usepackage{url} -\usepackage{hyperref} +\usepackage{times,amsmath,amssymb,graphicx,booktabs,listings,xcolor} +\usepackage{xurl} +\usepackage[colorlinks=true,allcolors=blue]{hyperref} \usepackage{microtype} - -\hypersetup{ - colorlinks=true, - linkcolor=blue, - citecolor=blue, - urlcolor=blue, -} - -\lstset{ - basicstyle=\ttfamily\footnotesize, - breaklines=true, - frame=single, - numbers=none, - keywordstyle=\color{blue}\bfseries, - commentstyle=\color{gray}\itshape, - stringstyle=\color{teal}, - showstringspaces=false -} - -\title{NakedAgent: A Zero-Dependency Coding Agent Architecture via Standard-Library Primitives and Provable Replay} - -\author{ - \textbf{Reuben Bowlby} \\ - HUMMBL, LLC \\ - \texttt{reuben@hummbl.io} \\ - ORCID: \href{https://orcid.org/0009-0002-5620-1103}{0009-0002-5620-1103} -} - -\date{September 2026} - +\setlength{\emergencystretch}{2em} +\lstset{basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,showstringspaces=false} +\title{NakedAgent: Standard-Library Coding Agents with Explicit Effect Boundaries and Offline Replay} +\author{\textbf{Reuben Bowlby}\\HUMMBL, LLC\\\texttt{reuben@hummbl.io}\\ +ORCID: \href{https://orcid.org/0009-0002-5620-1103}{0009-0002-5620-1103}} +\date{Living revision, October 10, 2026 --- unreleased} \begin{document} - \maketitle - \begin{abstract} -Autonomous software engineering agents driven by Large Language Models (LLMs) increasingly rely on expansive, multi-tiered dependency trees. Mainstream agent scaffolds (e.g., SWE-agent, Aider, OpenHands) import between 20 and 50+ third-party packages to facilitate HTTP communication, terminal UI rendering, provider protocol abstraction, and telemetry. While feature-rich, this architectural paradigm introduces substantial supply-chain vulnerability attack surfaces, runtime version fragility, and heavy deployment footprints. In this paper, we introduce NakedAgent, an autonomous coding agent engineered under a strict zero-runtime-dependency constraint: it executes entirely upon standard Python 3.10+ library modules (urllib, json, subprocess, re, and hashlib). NakedAgent decouples tool invocation from proprietary JSON function-calling schemas by utilizing a Markdown fenced code-block protocol, enabling consistent tool use across local models (such as Qwen 2.5-Coder and Gemma 3) and hosted endpoints. To establish auditability and tamper-evident transcript integrity, we model agent execution as a pure Mealy state machine reducer $(S_t, E_t) \to (S_{t+1}, A_{t+1})$, combined with SHA-256 Merkle-linked state hashing recorded in an append-only JSON Lines (JSONL) event log. This architecture enables zero-inference offline replay verification, validating tool execution histories without querying an LLM. We present empirical findings from a SWE-bench Lite pilot with local 12B models, delineating scaffold-induced failure modes from model-inherent reasoning bounds. Finally, we formulate the Omakase extensibility doctrine, demonstrating how clean tool substitution seams empower extensible agent behaviors without framework bloat. +NakedAgent explores how much of a terminal coding agent can be implemented using Python's standard library. Its Python package declares no third-party runtime dependencies and provides four foundation tools, a fenced-text invocation grammar, two HTTP wire adapters, and a tool-substitution interface that also updates prompt examples. An optional functional execution path separates deterministic state transitions from model and tool effects. Its versioned JSONL verifier replays reported events without model inference or tool execution and detects inconsistencies against recorded hashes and final-state fields. This is a transcript-consistency mechanism: it neither authenticates a rewritten transcript nor proves that reported effects occurred. We specify the actual hash inputs, distinguish three execution paths and their budgets, and provide offline counterexamples that delimit these guarantees. A previously recorded three-instance SWE-bench Lite pilot supplies a failure-analysis case study, including invalid harness runs and a reported corrected run with zero resolved tasks. Its absent raw trajectories and uncontrolled sampling prevent comparative performance conclusions. The contribution is an inspectable architecture and a reproducibility contract, not a proof of secure or capable autonomous execution. \end{abstract} -\section{Introduction} -Large Language Models have catalyzed a paradigm shift in software engineering automation~\cite{jimenez2024swebench,yang2024sweagent}. Modern coding agents read repositories, analyze stack traces, formulate patch hypotheses, and execute shell commands iteratively within feedback loops~\cite{yao2022react,shinn2023reflexion}. +\section{Scope and contribution} +Coding agents combine model outputs with repository inspection, editing and command execution. SWE-agent studies how the agent--computer interface affects performance~\cite{yang2024sweagent}; SWE-bench provides issue-resolution tasks and test-based evaluation~\cite{jimenez2024swebench}. NakedAgent asks a narrower architectural question: can the interaction loop, tool dispatch, provider transport and optional replay layer be built without third-party \emph{Python runtime packages}? -However, existing state-of-the-art coding agent frameworks—including Aider~\cite{gauthier2023aider}, SWE-agent~\cite{yang2024sweagent}, gptme~\cite{gptme2023}, and OpenHands~\cite{openhands2024}—are characterized by substantial dependency bloat. Typical installations pull in dozens of transitive packages spanning asynchronous networking libraries (\texttt{httpx}, \texttt{aiohttp}), rich terminal user interface kits (\texttt{rich}, \texttt{prompt\_toolkit}), distributed telemetry, and vendor-specific SDKs. +The answer at the inspected source revision is constructive: these components exist using the standard library~\cite{nakedagentsource}. That fact does not establish competitive task success, lower defect rates, or lower total supply-chain risk. Python, the operating system, model server, model weights, command-line programs and optional plugins remain dependencies of a deployment. The package also uses a setuptools build backend; build-time requirements are outside the runtime count. -This dependency proliferation incurs three critical systemic liabilities: -\begin{enumerate} - \item \textbf{Supply-Chain Attack Surfaces}: Every additional package imported into an agent runtime introduces latent Common Vulnerabilities and Exposures (CVEs) and vectors for malicious compromise~\cite{ohm2020supplychain,ladisa2023supplychain}. Because coding agents inherently possess arbitrary shell execution and filesystem modification privileges within their execution sandbox, compromising an agent's dependency tree yields immediate remote code execution. - \item \textbf{Version Fragility and Upstream Churn}: Minor version updates across transitive dependencies frequently break agent loops or corrupt internal prompt structures, undermining long-term production stability. - \item \textbf{Container and Footprint Overhead}: Deploying agents into resource-constrained edge systems, embedded devices, or air-gapped secure enclaves becomes prohibitive when megabytes of external dependencies must be containerized and verified. -\end{enumerate} - -To confront this trade-off, we propose \textbf{NakedAgent}. NakedAgent poses a fundamental research question: \textit{Can an autonomous coding agent achieve robust, auditable, and extensible code modification loops while maintaining exactly zero runtime dependencies?} +This revision contributes (1) a source-bound description of the architecture and its substitution seam; (2) a conditional replay argument with an explicit threat model and executable counterexamples; and (3) a separation of historical pilot observations from a prospective evaluation protocol. It does not claim invention of fenced tool calls, search/replace editing, functional reducers or hash chains. -By restricting the runtime strictly to the standard Python library (\texttt{stdlib}), NakedAgent eliminates supply-chain exposure by construction. In this paper, we describe the architectural foundations of NakedAgent, its formal Mealy state machine execution lane, its tamper-evident Merkle event logging and offline replay mechanism, its empirical performance on SWE-bench Lite, and the Omakase extensibility doctrine. +\paragraph{Version boundary.} +All present-tense implementation statements refer to the Python reference at commit +\url{0c96f1a8adac035b9d204ebc78c5eb4175c13685}. +The package metadata says version 0.1.0, while the current event-log schema is v0.4; these are distinct version namespaces. The September paper PDF, HTML, compiled bibliography and arXiv bundle remain historical artifacts. This unreleased TeX revision supersedes their claims for review, without asserting a new DOI or deposit. The repository-recorded historical DOI is \url{https://doi.org/10.5281/zenodo.23001340}; its remote contents were not independently verified in this revision. -\section{System Architecture} +\section{Architecture and effect boundaries} +\subsection{Dependency scope} +Let $D_P$ be the third-party Python packages required by the foundation runtime, excluding plugins and external services. The pinned \texttt{pyproject.toml} declares $D_P=\varnothing$. The companion script statically enumerates absolute imports in \texttt{nakedagent/*.py} and checks them against the executing interpreter's standard-library module inventory. This supports the package-level claim, not a complete software bill of materials or a guarantee about dynamically imported plugins. -\subsection{Foundational Invariants} -NakedAgent is governed by three primary architectural invariants: -\begin{equation} - |\mathcal{D}_{\text{runtime}}| = 0 -\end{equation} -where $\mathcal{D}_{\text{runtime}}$ represents the set of external third-party packages required to execute the agent. +The provider adapter in \texttt{llm.py} uses \texttt{urllib.request} for Ollama's \texttt{/api/chat} and an OpenAI-compatible chat endpoint. API keys are obtained from a named environment variable. The Ollama default is \texttt{http://localhost:11434}; HTTP over loopback is still network communication, and the configurable host can point elsewhere. Compatibility with every server advertising a similar API is not established. The source supports incremental response parsing through the \texttt{stream} argument, so absence of a rich terminal framework is not absence of streaming transport. -\begin{equation} - \forall t \in \mathcal{T}_{\text{foundation}}, \quad t: (\mathcal{A}_{\text{args}}, \mathcal{W}_{\text{context}}) \to \mathcal{R}_{\text{result}} -\end{equation} -where $\mathcal{T}_{\text{foundation}}$ is a minimal set of four deterministic tools operating within workspace $\mathcal{W}_{\text{context}}$. +\subsection{Four foundation tools} +The registry exposes functions with the interface +\[ + t:(\text{args},\text{content},\text{workspace})\longrightarrow\text{result string}. +\] +This signature standardizes dispatch; it does not make filesystem or subprocess effects pure or deterministic. -\subsection{The Four Foundation Tools} -Rather than supplying extensive specialized APIs, NakedAgent constrains its foundation to four primitive tools: \begin{itemize} - \item \textbf{\texttt{read} [path]}: Inspects file contents within the workspace boundary. To assist model attention across long documents, line numbering is formatted cleanly, and path escapes traversing outside the target workspace are rejected. - \item \textbf{\texttt{write} [path]}: Atomically writes or overwrites content to disk. Directory paths are created on demand, preventing path traversal vulnerabilities. - \item \textbf{\texttt{patch} [path]}: Applies search-and-replace hunks. Rather than relying on non-deterministic diff algorithms, the patch tool requires exact, unique substring matches: - \begin{equation} - \text{count}(S_{\text{search}}, F_{\text{content}}) = 1 - \end{equation} - If the search target occurs zero times (missing) or more than once (ambiguous), the patch fails closed with an explicit diagnostic message, preventing silent code corruption. - \item \textbf{\texttt{shell} [cmd]}: Executes system commands via \texttt{subprocess.run}. In interactive mode, shell invocations require human confirmation. In automated or headless runs, shell access is guarded by an explicit \texttt{--allow-shell} flag and a tokenized, prefix-based allowlist (\texttt{--shell-allowlist}). +\item \texttt{read} resolves a path relative to the workspace, rejects a resolved escape, and returns text or a directory listing. It does \emph{not} add line numbers, because a model might copy them into an exact-match patch. +\item \texttt{write} creates parent directories and calls \texttt{Path.write\_text}. The source does not implement a temporary-file-and-rename atomic-write protocol. +\item \texttt{patch} accepts a SEARCH/REPLACE block and requires exactly one occurrence of the search text in the current decoded file. It refuses missing or ambiguous matches and invalid UTF-8. Unique matching localizes the chosen replacement; it does not prove that the edit is correct, preserve all byte-level encodings, or eliminate concurrent-write races. +\item \texttt{shell} prompts for a complete command in interactive mode. Noninteractive execution requires both an allow switch and a matching tokenized command prefix; it executes an argument vector with \texttt{shell=False}. Interactive approval selects a different path using \texttt{shell=True}. \end{itemize} -\subsection{Schema-Free Tool Calling via Code Fences} -Most commercial agent frameworks rely on proprietary JSON function-calling formats (e.g., OpenAI Tool Calls, Anthropic Tool Use). However, open-weights and smaller localized models (e.g., 7B--14B parameter models running on consumer GPUs via Ollama) frequently struggle with strict JSON schema serialization, resulting in decoding failures and aborted trajectories. +The file tools check workspace containment and protect top-level \texttt{.git} and \texttt{.nakedagent} targets. These are application checks, not an operating-system sandbox. Shell commands and arbitrary plugin code have separate authority, and filesystem races or host-specific aliases require separate evaluation. -NakedAgent resolves this via \textit{Markdown Code-Fence Tool Parsing}. The system prompt instructs the language model to invoke tools by emitting standard triple-backtick fenced blocks where the language identifier denotes the tool: -\begin{lstlisting}[language=bash] +\subsection{Fenced-text invocation} +A response is parsed into an \emph{ordered list} of tool calls. The grammar uses a column-zero triple-backtick opener, a tool name, optional arguments and a body. The parser is a line scanner with a stack of fence lengths, not a single regular-expression extraction. It normalizes CRLF, tracks nested fences and emits a reserved refusal call for an unclosed block. See \texttt{toolcall.py} and the foundation specification for exact rules. + +\begin{lstlisting} ```patch src/calculator.py <<<<<<< SEARCH -def add(a, b): - return a - b +return a - b ======= -def add(a, b): - return a + b +return a + b >>>>>>> REPLACE ``` \end{lstlisting} -A deterministic regular expression scanner extracts tool blocks from the generated token stream: -\begin{equation} - \mathcal{P}_{\text{tool}}: \text{Response} \to \{(\text{name}_i, \text{target}_i, \text{payload}_i)\}_{i=1}^k -\end{equation} -Because every modern instruction-tuned model is pre-trained extensively on Markdown code blocks, tool-calling success rates remain high even on small 7B parameter models without fine-tuned JSON function calling capabilities. - -\subsection{Transport Abstraction} -Network transport is implemented via standard \texttt{urllib.request}. NakedAgent natively supports two wire protocols: -\begin{enumerate} - \item \textbf{Ollama Local API}: Default local inference via \texttt{/api/chat}, operating with zero network exposure. - \item \textbf{OpenAI-Compatible Wire API}: Supporting vLLM, LM Studio, and remote API gateways using standard Bearer token authentication loaded from environment variables (\texttt{OPENAI\_API\_KEY}). -\end{enumerate} - -\section{Provable Execution and Replay Verification} - -\subsection{The Mealy Machine State Reducer} -To provide mathematical rigor and eliminate unverified hidden state, NakedAgent models automated runs as a finite state Mealy machine: -\begin{equation} - \delta: (S_t, E_t) \to (S_{t+1}, [A_{t+1}]) -\end{equation} -where $S_t$ is the immutable agent state at turn $t$, $E_t$ is an incoming event, and $A_{t+1}$ represents outward actions dispatched to the environment (such as model requests or tool executions). - -The agent state $S_t$ is a frozen dataclass encapsulating: -\begin{equation} - S = \langle \text{status}, \text{messages}, \text{budget}, \text{pending\_actions}, \text{meta} \rangle -\end{equation} -Transitions are purely functional and side-effect free: all I/O is isolated in a thin driver layer (\texttt{driver.py}). - -\subsection{Merkle State Hashing and Event Logging} -Every transition is logged sequentially into an append-only JSONL stream. Each record is cryptographically linked to the preceding state through SHA-256 Merkle chaining: -\begin{equation} - H_0 = \text{SHA-256}(\text{GenesisHeader}) -\end{equation} -\begin{equation} - H_t = \text{SHA-256}(H_{t-1} \parallel \text{CanonicalJSON}(E_t) \parallel \text{Digest}(S_t)) -\end{equation} - -This cryptographic formulation guarantees: -\begin{itemize} - \item \textbf{Tamper Evidence}: Any post-hoc mutation of tool outputs, prompt injections, or agent actions invalidates all downstream hashes in the chain. - \item \textbf{Strict Causality}: Events cannot be reordered or inserted retroactively. -\end{itemize} +This avoids requiring a provider's native function-calling wire format; it still imposes a tool-call grammar on model text. Tagged examples at column zero can be interpreted as requests. No measured advantage over native tool calling, or general success rate for small models, follows from the grammar alone. The implementation acknowledges inspiration from gptme~\cite{gptme} and uses a search/replace convention comparable to Aider's documented edit formats~\cite{aiderformats}. -\subsection{Zero-Inference Offline Replay} -Auditing AI agent trajectories historically required either re-executing expensive LLM calls (which introduces non-determinism) or trusting unverified raw text transcripts. +\section{Execution paths and budgets} +\begin{table*}[t] +\centering\small +\begin{tabular}{p{0.17\textwidth}p{0.24\textwidth}p{0.48\textwidth}} +\toprule +Path & Transition/effect boundary & Scope of the contract \\ +\midrule +Default CLI loop & \texttt{loop.py}; model call, parser, tools & Iterative tool-call rounds; does not automatically produce the strict v0.4 replay log. \\ +One-shot event-log driver & \texttt{driver.py} and \texttt{functional.py} & Selected by \texttt{--event-log}; records admitted events, separates reducer from effects and checks available event slots before dispatch. No Sidekick Noul gate is automatically installed. \\ +Sidekick harness & \texttt{sidekick.py}; explicit provider and gate & Reuses the reducer and supports ALLOW/BLOCK/ESCALATE before tool dispatch. Its in-memory trace export is not the complete v0.4 JSONL file format. \\ +\bottomrule +\end{tabular} +\caption{Distinct Python execution paths at the inspected commit. A property of one path is not automatically a property of the others.} +\label{tab:paths} +\end{table*} -Under NakedAgent, any recorded event log $\mathcal{L} = [e_1, e_2, \dots, e_N]$ can be verified offline via: -\begin{lstlisting}[language=bash] -$ python -m nakedagent.replay run.jsonl -PASS run.jsonl: verified 5 events; final hash 6d1e09... +In the functional path, an admitted USER\_INPUT, MODEL\_REPLY, TOOL\_RESULT, ESCALATE, ALARM or TERMINATION consumes one event slot. Thus \texttt{max\_steps} is an event budget, not a model-token budget, wall-clock limit or number of successful tool executions. The default imperative loop counts rounds differently (Table~\ref{tab:paths}). Reporting only ``25 steps'' without naming the path is ambiguous. + +At the final event slot the reducer becomes terminal and suppresses returned dispatch actions. For example, with a budget of two, USER\_INPUT followed by a MODEL\_REPLY containing a tool request consumes the budget and dispatches no tool. With three slots, one synthetic tool result can be admitted. A transition receipt can contain parsed proposed actions even when the terminal transition returns no executable actions. These distinctions are covered by the companion check. + +The Sidekick gate's routes are ALLOW, BLOCK and ESCALATE. BLOCK adds an ALARM and terminates; ESCALATE records the held action and suspends unless the event budget is exhausted. A subsequent user event lifts suspension and the model re-decides. That is not a signed human authorization token or an atomic approval of the exact previous action. The default gate matches a short list of destructive strings and assigns fixed scores; it has no escalation band and supplies no calibrated risk probability. Broader gate policy is an extension point, not an established safety result. + +The standard-library MCP client is a subprocess/stdio JSON-RPC implementation with a declared 2024-11-05 protocol version. It mounts external tools into Sidekick before the first event. Stdio is a transport, not a guarantee that the child process lacks network access. The protocol's transport specification~\cite{mcptransport} does not by itself establish implementation conformance or authority isolation. + +\section{What offline replay establishes} +\subsection{Reducer and hash contract} +The core has Mealy-style form +\[ + \delta(S_i,E_{i+1})=(S_{i+1},A_{i+1}). +\] +It is a deterministic reducer over structured state, not a finite-state automaton in the strict sense: arbitrary text and metadata give an unbounded abstract state space, even with a bounded event count. The state contains a step counter and budget, message history, transition receipts, terminal status/reason, declared policy and suspension status. It does not contain a pending-action/result matching relation. + +Let $J$ be the implementation's UTF-8 encoding of JSON with sorted keys, compact separators, ASCII escaping and nonfinite numbers disallowed. Let $H$ denote SHA-256, $P$ the system prompt, $B$ the event budget and $Q$ the declared policy. Genesis is +\begin{align*} + G &= \{\text{system\_prompt}:P,\text{max\_steps}:B,\text{policy}:Q\},\\ + h_0 &= H(J(G)). +\end{align*} +For each admitted transition, the following \emph{named fields} are hashed: +\begin{lstlisting} +{ + "prev_hash": h_previous, + "step_index": i, + "event": {"event_type": kind, + "payload": text, + "metadata": metadata}, + "actions": [{"tool_name": name, + "args": args, + "content": content}, ...] +} \end{lstlisting} -The replay verifier re-executes the pure reducer $\delta(S_t, E_t)$ against the recorded tool results without initiating any network calls or model inferences. If the reconstructed Merkle state hash matches the terminal hash in the log, the execution is mathematically certified to have followed the reducer's operational semantics. +The receipt calls this digest \texttt{state\_hash}, but it is not a direct serialization of every state field. State is reconstructed by replay. This is a linear hash chain, rather than a branching Merkle tree with inclusion proofs. Nor is the Python serialization a claim of cross-language JSON canonicalization. Ports need separate hash-contract tests; the current foundation conformance specification explicitly excludes the functional/replay layer. -\textit{Epistemic Scope}: Importantly, this cryptographic replay certifies \textbf{internal transcript causality and reducer determinism}---specifically, that the internal state transitions $\delta(S_t, E_t) \to S_{t+1}$ followed verified rules without offline transcript tampering. It does \textit{not} certify external semantic correctness, external environment isolation, or model resistance to prompt injection, which must remain defended by OS-level sandboxing. +\subsection{Conditional consistency argument} +Fix the reducer/parser implementation, initial frame, and ordered event values. If their operations are deterministic on these values, repeated replay produces the same state and proposed actions, by induction on the event index: the initial state is fixed; equality of each predecessor state and event yields equality of the next state and receipt. This is an argument about the program's defined transition function. The companion examples are checks of particular cases, not a machine-checked proof of every implementation path. -\section{The Omakase Extensibility Doctrine} +The strict verifier parses the versioned header, contiguous event sequence and required final trailer; recomputes per-event hashes; compares them to the supplied hashes; and checks final status, reason, step count and hash. Post-terminal events and ignored suspended-state events are rejected. A successful log ends terminal or suspended. No provider, tool or plugin is executed during verification. -\subsection{Defaults over Decisions} -Inspired by the Japanese concept of \textit{Omakase} (``chef's choice''), NakedAgent provides highly tuned, opinionated defaults: out-of-the-box, a developer requires no configuration flags or parameter tuning. +With a fixed trusted reference hash outside the attacker's control and suitable hash-security assumptions, modifying committed inputs should produce a mismatch. The stock verifier has no external reference-hash argument or authenticated recorder. It compares values supplied in the same local file. A writer able to change the file can generate a different valid event sequence and recompute its entire chain and trailer. It can also replace a run with another complete, internally valid run. The required trailer catches an incomplete ending, not all malicious history replacement. -\subsection{The Plugin Seam} -Extensibility is achieved not through heavyweight frameworks or abstract base classes, but via a filesystem convention seam: -\begin{itemize} - \item Plugins are plain Python files placed in \texttt{.nakedagent/plugins/} (repo-local) or \texttt{\~{}/.nakedagent/plugins/} (user-global). - \item A plugin exposes an optional \texttt{TOOLS: dict[str, ToolFunc]} and an optional \texttt{DISABLE: list[str]}. -\end{itemize} +\subsection{Threat model and counterexamples} +The relevant cases differ: +\begin{enumerate} +\item \textbf{Accidental or partial modification:} editing a payload while retaining its recorded hashes is rejected. +\item \textbf{Coherent replacement:} editing events and recomputing hashes can pass. There is no signature-based origin authentication, non-repudiation or independently witnessed history in the current log. +\item \textbf{False effect report:} a TOOL\_RESULT can be admitted without a prior matching tool request. The reducer records it as input, so replay alone does not establish dispatch causality or that the stated result is true. +\item \textbf{Different environment:} identical reported inputs replay identically even if the model identity, executed binary, files or external effects were misreported. The declared model/workspace fields are commitments to labels, not measurements of those resources. +\end{enumerate} -\subsection{Dynamic Prompt Adaptation} -In classical agent architectures, modifying a tool requires manually modifying the system prompt text to reflect the new syntax. In NakedAgent, every tool function attaches an intrinsic \texttt{.usage} metadata attribute. +The companion \texttt{paper/check\_claims.py} exercises these distinctions. In particular, coherent replacement and an unrequested TOOL\_RESULT are \emph{expected acceptance} cases. Their acceptance limits the claim rather than demonstrating a failure of deterministic hashing. A transparency design with signed checkpoints, retained independent copies and consistency verification would add different guarantees; Certificate Transparency illustrates such additional machinery~\cite{rfc9162}. We do not claim that NakedAgent implements that protocol. -When plugins are discovered at startup, the system prompt generator dynamically renders the tool documentation block: -\begin{equation} - \text{Prompt}_{\text{system}} = \mathcal{B}_{\text{core}} \cup \bigcup_{t \in \mathcal{T}_{\text{active}}} t\text{.usage} -\end{equation} -If a plugin overrides \texttt{patch} with a unified-diff implementation, its replacement \texttt{.usage} example automatically replaces the foundation documentation in the prompt without touching foundation code. +The event writer flushes records but does not establish immutable storage. A new log normally opens in write mode; resuming a suspended run replaces its final trailer before appending continuation events. ``Append-only'' describes ordinary event accumulation during a run, not an enforced storage property or crash-proof durability. Logs may contain prompts, source text and tool results; confidentiality and retention need deployment policy. -Furthermore, setting \texttt{DISABLE = ['shell', 'write']} prunes both the dispatch registry and the prompt examples, instantly yielding a provably read-only agent. +\section{Omakase: replace the opinion, retain the seam} +NakedAgent's Omakase doctrine offers four selected tools while allowing operators to substitute implementations without replacing the dispatcher. A Python plugin exports \texttt{TOOLS}, \texttt{DISABLE}, or both. Workspace-local plugins require explicit trust; user-global plugins are always considered. Later directories and files override earlier tools, after which disabled names are removed. -\section{Sidekick Harness and Safety Tripwires} -For governed environments, NakedAgent incorporates the \textit{Sidekick Harness} and \textit{JevNoulGate} tripwires: -\begin{itemize} - \item \textbf{Bounded Slot Budgets}: Every autonomous run is assigned a strict maximum turn budget $B_{\text{max}}$. Upon exhaustion, execution transitions immediately to a suspended state without unbounded recursion. - \item \textbf{Tri-State Escalation}: Sensitive operations (such as high-impact shell commands) pass through a tripwire evaluator yielding $\text{ALLOW}$, $\text{DENY}$, or $\text{ESCALATE}$. In an escalate condition, the event log is sealed in a suspended state awaiting human sign-off. - \item \textbf{Zero-Dependency MCP Client}: NakedAgent includes a standard-library JSON-RPC 2.0 client supporting the Model Context Protocol (MCP) over \texttt{stdio}, enabling tool sharing across modern agent meshes without external SDKs. -\end{itemize} +The useful coupling is between registry and prompt: a tool's optional \texttt{.usage} attribute contributes its example, and tools without that attribute are listed by name. Replacing a patch implementation and its usage text updates the advertised interface together. This reduces one source of prompt/dispatch disagreement; it does not check that the example correctly describes the function or that a model follows it. -\section{Exploratory Pilot: Failure Modes of Local Models under Minimal Scaffolding} +Disabling \texttt{shell}, \texttt{write} and \texttt{patch} removes the three mutating \emph{foundation} tools. Disabling only shell and write leaves patch available. Neither configuration proves a whole process read-only: plugin imports execute arbitrary Python and may add effects under any tool name. Dependency minimalism and extension freedom therefore pull in different directions. A deployment must inventory its active tools, imported plugins and external programs separately from the foundation package. -\subsection{Experimental Setup} -To explore the failure dynamics, harness friction, and baseline capabilities of minimal agent architectures under local inference, we conducted an exploratory pilot on SWE-bench Lite~\cite{jimenez2024swebench}. -\begin{itemize} - \item \textbf{Hardware}: NVIDIA GeForce RTX 3080 Ti (12 GB VRAM), Docker Desktop (WSL2), AMD Ryzen CPU. - \item \textbf{Model}: Google \texttt{gemma3:12b} served locally via Ollama with 4-bit quantization. Note that while mini-swe-agent was configured with temperature $T=0$, NakedAgent was evaluated under Ollama's default sampling temperature ($\approx 0.8$), which we note as an uncontrolled exploratory variable. - \item \textbf{Instances}: SWE-bench Lite instances \texttt{astropy\_\_astropy-12907}, \texttt{-14182}, and \texttt{-14365}. - \item \textbf{Baselines}: Mini-SWE-agent 2.4.6 (native tools mode vs. text-based backtick mode). -\end{itemize} +\section{Historical pilot and evidence limits} +\subsection{What survives in the repository} +The September 14, 2026 report \texttt{bench/RESULTS.md} records an exploratory comparison on three SWE-bench Lite instances: astropy\_\_astropy-12907, -14182 and -14365. It reports local \texttt{gemma3:12b}, an RTX 3080 Ti with 12 GB VRAM, Docker Desktop/WSL2, a 30-step cap, 60-second command timeout, and mini-swe-agent 2.4.6. The original run used a worktree with uncommitted modifications; the report does not identify a complete immutable source snapshot, model digest or full sampling configuration. -\subsection{Results and Observations} -Table~\ref{tab:swebench} summarizes the comparative trajectories across configurations. Crucially, across all tested configurations, the true ground-truth resolution rate was \textbf{0/3 resolved (0\%)}. +The repository tracks a runner, scorer and narrative summary. The listed prediction/trajectory directories and raw scoring outputs are absent from the pinned tree. Table~\ref{tab:pilot} therefore reports \emph{source-reported observations}, not reproduced results. No model or SWE-bench run was performed for this revision. A model tag alone does not verify quantization or immutable weights; the earlier manuscript's precise quantization and default-temperature claims are not adopted without a run manifest. \begin{table*}[t] -\centering -\caption{SWE-bench Lite Exploratory Pilot Trajectory Results with Local Gemma 3 12B} -\label{tab:swebench} -\small -\resizebox{\textwidth}{!}{% -\begin{tabular}{lccccc} +\centering\small +\begin{tabular}{p{0.20\textwidth}p{0.48\textwidth}p{0.22\textwidth}} \toprule -\textbf{Configuration} & \textbf{Astropy-12907} & \textbf{Astropy-14182} & \textbf{Astropy-14365} & \textbf{Patches} & \textbf{Resolved} \\ +Historical configuration & Reported observation & Evidential status \\ \midrule -A1: mini-swe-agent (native tools) & BadRequest (No tool API) & BadRequest (No tool API) & BadRequest (No tool API) & 0/3 & 0/3 (0\%) \\ -A2: mini-swe-agent (text backticks) & RepeatedFormatError (9 calls) & LimitsExceeded (30 calls) & RepeatedFormatError (11 calls) & 0/3 & 0/3 (0\%) \\ -B: NakedAgent (bash-only parity) & Stopped after failed \texttt{sed} & Invalid \texttt{sed} regex & Found bug, explained, no edit & 0/3 & 0/3 (0\%) \\ -C1: NakedAgent (system mode, raw) & Invalid (harness bracket bug) & Invalid (harness bracket bug) & Invalid (harness bracket bug) & 0/3 & 0/3 (0\%) \\ -C2: NakedAgent (CRLF text pipe) & Patch applied (whole-file rewrite) & Patch applied (whole-file rewrite) & Patch applied (whole-file rewrite) & 3/3 (invalid CRLF) & 0/3 (0\%) \\ -C3: NakedAgent (binary write fix) & 1-hunk patch (7 regressions) & No patch (nudges exhausted) & No patch (nudges exhausted) & 1/3 valid & 0/3 (0\%) \\ +A1 mini-swe-agent native tools & Provider rejected native tool requests on all three instances & Setup incompatibility; not a reasoning-capability measurement \\ +A2 text backticks & Format failures after 9 and 11 calls; one 30-call limit & No patches reported; no raw trajectories reproduced \\ +B NakedAgent bash-only & Failed edits or explanation without editing; no patches & Descriptive failure analysis; unmatched sampling \\ +C1 system-mode adapter & Placeholder angle brackets copied into shell commands & Invalid harness run; exclude from comparative scoring \\ +C2 placeholder correction & Three patches, but text-pipe CRLF conversion rewrote whole files & Invalid harness output; historical scorer also reports 0/3 \\ +C3 binary-pipe correction & One single-hunk patch; two empty patches; scorer reports two target failures and seven regressions for 12907 & Reported 0/3 resolved; not independently reproduced \\ \bottomrule -\end{tabular}% -} +\end{tabular} +\caption{Historical pilot evidence. These are repeated configurations on the same three selected instances, not eighteen independent tasks or a controlled benchmark.} +\label{tab:pilot} \end{table*} -\subsection{Error Taxonomy: Scaffold vs. Model Bounds} -The pilot revealed critical insights into the interaction between minimal scaffolds and localized language models: -\begin{enumerate} - \item \textbf{Provider-Native Tool Incompatibility (A1)}: Local Ollama serving of \texttt{gemma3:12b} rejected native function calling schemas outright. Scaffolds strictly bound to provider tool APIs failed before execution began. - \item \textbf{Syntax Brittle Failures (A2)}: When forced to write raw bash blocks, small models struggled with command escaping and prompt-format compliance, resulting in \texttt{RepeatedFormatError} loops. - \item \textbf{Placeholder Echoing (C1)}: In early trials, the tool usage prompt included abstract placeholders such as \texttt{}. Gemma 3 literally echoed the angle brackets (\texttt{}), causing bash syntax errors. Replacing prompt placeholders with concrete, literal examples eliminated this error mode. - \item \textbf{Host OS Newline Ingestion (C2)}: Running on Windows host systems communicating with Linux Docker containers resulted in Windows CRLF ($\backslash r \backslash n$) conversions over text pipes, causing git to treat entire files as modified. NakedAgent resolved this by enforcing binary streams across all tool I/O boundaries. - \item \textbf{Localized Patching (C3)}: With binary I/O and concrete usage examples, NakedAgent enabled Gemma 3 to localize and generate clean single-hunk patches on complex Astropy files without external diff packages. - \item \textbf{Syntactic Compliance vs. Semantic Correctness}: Although configuration C3 generated a syntactically valid single-hunk patch on \texttt{astropy-12907}, test-suite evaluation revealed 7 unit test regressions, resulting in 0/3 resolved tasks. Minimal scaffolding effectively removes harness friction, but semantic problem-solving on complex issues remains bounded by local model parameters. -\end{enumerate} - -\section{Security Analysis} +\subsection{Interpretation} +The pilot preserves useful engineering hypotheses: provider capability mismatch can prevent a run; tool examples can be echoed literally; Windows text pipes can contaminate Linux patch bytes; and a localized patch can still fail semantic tests. These observations motivate controlled probes. They do not show that a smaller model's parameters caused the final failure or that minimal scaffolding removes all harness friction. Both the adapter and sampling conditions changed across configurations. In particular, mini-swe-agent used temperature zero while the NakedAgent report says the provider default was used; the numeric value of that default was not recorded. -\subsection{Attack Surface Elimination} -By enforcing $|\mathcal{D}_{\text{runtime}}| = 0$, NakedAgent eliminates the entire attack surface associated with dependency confusion, typosquatting, compromised sub-dependencies, and unpatched upstream CVEs in network libraries. +Invalid C1/C2 outputs must be distinguished from valid unsuccessful attempts. The reported C3 zero of three is descriptive of this small selected case study, not an estimate of general SWE-bench performance. Absence of raw trajectories prevents reclassification of individual failures or verification of the seven-regression count here. -\subsection{Shell Sandboxing and Injection Defense} -Because the shell tool is the most sensitive primitive, NakedAgent implements multi-stage defensive gates: -\begin{itemize} - \item In automated execution, commands must match an approved argv prefix. - \item Arguments are tokenized using POSIX shell lexical parsing (\texttt{shlex.split}), preventing command-chaining injection attacks (such as semicolons \texttt{;}, logical operators \texttt{\&\&}, or command substitutions \texttt{\$(...)}). - \item Path validation routines guarantee that file writes cannot escape designated workspace roots via directory traversal (\texttt{../../}). -\end{itemize} +\subsection{Prospective evaluation} +A comparative study should freeze source and plugin hashes, model/quantization digest, provider version, prompts, decoding parameters, task selection, base commits, container images and scoring version. Repeat matched runs across preregistered task strata and report run-level variation. Keep invalid infrastructure attempts separate from valid attempts; disclose both and a predetermined rerun rule. Save request/response traces, exact patch bytes, exit status and complete grader output. -\subsection{Plugin Gating and Protected Surfaces} -Eliminating third-party dependencies does not eliminate environmental risks from untrusted repositories or ambient shell binaries. To prevent repository-drop code execution (where a cloned project includes malicious code in \texttt{.nakedagent/plugins/}), NakedAgent disables workspace-local plugins by default, requiring an explicit \texttt{--trust-plugins} CLI flag. Furthermore, the \texttt{write} and \texttt{patch} tools enforce protected surface boundaries preventing modifications to \texttt{.git/} or \texttt{.nakedagent/}, and automated production workloads are strictly recommended to execute within containerized sandboxes. +Separate outcomes should include parse acceptance, tool dispatch, patch applicability, target-test improvement, preserved regression tests, final resolution, latency and token/cost budgets. A useful ablation changes one of fenced-vs-native invocation, concrete-vs-placeholder examples, or file-tool-vs-shell editing while holding the other conditions fixed. The historical pilot cannot substitute for those comparisons. -\section{Related Work} -\textbf{Coding Agents}: SWE-agent~\cite{yang2024sweagent} introduced specialized agent-computer interfaces (ACI) tailored for repository navigation. Aider~\cite{gauthier2023aider} popularized in-terminal pair programming with git-anchored search/replace editing. OpenHands~\cite{openhands2024} established a generalist agent runtime platform. NakedAgent distinguishes itself by proving that these core capabilities can be achieved without the 30+ package dependencies characteristic of existing systems. +\section{Security and engineering limitations} +Removing third-party Python runtime packages reduces one set of update and package-selection obligations. It does not eliminate vulnerabilities in the interpreter, TLS stack, executable search path, provider, plugins or project scripts. No dependency-count survey or measured risk reduction is claimed for competing agents. Nor does the existence of a small implementation establish production readiness. -\textbf{Execution Proofs and Governance}: Krineia~\cite{bowlby2026krineia} established formal standards for append-only audit logs in AI governance. NakedAgent implements these principles at the code generation layer through its Merkle-linked Mealy state reducer and zero-inference replay verifier. +The noninteractive shell boundary depends on direct argument-vector execution and the selected executable's semantics, not on tokenization alone. An allowed interpreter or program with execution-capable flags may exercise broad authority. Interactive shell syntax, Windows batch files and environment-dependent executable resolution require separate analysis. Workspace path checks are not a proof against all symlink races, mount changes or filesystem aliases. -\section{Discussion and Limitations} -While standard-library minimalism delivers security and portability, it entails calculated trade-offs: -\begin{itemize} - \item \textbf{Lack of Streaming UI}: Without third-party libraries like \texttt{rich} or \texttt{blessed}, streaming token visualization in the terminal is intentionally rudimentary. - \item \textbf{Network Transport}: Using \texttt{urllib.request} lacks built-in HTTP/2 multiplexing and requires manual connection pooling for high-throughput concurrency. - \item \textbf{Patch Complexity}: The foundation \texttt{patch} tool relies on exact substring matching. While unambiguous, it requires models to reproduce exact surrounding context lines, whereas fuzzy diff engines tolerate minor whitespace variations. -\end{itemize} +Replay also has engineering limits. It verifies one pinned semantics against reported events, not that every event admitted by the reducer could be produced by the live driver. The current genesis does not automatically measure the provider endpoint, binary, plugin source, full shell policy or workspace snapshot. The immutable tuple-based trace/history construction also copies growing sequences; no asymptotic or empirical performance advantage is established. Bounded event counts do not bound individual model responses, file sizes or arbitrary tool costs. -\section{Conclusion and Availability} -NakedAgent demonstrates that a zero-runtime-dependency coding agent is not merely a theoretical curiosity, but a robust, auditable, and production-viable architecture. By relying exclusively on Python standard-library primitives, Markdown code-fence parsing, an Omakase plugin seam, and a Merkle-verified Mealy state reducer, NakedAgent provides an uncompromised foundation for autonomous software engineering. +The contribution remains useful within these limits: one can inspect the effect boundary, substitute a tool together with its prompt example, test the parser without a model and replay a recorded state trajectory without re-executing effects. Those are concrete mechanisms on which a larger evaluation or stronger recorder can build. -\subsection{Availability} -The complete source code, test suite, benchmark harnesses, and replication logs are publicly available: -\begin{itemize} - \item \textbf{GitHub}: \url{https://github.com/hummbl-io/nakedagent} - \item \textbf{PyPI}: \texttt{pip install nakedagent} - \item \textbf{Zenodo Archive}: DOI: \href{https://doi.org/10.5281/zenodo.23001340}{10.5281/zenodo.23001340} -\end{itemize} +\section{Reproducibility and availability} +The source is linked at an exact commit~\cite{nakedagentsource}. The paper README maps each claim to source files, distinguishes frozen artifacts from the living manuscript, and gives build commands. The post-hoc notebook records which checks were actually run. The following checks require no model service: +\begin{lstlisting} +python paper/check_claims.py +python -m examples.replay_demo +python -m unittest discover -s tests +\end{lstlisting} +The first command verifies the pinned runtime source inventory and eight synthetic claim checks. The repository's broader tests include parser, path, budget, log, resume and transport cases; passing them is regression evidence, not exhaustive verification. Rebuilding this paper does not regenerate historical benchmark trajectories or a deposited PDF. A future release should archive the exact manuscript, code snapshot, check outputs and any newly collected experimental data together. \bibliographystyle{plain} \bibliography{references} - \end{document} diff --git a/paper/references.bib b/paper/references.bib index 1ccf123..7424cd0 100644 --- a/paper/references.bib +++ b/paper/references.bib @@ -1,79 +1,47 @@ -@article{jimenez2024swebench, - title={SWE-bench: Can Language Models Resolve Real-world Github Issues?}, - author={Jimenez, Carlos E and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik}, - journal={International Conference on Learning Representations (ICLR)}, - year={2024} +@misc{nakedagentsource, + author = {{HUMMBL}}, + title = {{NakedAgent}: Python reference source, commit 0c96f1a8adac035b9d204ebc78c5eb4175c13685}, + year = {2026}, + howpublished = {\url{https://github.com/hummbl-io/nakedagent/tree/0c96f1a8adac035b9d204ebc78c5eb4175c13685}}, + note = {Source inspected October 10, 2026} +} +@inproceedings{jimenez2024swebench, + title = {{SWE-bench}: Can Language Models Resolve Real-World {GitHub} Issues?}, + author = {Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik}, + booktitle = {International Conference on Learning Representations}, + year = {2024}, + note = {\url{https://arxiv.org/abs/2310.06770}} } - @article{yang2024sweagent, - title={SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering}, - author={Yang, John and Jimenez, Carlos E and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir}, - journal={arXiv preprint arXiv:2405.15793}, - year={2024} -} - -@misc{gauthier2023aider, - author = {Paul Gauthier}, - title = {Aider: AI Pair Programming in Your Terminal}, - year = {2023}, - publisher = {GitHub}, - howpublished = {\url{https://github.com/paul-gauthier/aider}} -} - -@misc{gptme2023, - author = {Erik Bj{\"a}reholt}, - title = {gptme: A Personal AI Assistant in Your Terminal}, - year = {2023}, - publisher = {GitHub}, - howpublished = {\url{https://github.com/ErikBjare/gptme}} -} - -@article{openhands2024, - title={OpenHands: An Open Platform for AI Software Developers as Generalist Agents}, - author={Wang, Xingyao and Li, Boxuan and Song, Yufan and Frank, Frank and others}, - journal={arXiv preprint arXiv:2407.16741}, - year={2024} -} - -@article{yao2022react, - title={ReAct: Synergizing Reasoning and Acting in Language Models}, - author={Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan}, - journal={International Conference on Learning Representations (ICLR)}, - year={2023} -} - -@article{shinn2023reflexion, - title={Reflexion: Language Agents with Verbal Reinforcement Learning}, - author={Shinn, Noah and Cassano, Federico and Berman, Edward and Gopinath, Ashwin and Narasimhan, Karthik and Yao, Shunyu}, - journal={Advances in Neural Information Processing Systems (NeurIPS)}, - year={2023} -} - -@article{schick2023toolformer, - title={Toolformer: Language Models Can Teach Themselves to Use Tools}, - author={Schick, Timo and Dwivedi-Yu, Jane and Dess{\`\i}, Roberto and Raileanu, Roberta and Lomeli, Maria and Zettlemoyer, Luke and Cancedda, Nicola and Scialom, Thomas}, - journal={Advances in Neural Information Processing Systems (NeurIPS)}, - year={2023} -} - -@article{bowlby2026krineia, - title={Append-Only as Proof: A Formal Account of Governance Sovereignty in AI Audit Systems}, - author={Bowlby, Reuben}, - journal={Zenodo Preprint}, - year={2026}, - doi={10.5281/zenodo.21957892} -} - -@article{ohm2020supplychain, - title={Backstabber's Knife Collection: A Review of Open Source Software Supply Chain Attacks}, - author={Ohm, Marc and Plate, Henrik and Sykosch, Arnold and Meier, Michael}, - journal={Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA)}, - year={2020} -} - -@article{ladisa2023supplychain, - title={Risk Taxonomy for Open Source Software Supply Chains}, - author={Ladisa, Piergiorgio and Plate, Henrik and Martinez, Matias and Falleri, Jean-R{\'e}my}, - journal={IEEE Transactions on Software Engineering}, - year={2023} + title = {{SWE-agent}: Agent-Computer Interfaces Enable Automated Software Engineering}, + author = {Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir}, + journal = {arXiv:2405.15793}, + year = {2024}, + note = {\url{https://arxiv.org/abs/2405.15793}} +} +@misc{gptme, + author = {{gptme contributors}}, + title = {gptme: terminal agent project}, + howpublished = {\url{https://github.com/gptme/gptme}}, + note = {Accessed October 10, 2026} +} +@misc{aiderformats, + author = {{Aider contributors}}, + title = {Edit formats}, + howpublished = {\url{https://aider.chat/docs/more/edit-formats.html}}, + note = {Accessed October 10, 2026} +} +@techreport{rfc9162, + author = {Laurie, Ben and Messeri, Emilia and Stradling, Rob}, + title = {Certificate Transparency Version 2.0}, + institution = {RFC Editor}, + number = {RFC 9162}, + year = {2021}, + note = {Experimental protocol. \url{https://www.rfc-editor.org/rfc/rfc9162}} +} +@misc{mcptransport, + author = {{Model Context Protocol contributors}}, + title = {Transports, protocol revision 2024-11-05}, + howpublished = {\url{https://modelcontextprotocol.io/specification/2024-11-05/basic/transports}}, + note = {Accessed October 10, 2026} }