Improve VS Code run arguments and agent docs

AGENTS.md

@@ -126,6 +126,33 @@ The extension auto-discovers a Python interpreter for the MCP server in this ord
 The MCP client is reset whenever `stataCode.*` settings change so users don't have
 to reload the window.
 
+## Agent-ergonomics surface (0.11)
+
+Five knobs and two tools were added in 0.11 in response to a report from an
+agent that drove a full empirical paper through the MCP server. If you are
+touching `runner.execute()` or the `stata_run` schema, these are the invariants
+to preserve:
+
+| Surface | Invariant that must not regress |
+| --- | --- |
+| `include_results` (default `scalars`) | `results.estimation` is **always** built from the complete matrix values — following `matrix://` refs when needed — *before* the wire representation is trimmed. Trimming the payload must never blank `se` / `p_value` / CI. See `_collect_results` in `core/runner.py`. |
+| `include_estimation`, `max_coefficients` | `estimation.n_coefficients` always reports the model's true term count; `coefficients_truncated` flags any cut. A trimmed table must never be indistinguishable from a smaller model. |
+| `timeout_ms` | Budgets the **whole** call including the wait for the session's worker lock. Acquiring that lock without a deadline is the bug that made queued calls hang forever (`WorkerProcess.execute`). Contention returns `rc=-5` / `session_busy` and must **not** kill the healthy worker. |
+| `run_in_background` | Jobs live in `core/jobs.py`. Running jobs are never evicted from the registry; a job's `status` is published *after* its result, so a reader seeing a terminal status also sees the payload. |
+| `auto_close_logs` | Only handles opened by **this** run are closed. A handle opened by an earlier run belongs to the caller. |
+| `track_output_files` | Detection is a working-dir snapshot diff, capped at `MAX_SNAPSHOT_ENTRIES`; past the cap it must skip and warn rather than report a partial answer as complete. |
+
+Two cross-cutting rules that are easy to break:
+
+- **Stata missings are not numbers.** `sfi` returns `.` and `.a`–`.z` as doubles
+  at or above `2**1023`. Every numeric crossing the wire goes through
+  `_norm_stata_number`. Adding a new numeric field means routing it through that
+  helper too.
+- **Internal probes are r-class.** `graph dir`, `log query` and `graph export`
+  all clobber `r()`. Any new housekeeping command must run inside
+  `_preserved_returns(rt)`, or the caller's `summarize` → `display r(mean)`
+  pattern breaks across calls.
+
 ## Notebook cell repair loop (Phase 1)
 
 `stata_run` is intentionally cell-agnostic: it accepts a code string and a few
@@ -148,6 +175,9 @@ Recommended loop (opt-in; never run without an explicit "fix and rerun" request)
 4. On failure:
    - `error.line` is **already cell-relative** because the agent submitted the
      cell's source verbatim — no off-by-one math against the notebook file.
+     (`error.source_file` is `null` in this case. It is only populated when the
+     cell itself invoked a `do`/`run` script, and then `line` indexes into that
+     script rather than into the cell.)
    - `error.context.failing` is the failing command line; use it as a content
      fingerprint when the user describes the failure later.
 5. If the user authorised repairs, apply the edit via

README.en.md

@@ -81,7 +81,7 @@ Then just ask:
 
 A fourth frontend, a **plain-terminal CLI** (`stata-code run` / `lint` / `setup`), gives any agent that can shell out — or a bare terminal — the same typed `RunResult`. And the core runs two backends: **pystata** (Stata 17+, in-memory sessions) or a **console backend** (Stata 13+ in batch mode, no pystata) — both returning the identical schema.
 
-**Status: v0.10 (July 2026)** — the core, MCP server, Jupyter kernel, VS Code extension, and CLI work end-to-end against Stata 18 MP; the console backend broadens coverage to Stata 13+ without pystata. The test suite covers schema, runner, console parser, MCP, kernel, notebook, run-index, subprocess-pool, command policy, linter, and VS Code modules; CI also checks linting, type safety, schema generation, package metadata, and VSIX packaging. License: **MIT**.
+**Status: v0.11 (July 2026)** — the core, MCP server, Jupyter kernel, VS Code extension, and CLI work end-to-end against Stata 18 MP; the console backend broadens coverage to Stata 13+ without pystata. v0.11 is an agent-ergonomics release: bounded result payloads, background execution for multi-minute jobs, error localization inside `do`-files, automatic log-handle cleanup, and generated-file reporting — see the [changelog](CHANGELOG.md). The test suite covers schema, runner, console parser, MCP, kernel, notebook, run-index, subprocess-pool, command policy, linter, and VS Code modules; CI also checks linting, type safety, schema generation, package metadata, and VSIX packaging. License: **MIT**.
 
 Four workflows the current tree explicitly supports for end users and agents:
 
@@ -474,11 +474,14 @@ The extension still requires the MCP extra on your system Python (`pip install "
 
 ## Token-Economy Defaults
 
-A typical `stata_run` response is about **10x smaller** than servers that dump logs and images directly. Three design choices drive this:
+A typical `stata_run` response is about **10x smaller** than servers that dump logs and images directly. Four design choices drive this:
 
-1. **Logs return `head` + `tail` + `ref`** by default. Full logs are fetched on demand via `get_log(ref)`. A Stata regression log can be about 6,000 tokens; `stata-code` returns about 600 by default.
-2. **Graphs return refs, not inline base64**. A 30 KB PNG can become about 50,000 base64 tokens; returning a ref avoids that unless the agent actually needs the bytes.
-3. **Errors are typed**. Agents can check `err.kind == "varname_not_found"` instead of regex-parsing English logs.
+1. **Logs return `head` + `tail` + `ref`** by default. Full logs are fetched on demand via `get_log(ref)`, or grepped in place with `search_log(ref, pattern)`. A Stata regression log can be about 6,000 tokens; `stata-code` returns about 600 by default.
+2. **Graphs return refs, not inline base64**. A 30 KB PNG can become about 50,000 base64 tokens; returning a ref avoids that unless the agent actually needs the bytes. When they are requested inline, they come back as real MCP image content blocks — viewable by a vision-capable client, rather than base64 buried in a JSON string that costs tokens and shows nothing.
+3. **Each estimation is described once.** By default (`include_results: "scalars"`) `r()` / `e()` scalars and macros are inline, while every matrix becomes a `matrix://` stub carrying just its shape. Without this, one regression ships the same numbers four times — `e(b)`, `e(V)`'s label lists, `e(beta)`, `r(table)` — on top of `results.estimation`, which already holds the typed coefficient table. A 123-term regression drops from ~57 KB to ~28 KB. Raw values stay one `get_matrix(ref)` away, and `include_results: "full"` restores the old shape.
+4. **Errors are typed**. Agents can check `err.kind == "varname_not_found"` instead of regex-parsing English logs.
+
+Two further knobs matter for fixed-effect-heavy work: `include_estimation: "summary"` keeps the model-level block and drops per-term rows, and `max_coefficients` caps the table. Either way `estimation.n_coefficients` reports the model's true size and `coefficients_truncated` flags the cut, so a trimmed table is never mistaken for a smaller model.
 
 For example, a misspelled variable returns a structured error:
 
@@ -490,18 +493,27 @@ For example, a misspelled variable returns a structured error:
     "kind": "varname_not_found",
     "varname": "mpgg",
     "line": 3,
+    "source_file": null,
     "context": {
       "before": ["use auto"],
       "failing": "summarize mpgg",
       "after": []
     },
     "suggestions": [
       {"action": "Did you mean `mpg`?", "command": "describe"}
-    ]
+    ],
+    "recovery": {
+      "category": "user_code",
+      "retriable": false,
+      "needs_code_change": true,
+      "needs_user_input": false
+    }
   }
 }
 ```
 
+When the failing command lives inside a script you invoked (`do "analysis.do"`), `line` and `context` point *inside that script* and `source_file` names it — so an agent does not have to re-read the file to find the offending line.
+
 The full schema is in [SCHEMA.md](SCHEMA.md).
 
 ---
@@ -551,7 +563,7 @@ stata_code/
 
 `stata-code` owns the **agent-native, typed-contract** lane: one structured
 `RunResult` schema across MCP, Jupyter, VS Code, and a plain-terminal CLI; a
-32-kind error taxonomy with recovery contracts; token-economy refs; and now a
+34-kind error taxonomy with recovery contracts; token-economy refs; and now a
 console backend (Stata 13+) plus a zero-Python binary. Editor-first tools like
 `stata-all-in-one` lead on human IDE polish and hand the agent raw log text;
 `stata-code` matches their onboarding/version reach while keeping the typed
@@ -572,16 +584,20 @@ teardown.
 - Graph capture: `png` / `svg` / `pdf` with ref store and source-command attribution
 - Log truncation with ref store
 - Warning extraction: 5 categories + generic notes
-- 32-kind error taxonomy with canonical suggestions
+- 34-kind error taxonomy with canonical suggestions and a machine-readable `recovery` verdict (retriable / needs-code-change / needs-user-input)
 - MCP server: 21 tools, including notebook navigation / search / atomic edits, the run-bundle index (`list_runs`), log grep (`search_log`), dataset inspection (`inspect_data`), static linting (`lint_do`), and package installation (`install_package`)
 - Command-safety guard: OS-escape / file-deletion commands (`shell`, `winexec`, `erase`, `rm`, `rmdir`, `!`) are blocked before Stata runs; configurable via `STATA_CODE_COMMAND_POLICY` / `STATA_CODE_POLICY_ALLOW` / `STATA_CODE_POLICY_BLOCK`
 - Bash / plain-terminal surface: `stata-code run` (a `.do` file, `-e` snippets, or stdin) prints the same structured `RunResult` any agent that can shell out can consume; `stata-code lint` runs the linter; `stata-code setup` writes MCP client configs
 - Console (batch) backend (`core/console.py`, `--backend console`, `run_console()`): drives the Stata command-line executable, parses the log into the same typed `RunResult`, and supports **Stata 13+ with no pystata**
 - Zero-Python standalone binary ([`scripts/build_standalone.py`](scripts/build_standalone.py) + CI workflow template [`packaging/standalone.github-workflow.yml`](packaging/standalone.github-workflow.yml)); with `--backend console` it is a fully Python-free path to typed results
 - One-click VS Code onboarding: the extension offers to create a workspace `.venv` and install the server (command palette: “Stata: Set Up MCP Server”)
 - Jupyter kernel: rewired to the v1.0 pipeline, kernel logos bundled
-- Matrix size cap + `get_matrix(ref)` for large matrices (>10k cells)
-- Subprocess-backed hard timeout and cancellation for the public Python API and MCP server: `timeout_ms`, `cancel(session_id)`, and MCP `cancel_session`
+- Result-payload budget: `include_results` (matrices become `matrix://` stubs by default), `include_estimation`, and `max_coefficients`, with `get_matrix(ref)` to pull raw values on demand
+- Subprocess-backed hard timeout and cancellation for the public Python API and MCP server: `timeout_ms` (now an advertised `stata_run` argument, and it budgets queueing too — contention returns `rc=-5` / `session_busy` instead of blocking), `cancel(session_id)`, and MCP `cancel_session`
+- Background execution for multi-minute jobs: `run_in_background` returns a job id; `stata_run_status` (with a bounded `wait_ms`) and `list_background_runs` poll it
+- Error localization inside `do`/`run` scripts: `error.line` + `error.context` resolve within the invoked file and `error.source_file` names it; a failed run still carries a full searchable log
+- Log-handle hygiene: handles leaked by a failed run are closed automatically (`auto_close_logs`), so an aborted script cannot make every later run in that session fail with r(604)
+- Generated-file reporting: `result.outputs` lists the tables, exports and datasets each run wrote, independent of the run-bundle options
 - Per-cell repair loop on `.ipynb` via `notebook_outline` / `notebook_get_cell` / `notebook_edit_cell` with optimistic-concurrency `expected_source` guards and `origin_cell_id` echo on `RunResult`
 - Persistent run bundles + `list_runs` query over `manifest.json` files (filter by cell / origin / session / since / ok; page with limit / offset)
 - Read-only `stata-code doctor` / `verify` diagnostics for package version,
@@ -596,7 +612,7 @@ teardown.
 
 ### Next Up
 
-- Streaming / progress for long runs (`log.complete:false`, incremental log lines) so 20-minute `boottest` / `csdid` jobs report before they finish
+- Streaming / incremental progress for long runs (`log.complete:false`, partial log lines). v0.11's `run_in_background` already unblocks the caller for 20-minute `boottest` / `csdid` jobs, but a running job still reports nothing until it finishes
 - Hard timeout / cancellation for the Jupyter kernel (move it from the direct in-process runner to the subprocess pool, or an equivalent)
 - Console backend: graph capture and richer matrix coverage (values currently materialized for the estimation matrices; state is per-call)
 - Human IDE polish to match editor-first tools: inline graph rendering + DPI export, a scalable data viewer, and an optional "attach to a running Stata" backend