diff --git a/SPEC.md b/SPEC.md index 3d5a735..397161e 100644 --- a/SPEC.md +++ b/SPEC.md @@ -115,7 +115,7 @@ The engine guarantees no graph reload, no atomic engine/package version pair, an - `fkst.codex_runs()` is a read-only bounded observability surface over engine codex run records. It returns running and recent codex runs with `role`, `started_at`, `started_at_ms`, `timeout_seconds`, `lease_expires_at`, `lease_expires_at_ms`, `status` (`running` / `done` / `failed`), bounded `output_tail`, and optional `exit_code`; it does not expose runtime paths or unbounded stdout/stderr. The lease deadline is the codex run's own wall-clock timeout contract, derived as `started_at_ms + timeout_seconds * 1000`. `recent` is the last `50` completed codex runs by record time for bounded observability, not a time-bounded just-finished handoff window. - `fkst.observe([opts]) -> table` is the in-process adapter for generic durable delivery facts. Snapshot mode is the same projection emitted by `fkst-framework observe --json`; its `opts` may contain only `limit` (1..10000), `include = {"queues","errors","events","entities"}`, `since = `, and `page = { section = "dead_letters", after? = }`. Lineage mode is `lineage = { queue = , dept = , source_ref = { kind = , ref = } }` and cannot be combined with snapshot options. The lineage table and nested source reference reject unknown keys and empty required strings; supported kinds are `file`, `file_watch`, `cron`, `git`, and `external`. It returns only `live_delivery` and `terminal_dead_letter` when present, has no `limit` or truncation fields, and selects the earliest live delivery plus latest terminal tombstone in one read transaction through the ordered `(queue, dept, source_ref, record-kind, time, delivery-id)` index. Unsupported or business-shaped keys fail closed. In snapshot mode, `page` and `since` cannot be combined. Without `page`, `since` is applied to `delivery_id` before `limit` truncation, independently for live deliveries and DLQ entries, and request/result serialization remains byte-identical. With `page`, DLQ selection uses durable keyset order `(dead_at_ms, delivery_id)` and a `limit + 1` read; the snapshot adds `page = { section = "dead_letters", next? = }`, where `next` exists only when the lookahead proves another retained row exists. The cursor is versioned and bound to `"dead_letters"`; malformed encoding, shape, version, section, or delivery identity fails closed with `observe dead-letter cursor invalid`. Each call is one read transaction over the current mutable DLQ. Continuation traverses retained rows under that mutable-history model and does not provide cross-request snapshot isolation. The current snapshot contains queue depth state, queue `subscriber_status` (`"current"`, `"absent"`, or `"unknown"`), live delivery entries, DLQ entries, source metadata, limits, and truncation flags; it never emits full payload bodies and does not encode idle, board, audit, skip, workflow, or package-specific semantics. Live owner-process observe may report `"current"` or `"absent"` from the loaded graph; offline database observe reports `"unknown"` because durable rows are not current graph authority. Packages derive those meanings outside the engine. Production reads `FKST_DURABLE_ROOT` and uses the same live-socket/offline projection core as the CLI. - 边界资源必须静态枚举并经 adapter mediation 访问。当前 engine registry 锚点是 `fkst-framework boundary-resources` 与 `crates/fkst-framework/src/boundary_resource.rs`，覆盖 `codex.process`、`shell.process`、`git.process`、`runtime.filesystem` 与 `wall-clock`。可分类的 adapter failure 使用 `error_class`，值域为 `quota-exhausted`、`auth-degraded`、`provider-unavailable`、`provider-throttle`；package 不应从 stderr 文本反推边界状态。 -- `fkst-framework test` 额外注册 test-mode-only `fkst.test` table。除断言与 `run_department` 外，`fkst.test.mock_command(pattern, result)`、`fkst.test.mock_observe(snapshot)`、`fkst.test.with_command_cassette(opts, fn)` 与 `fkst.test.command_calls()` 可劫持测试态外部边界。`mock_command` 劫持 `exec_sync`、codex SDK 与 git SDK 外部命令调用；匹配基于渲染命令行的前缀或子串，mock 按注册顺序一次性消费，未 mock 且无 active cassette 的外部命令 fail closed 且不启动真实进程。`mock_observe(snapshot)` sets the deterministic raw result used by `fkst.observe()` in that test and in any `run_department`/test-runtime child Lua state spawned from it. Snapshot-mode calls still apply `since`, `limit`, `include`, and `page`; lineage-mode calls validate their closed request shape and return the bounded mock result without snapshot transforms. Unmocked test-mode `fkst.observe()` fails closed. Mocked `exec_sync` bypasses read coalescing so command call accounting remains deterministic. `with_command_cassette({ path, mode, redact? }, fn)` 只在 callback 作用域内启用 VCR-style record/replay；`mode` 是 `"record"` 或 `"replay"`，cassette JSON schema 是 `"fkst.test.command-cassette.v1"`，ordered entries 记录 `rendered`、`program`、`args`、`stdin`、optional `cwd`、sorted `env`、`stdout`、`stderr`、`exit_code`。Replay 不启动真实外部进程，并在 boundary mismatch、entry exhausted、unused entries、unsupported schema 或 malformed cassette 时 fail closed。`redact` 用 exact non-empty value replacement 处理 command boundary 与 outputs，默认 replacement 是 `""`。production `run`、`supervise`、`--self-test` 与 conformance 不注册该 mock/cassette state。`setup_worktree` 在 test mode 也经同一 git mock runner，但不模拟 worktree filesystem 副作用。`fkst-framework test --report-json ` 写入 authoritative machine-readable 测试结果与清单，schema 为 `fkst.test.report.v1`，条目字段是 `owner_namespace`、`file`、`name`、`status` 和失败时的 `error`；`owner_namespace`、`file`、`name` 三元组是身份，不提供拼接 `id`。报告由 Rust 侧枚举的 `test_*` key 构造，Lua `print` 不能伪造条目。stdout 的 `PASS` / `FAIL` / summary 行只作为 legacy human / compatibility surface，不是 machine-authoritative。 +- `fkst-framework test` 额外注册 test-mode-only `fkst.test` table。除断言与 `run_department` 外，`fkst.test.mock_command(pattern, result)`、`fkst.test.mock_observe(snapshot)`、`fkst.test.with_command_cassette(opts, fn)` 与 `fkst.test.command_calls()` 可劫持测试态外部边界。`mock_command` 劫持 `exec_sync`、codex SDK 与 git SDK 外部命令调用；匹配基于渲染命令行的前缀或子串，mock 按注册顺序一次性消费，未 mock 且无 active cassette 的外部命令 fail closed 且不启动真实进程。`mock_observe(snapshot)` sets the deterministic raw result used by `fkst.observe()` in that test and in any `run_department`/test-runtime child Lua state spawned from it. Snapshot-mode calls still apply `since`, `limit`, `include`, and `page`; lineage-mode calls validate their closed request shape and return the bounded mock result without snapshot transforms. Unmocked test-mode `fkst.observe()` fails closed. Mocked `exec_sync` bypasses read coalescing so command call accounting remains deterministic. `with_command_cassette({ path, mode, redact? }, fn)` 只在 callback 作用域内启用 VCR-style record/replay；`mode` 是 `"record"` 或 `"replay"`，cassette JSON schema 是 `"fkst.test.command-cassette.v1"`，ordered entries 记录 `rendered`、`program`、`args`、`stdin`、optional `cwd`、sorted `env`、`stdout`、`stderr`、`exit_code`。Replay 不启动真实外部进程，并在 boundary mismatch、entry exhausted、unused entries、unsupported schema 或 malformed cassette 时 fail closed。`redact` 用 exact non-empty value replacement 处理 command boundary 与 outputs，默认 replacement 是 `""`。production `run`、`supervise`、`--self-test` 与 conformance 不注册该 mock/cassette state。`setup_worktree` 在 test mode 也经同一 git mock runner，但不模拟 worktree filesystem 副作用。`fkst-framework test --report-json ` writes the authoritative machine-readable result inventory with schema `fkst.test.report.v1`. Every entry contains `owner_namespace`, `file`, `name`, and `status`. Failed entries also contain the unchanged `error` plus `failure_kind`: `assertion_failure` only for a structurally tagged engine-provided assertion after test-function invocation, and `test_error` for every other test-function error or `` failure. Pass entries omit both fields. The new field is diagnostic and does not change status, counts, continuation, exit codes, or downstream retry policy. The `owner_namespace`, `file`, and `name` tuple is the identity; there is no concatenated `id`. Rust enumerates report entries from `test_*` keys, so Lua `print` cannot forge them. Stdout `PASS`, `FAIL`, and summary lines remain a legacy human and compatibility surface, not the machine-authoritative channel. - `json` 是 decode-only：`json.decode` 暴露 engine 自身 JSON wire format 的解析（event 进、`raise` 出都是 JSON），Lua 值经 `raise` 出引擎，故不提供 `json.encode`；`json` table 锁定为只含 `decode`，新增 encode 或其它 key 必须另走 evidence + conformance。`raise` 不推断 bare Lua empty table 的数组 / 对象意图：裸 `{}` 序列化为 JSON object `{}`；由 `json.decode("[]")` 构造的 array-tagged empty table 经 `raise` 保持为 JSON array `[]`。需要可能为空的数组字段时，package 必须显式用 `json.decode("[]")` 形成 array-tagged table；engine 不提供 `json.array`、不提供 schema / field-name 推断，也不启用全局 empty-table-as-array 开关。 - 新增 SDK 函数必须经 evidence、深度共识与 conformance 覆盖，不能由单个 codex 实例直接扩张。 diff --git a/crates/fkst-framework/src/test_assertions.rs b/crates/fkst-framework/src/test_assertions.rs index 71cd343..8ef4467 100644 --- a/crates/fkst-framework/src/test_assertions.rs +++ b/crates/fkst-framework/src/test_assertions.rs @@ -1,4 +1,22 @@ use mlua::{Function, Lua, Table, Value}; +use std::fmt; + +#[derive(Debug)] +struct AssertionFailure(String); + +impl fmt::Display for AssertionFailure { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(&self.0) + } +} + +impl std::error::Error for AssertionFailure {} + +pub(crate) fn is_assertion_failure(error: &mlua::Error) -> bool { + error + .chain() + .any(|cause| cause.downcast_ref::().is_some()) +} pub(crate) fn register(lua: &Lua, test: &Table) -> mlua::Result<()> { test.set( @@ -73,7 +91,7 @@ fn assertion_error(name: &str, msg: Option, detail: String) -> mlua::Err Some(msg) if !msg.is_empty() => format!("{name}: {msg}: "), _ => format!("{name}: "), }; - mlua::Error::runtime(format!("{prefix}{detail}")) + mlua::Error::external(AssertionFailure(format!("runtime error: {prefix}{detail}"))) } fn display_value(value: Value) -> String { diff --git a/crates/fkst-framework/src/test_runner.rs b/crates/fkst-framework/src/test_runner.rs index 4135071..4a3f4f0 100644 --- a/crates/fkst-framework/src/test_runner.rs +++ b/crates/fkst-framework/src/test_runner.rs @@ -115,12 +115,14 @@ pub(crate) fn run_tests( report.push_pass(&file.owner_namespace, &relpath, &name); } Err(err) => { + let failure_kind = TestReportFailureKind::for_test_error(&err); println!("FAIL {relpath}::{name}: {err}"); failed += 1; report.push_fail( &file.owner_namespace, &relpath, &name, + failure_kind, err.to_string(), ); } @@ -134,6 +136,7 @@ pub(crate) fn run_tests( &file.owner_namespace, &relpath, "", + TestReportFailureKind::TestError, format!("{err:#}"), ); } @@ -198,16 +201,25 @@ impl TestReport { file: file.to_string(), name: name.to_string(), status: TestReportStatus::Pass, + failure_kind: None, error: None, }); } - fn push_fail(&mut self, owner_namespace: &str, file: &str, name: &str, error: String) { + fn push_fail( + &mut self, + owner_namespace: &str, + file: &str, + name: &str, + failure_kind: TestReportFailureKind, + error: String, + ) { self.tests.push(TestReportEntry { owner_namespace: owner_namespace.to_string(), file: file.to_string(), name: name.to_string(), status: TestReportStatus::Fail, + failure_kind: Some(failure_kind), error: Some(error), }); } @@ -226,6 +238,8 @@ struct TestReportEntry { name: String, status: TestReportStatus, #[serde(skip_serializing_if = "Option::is_none")] + failure_kind: Option, + #[serde(skip_serializing_if = "Option::is_none")] error: Option, } @@ -236,6 +250,23 @@ enum TestReportStatus { Fail, } +#[derive(Debug, Serialize)] +#[serde(rename_all = "snake_case")] +enum TestReportFailureKind { + AssertionFailure, + TestError, +} + +impl TestReportFailureKind { + fn for_test_error(error: &mlua::Error) -> Self { + if crate::test_assertions::is_assertion_failure(error) { + Self::AssertionFailure + } else { + Self::TestError + } + } +} + fn write_report_json(path: &Path, report: &TestReport) -> Result<()> { let parent = path .parent() diff --git a/crates/fkst-framework/tests/test_runner_cli.rs b/crates/fkst-framework/tests/test_runner_cli.rs index 7a99963..ac7cac3 100644 --- a/crates/fkst-framework/tests/test_runner_cli.rs +++ b/crates/fkst-framework/tests/test_runner_cli.rs @@ -157,6 +157,15 @@ fn read_report(path: &Path) -> serde_json::Value { serde_json::from_slice(&fs::read(path).unwrap()).unwrap() } +fn read_legacy_report_bytes(path: &Path) -> String { + fs::read_to_string(path) + .unwrap() + .lines() + .filter(|line| !line.trim_start().starts_with("\"failure_kind\":")) + .collect::>() + .join("\n") +} + fn stdout(output: &Output) -> String { String::from_utf8_lossy(&output.stdout).to_string() } @@ -1861,6 +1870,27 @@ return { .any(|test| test["file"] == "forged" || test["name"] == "test_fake"), "report: {report}" ); + let expected = format!( + concat!( + "{{\n", + " \"schema\": \"fkst.test.report.v1\",\n", + " \"summary\": {{\n", + " \"passed\": 1,\n", + " \"failed\": 0\n", + " }},\n", + " \"tests\": [\n", + " {{\n", + " \"owner_namespace\": \"{}\",\n", + " \"file\": \"tests/forgery_test.lua\",\n", + " \"name\": \"test_real\",\n", + " \"status\": \"pass\"\n", + " }}\n", + " ]\n", + "}}" + ), + namespace(host.path()) + ); + assert_eq!(fs::read_to_string(&report_path).unwrap(), expected); } #[test] @@ -1901,13 +1931,125 @@ return { assert_eq!(tests[0]["file"], "tests/failing_report_test.lua"); assert_eq!(tests[0]["name"], "test_fails"); assert_eq!(tests[0]["status"], "fail"); - assert!( - tests[0]["error"] + let expected_error = concat!( + "runtime error: eq: expected \"expected\", got \"actual\"", + "\nstack traceback:", + "\n\t[C]: in field 'eq'", + "\n\ttests/failing_report_test.lua:4: in function \n" + ); + assert_eq!(tests[0]["error"], expected_error, "report: {report}"); + let expected_legacy_report = format!( + concat!( + "{{\n", + " \"schema\": \"fkst.test.report.v1\",\n", + " \"summary\": {{\n", + " \"passed\": 0,\n", + " \"failed\": 1\n", + " }},\n", + " \"tests\": [\n", + " {{\n", + " \"owner_namespace\": \"{}\",\n", + " \"file\": \"tests/failing_report_test.lua\",\n", + " \"name\": \"test_fails\",\n", + " \"status\": \"fail\",\n", + " \"error\": {}\n", + " }}\n", + " ]\n", + "}}" + ), + namespace(host.path()), + serde_json::to_string(expected_error).unwrap() + ); + assert_eq!( + read_legacy_report_bytes(&report_path), + expected_legacy_report + ); +} + +#[test] +fn test_report_json_classifies_failures_by_origin_without_parsing_error_text() { + let host = tempfile::Builder::new().prefix("repo").tempdir().unwrap(); + fs::create_dir_all(host.path().join("tests")).unwrap(); + fs::write( + host.path().join("tests/assertion_test.lua"), + r#" +local t = fkst.test +return { + test_assertion = function() t.eq("actual", "expected") end, +} +"#, + ) + .unwrap(); + fs::write( + host.path().join("tests/runtime_error_test.lua"), + r#" +return { + test_runtime_error = function() + error('eq: expected "expected", got "actual"') + end, +} +"#, + ) + .unwrap(); + for missing_module in ["missing_observation_alpha", "missing_observation_beta"] { + fs::write( + host.path().join(format!("tests/{missing_module}_test.lua")), + format!("return require({missing_module:?})\n"), + ) + .unwrap(); + } + let report_path = host.path().join("report.json"); + + let output = run_lua_tests_with_report(host.path(), host.path(), &report_path); + + assert_eq!( + output.status.code(), + Some(1), + "stdout: {}\nstderr: {}", + stdout(&output), + stderr(&output) + ); + let report = read_report(&report_path); + assert_eq!(report["summary"]["passed"], 0); + assert_eq!(report["summary"]["failed"], 4); + let tests = report["tests"].as_array().unwrap(); + assert_eq!(tests.len(), 4, "report: {report}"); + + let failures = tests + .iter() + .map(|entry| (entry["file"].as_str().unwrap(), entry)) + .collect::>(); + let assertion = failures["tests/assertion_test.lua"]; + assert_eq!(assertion["name"], "test_assertion"); + assert_eq!(assertion["status"], "fail"); + assert_eq!(assertion["failure_kind"], "assertion_failure"); + + let runtime_error = failures["tests/runtime_error_test.lua"]; + assert_eq!(runtime_error["name"], "test_runtime_error"); + assert_eq!(runtime_error["status"], "fail"); + assert_eq!(runtime_error["failure_kind"], "test_error"); + assert!( + runtime_error["error"] .as_str() .unwrap() - .contains("expected \"expected\", got \"actual\""), + .contains("eq: expected \"expected\", got \"actual\""), "report: {report}" ); + + for missing_module in ["missing_observation_alpha", "missing_observation_beta"] { + let file = format!("tests/{missing_module}_test.lua"); + let test_error = failures[file.as_str()]; + assert_eq!(test_error["name"], ""); + assert_eq!(test_error["status"], "fail"); + assert_eq!(test_error["failure_kind"], "test_error"); + assert!( + test_error["error"] + .as_str() + .unwrap() + .contains(missing_module), + "report: {report}" + ); + } } #[test] diff --git a/docs/architecture.md b/docs/architecture.md index cd34688..57affa8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -112,7 +112,7 @@ runner 不全树递归，不扫描 `raisers/` 或 `fkst/`。每个测试文件 } ``` -失败条目额外包含 `error`。加载或 eval 测试文件失败时，报告条目使用 `name = ""` 且计入 failed。报告条目来自 Rust 侧枚举出的测试文件和 `test_*` key；每个条目的身份是 `owner_namespace`、`file`、`name` 三元组，不提供可被分隔符碰撞污染的拼接 `id`。Lua `print` 不能向报告注入伪造测试；stdout 的 `PASS` / `FAIL` / summary 行保留为 legacy human / compatibility surface，不是 authoritative machine channel。 +Failed entries additionally contain `failure_kind` and `error`. `failure_kind = "assertion_failure"` means an engine-provided `fkst.test` assertion structurally marked the error after the test function was invoked. Every other test-function error and every load or eval failure uses `failure_kind = "test_error"`; load or eval entries use `name = ""`. Pass entries omit both fields. This taxonomy records where the failure originated and does not decide retryability. It does not change `status`, `error`, summary counts, test continuation, or exit codes. Report entries come from test files and `test_*` keys enumerated by Rust. Each entry is identified by the `owner_namespace`, `file`, and `name` tuple and has no delimiter-sensitive concatenated `id`. Lua `print` cannot inject report entries. The stdout `PASS`, `FAIL`, and summary lines remain a legacy human and compatibility surface, not the authoritative machine channel. `--coverage ` is an opt-in engine-owned Lua line coverage mode for the test runner. It installs an `mlua` `HookTriggers::EVERY_LINE` hook only for the covered run, names engine-loaded file chunks as `@`, and writes `/coverage.json` plus `/lcov.info` after the full run. `coverage.json` has the shape `{ "": { "covered_lines": [1, 2] } }`; `lcov.info` emits only `TN`, `SF`, `DA`, and `end_of_record` records. This is honest line coverage only: branch, condition, and mutation evidence are outside this surface. Files ending in `*_test.lua` and generated chunks named `=fkst:` are excluded from production coverage. Hooks are also applied to threads created through the standard `coroutine.create` table during coverage runs. Without `--coverage`, no hook is installed and the runner keeps the normal zero-coverage-overhead path.