Repository navigation
feat(engine): emergent validate, --log-stdout, and WAL for the event store - #92
Conversation
rrrodzilla
left a comment
There was a problem hiding this comment.
The shared preflight design is sound, and the WAL concurrency test verifies the reader/writer behavior directly. Please address the backup guidance and diagnostic issues below before merging.
Verified locally at bb5520e:
- cargo nextest run -p emergent-engine: 399 passed, 1 skipped.
- cargo clippy --workspace --all-targets -- -D warnings: passed.
- cargo fmt --all --check: passed.
The diagnostic findings below were reproduced using the built CLI.
|
|
||
| Both paths support `"auto"` for XDG data directory placement. | ||
|
|
||
| The SQLite database is opened in write-ahead-log mode (`journal_mode = WAL`, `synchronous = NORMAL`), so another process can read it while the engine runs, whether `sqlite3` or a dashboard tailing new events, without making the engine's inserts fail with `database is locked`. A reader sees the last committed events. SQLite keeps `events.db-wal` and `events.db-shm` next to the database while it is open; copy all three, or stop the engine, to back it up. `NORMAL` can lose the last few events on a power failure or OS crash, never on an engine crash, and never corrupts the file. |
There was a problem hiding this comment.
P2: Use a consistent backup procedure for a live database
Copying the database, WAL, and SHM files separately while the engine is writing does not guarantee a consistent snapshot. A write or checkpoint between copies can leave the backup with mismatched contents and potentially corrupt it.
Please recommend SQLite's backup API (including the CLI .backup command), VACUUM INTO, or stopping writes before copying. SQLite documents this risk and the safe alternatives here: https://sqlite.org/howtocorrupt.html#backup_or_restore_while_a_transaction_is_active
There was a problem hiding this comment.
Replaced the "copy all three" advice with consistent-snapshot guidance: sqlite3 events.db ".backup ..." (the online backup API), VACUUM INTO, or stop the engine before copying, with a link to the SQLite page you cited. This was the only place in docs/ and skills/ that gave backup advice. Fixed in d2bbcc1.
🤖 Generated with Claude Code
| let mut counts: std::collections::HashMap<String, usize> = std::collections::HashMap::new(); | ||
| let mut current: Option<String> = None; | ||
|
|
||
| for line in before.lines() { |
There was a problem hiding this comment.
P2: Derive diagnostic paths from TOML structure rather than scanning header-like lines
This scan counts table headers inside multiline strings as real tables. The following config has two sinks, but validate --json reports the unknown field at sinks[2].publishes, a nonexistent third sink, instead of sinks[1].publishes:
[[sinks]]
name = "first"
path = "/bin/sh"
args = ["""
[[sinks]]
"""]
[[sinks]]
name = "second"
path = "/bin/sh"
publishes = ["x"]This makes the machine-readable location unreliable for a control plane or editor. Please derive the path from parser-aware structure and add coverage for multiline strings. Quoted table names also need normalization: [["sinks"]] currently produces a path containing literal quotes around sinks.
There was a problem hiding this comment.
table_path_at now walks toml_edit's parsed document and picks the header table that starts last at or before the error offset, so a [[sinks]] inside a multiline string is not a table and [["sinks"]] becomes sinks[0]. When the text does not parse, path is omitted. Your example now reports sinks[1].publishes, and there are tests for it, for the quoted name, and for the ordinary case. This adds toml_edit as a direct dependency; toml already pulls it in at the same version, so Cargo.lock still has one copy. Fixed in e459834.
🤖 Generated with Claude Code
|
|
||
| /// Every `[engine].api_allowed_hosts` problem (pure function). | ||
| fn allowed_host_issues(&self) -> Vec<ConfigIssue> { | ||
| crate::api_host::parse_allowed_hosts(&self.engine.api_allowed_hosts) |
There was a problem hiding this comment.
P3: Collect each invalid allowed-host entry
parse_allowed_hosts returns immediately on the first invalid value, so this wrapper does not fulfill the new validation command's promise to report every semantic error in one run.
Reproduced with:
[engine]
api_allowed_hosts = ["https://a.example", "https://b.example"]The JSON report contains only the error for https://a.example. Please validate each entry and report its index, such as engine.api_allowed_hosts[0] and engine.api_allowed_hosts[1], while preserving startup's first-error behavior.
There was a problem hiding this comment.
validate now reports one invalid_api_allowed_host per bad entry at engine.api_allowed_hosts[i], using a new per-entry api_host::parse_allowed_host that parse_allowed_hosts also calls. Startup still refuses on the first bad entry, and a test covers both behaviors. Fixed in 58fe240.
🤖 Generated with Claude Code
…ent store - `emergent validate [--json] [--skip-path-check]` runs the engine's own startup checks (schema, names, restart policies, topics, api_allowed_hosts, primitive paths, IPC connection capacity) without starting anything, and reports every problem with a stable code and config path. Startup and validate share one code path (`preflight`). - `--log-stdout` / `EMERGENT_LOG_STDOUT=1` sends info-level logs to stdout for containers; the default and `--verbose` are unchanged. - The SQLite event store opens with journal_mode=WAL and synchronous=NORMAL, so an external reader no longer makes inserts fail with SQLITE_BUSY. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EuTzw5ZEExrSJe2GDkYr1F
`emergent validate` now reports one invalid_api_allowed_host issue per bad entry, at `engine.api_allowed_hosts[i]`, instead of stopping at the first. The per-entry decision is `api_host::parse_allowed_host`, which `parse_allowed_hosts` now calls; startup still refuses on the first error. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EuTzw5ZEExrSJe2GDkYr1F
…ture `table_path_at` scanned header-like lines, so a `[[sinks]]` inside a multiline string counted as a table and `[["sinks"]]` kept its quotes. It now walks `toml_edit`'s parsed document and picks the header table that starts last at or before the error offset, so strings are not tables and quoted names are unquoted. When the text does not parse, `path` is omitted. `toml_edit` becomes a direct dependency at the version `toml` already pulls in, so the lockfile still holds one copy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EuTzw5ZEExrSJe2GDkYr1F
Copying `events.db`, `-wal` and `-shm` one by one while the engine writes can capture a commit or checkpoint between copies and yield a corrupt backup. Recommend `sqlite3 .backup` (the online backup API), `VACUUM INTO`, or stopping the engine before copying. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EuTzw5ZEExrSJe2GDkYr1F
bb5520e to
d2bbcc1
Compare
|
Rebased onto main at v0.15.0 (acton-reactive 10). The one conflict was in main.rs: upstream added 🤖 Generated with Claude Code |
rrrodzilla
left a comment
There was a problem hiding this comment.
Re-reviewed at d2bbcc1. All three previous findings are resolved:
- Live SQLite backup guidance now recommends a consistent snapshot through
.backuporVACUUM INTO, or stopping the engine before copying. - Diagnostic paths come from parsed TOML structure. The multiline-string example reports
sinks[1].publishes, and quoted table names reportsinks[0].publishes. - Every invalid allowed-host entry is reported at its own index, while startup retains first-error behavior.
The rebase onto v0.15.0 preserves the issue #91 idle-read override, separate admission deadline, and regression tests. I found no additional blocking issues in the updated diff.
Verified locally:
cargo nextest run -p emergent-engine: 407 passed, 1 skipped.cargo clippy --workspace --all-targets -- -D warnings: passed.cargo fmt --all --check: passed.- Direct CLI checks for multiline strings, quoted table names, and multiple invalid hosts: passed.
feat(engine):
emergent validate,--log-stdout, and WAL for the event storeThree changes that make the engine easier to run under something else: a CI job, a control plane, or a container runtime.
1.
emergent validateChecks a config with the engine's own startup checks and starts nothing: no socket, no event store, no log file, no primitives.
It runs the checks startup runs, in the same order:
deny_unknown_fieldsmakes an unknown key an error[engine].api_allowed_hostspathexists (--skip-path-checkturns this one off, for a host that validates configs whose primitives are installed somewhere else)RESERVED_IPC_CONNECTIONSagainst the effective limit, where[engine].max_connectionstakes precedence over acton'sipc.toml, which takes precedence over acton's defaultIt reports every problem, not only the first, and exits
0when the engine would start and1when it would not.--jsonprints a single object on stdout. Log lines go to stderr, so stdout parses as JSON.{ "ok": false, "engine_version": "0.14.0", "errors": [ { "code": "unknown_field", "message": "unknown field `publishes` (line 540, column 1)", "path": "sinks[0].publishes" } ], "warnings": [] }The possible
codevalues areconfig_not_found,read_error,parse_error,unknown_field,invalid_name,duplicate_name,invalid_restart_policy,invalid_subscription_topic,invalid_api_allowed_host,path_not_foundandconnection_capacity. There is also one warning,wire_format_ignored.pathgives the location in the file (sinks[0]is the first[[sinks]]table). It is left out when there is nothing to point at.Startup and
validateshare one code pathconfig.rsnow returns a locatedConfigIssue(code, message, config path) instead of returning early.EmergentConfig::issues(PathCheck)collects them all.preflight::preflight(config, max_connections, PathCheck)adds the capacity check and the warnings.EmergentConfig::read(parse and resolve, no validation), thenpreflight, and refuses on the first error.validatecalls the same functions and prints everything they return.EmergentConfig::load,parseandvalidate()behave as before and return the sameConfigErrorvariants with the same messages, becausevalidate()is now "first issue fromissues()".create_ipc_configmoved topreflight::ipc_config, so startup andvalidateresolve the connection limit the same way.As a side effect, startup now runs the capacity check before it creates the event store or checks for a stale socket. Before, the check ran after both. When there are several errors, the startup error also says how many more there are and points at
emergent validate.2.
--log-stdout/EMERGENT_LOG_STDOUT=1Without
--verbose, the engine logs to~/.local/share/emergent/<name>/emergent.log. Inside a container that file is invisible todocker logsand is lost along with the container.--log-stdout, orEMERGENT_LOG_STDOUTset to1,true,yesoron, sends the same info-level lines to stdout. ANSI colour is used only when stdout is a TTY. The default and--verboseare unchanged, and--verbosewins if both are given. The decision is a pure function,logging::log_destination.3. WAL for the SQLite event store
SqliteEventStore::newnow setsPRAGMA journal_mode = WALandPRAGMA synchronous = NORMAL. In rollback-journal mode, any reader ofevents.db(sqlite3, a dashboard, a control plane tailing events) held a shared lock, and the engine's inserts failed withdatabase is lockedwhile that lock was held. In WAL mode readers work from a snapshot and never block the writer.NORMALis SQLite's recommended setting with WAL:BufWriterand is not fsynced per event.flush()already ranPRAGMA wal_checkpoint(TRUNCATE), so the shutdown path needs no change.Tests
preflight.rsunit tests cover:validate()returns--skip-path-checkremoving only the path checkwire_formatproducing a warningsinks[1].publishes, line and column)logging.rsunit tests cover the default,--verbose, the flag, and truthy and falsey env values.sqlite.rstests check that:journal_modeiswalandsynchronousis1store(), and it keeps its snapshot until it commitsBoth tests fail without the change: the second one fails with
SQLITE_BUSY.A new
tests/validate_cli.rsdrives the real binary. It checks:unknown_field,path_not_foundwith and without--skip-path-check,connection_capacity, andconfig_not_found--log-stdout) refuses a config over capacity through the same check, before creating any data directory or socketcargo fmt --all --check,cargo clippy --workspace --all-targetsandcargo test --workspaceall pass.Docs
docs/configuration.md: a Validation subsection forvalidate, a new Logging section, and a note on WAL and backups in the event-store sectionskills/emergent/references/configuration.md: the new flags and the WAL notedocker/README.md:EMERGENT_LOG_STDOUT, andvalidateunder troubleshootingCLAUDE.md: the new modules, and a rule that new pre-spawn checks go inissues/preflight🤖 Generated with Claude Code
https://claude.ai/code/session_01EuTzw5ZEExrSJe2GDkYr1F