Skip to content

feat(engine): emergent validate, --log-stdout, and WAL for the event store - #92

Merged
rrrodzilla merged 4 commits into
Govcraft:mainfrom
timaxorum:feat/validate-and-container-logging
Sep 30, 2026
Merged

rrrodzilla merged 4 commits into
Govcraft:mainfrom
timaxorum:feat/validate-and-container-logging

Conversation

@timaxorum

Copy link
Copy Markdown
Contributor

feat(engine): emergent validate, --log-stdout, and WAL for the event store

Three changes that make the engine easier to run under something else: a CI job, a control plane, or a container runtime.

1. emergent validate

Checks a config with the engine's own startup checks and starts nothing: no socket, no event store, no log file, no primitives.

emergent validate --config ./emergent.toml
emergent validate --config ./emergent.toml --json
emergent validate --config ./emergent.toml --json --skip-path-check

It runs the checks startup runs, in the same order:

  • the TOML schema, where deny_unknown_fields makes an unknown key an error
  • primitive names and duplicate names
  • restart policies
  • subscription topics
  • [engine].api_allowed_hosts
  • that each enabled primitive's path exists (--skip-path-check turns this one off, for a host that validates configs whose primitives are installed somewhere else)
  • the IPC connection-capacity check: enabled primitives + RESERVED_IPC_CONNECTIONS against the effective limit, where [engine].max_connections takes precedence over acton's ipc.toml, which takes precedence over acton's default

It reports every problem, not only the first, and exits 0 when the engine would start and 1 when it would not. --json prints a single object on stdout. Log lines go to stderr, so stdout parses as JSON.

{
  "ok": false,
  "engine_version": "0.14.0",
  "errors": [
    { "code": "unknown_field",
      "message": "unknown field `publishes` (line 540, column 1)",
      "path": "sinks[0].publishes" }
  ],
  "warnings": []
}

The possible code values are config_not_found, read_error, parse_error, unknown_field, invalid_name, duplicate_name, invalid_restart_policy, invalid_subscription_topic, invalid_api_allowed_host, path_not_found and connection_capacity. There is also one warning, wire_format_ignored. path gives the location in the file (sinks[0] is the first [[sinks]] table). It is left out when there is nothing to point at.

Startup and validate share one code path

  • Each rule in config.rs now returns a located ConfigIssue (code, message, config path) instead of returning early.
  • EmergentConfig::issues(PathCheck) collects them all.
  • The new preflight::preflight(config, max_connections, PathCheck) adds the capacity check and the warnings.
  • Startup does EmergentConfig::read (parse and resolve, no validation), then preflight, and refuses on the first error. validate calls the same functions and prints everything they return.
  • EmergentConfig::load, parse and validate() behave as before and return the same ConfigError variants with the same messages, because validate() is now "first issue from issues()".
  • create_ipc_config moved to preflight::ipc_config, so startup and validate resolve the connection limit the same way.

As a side effect, startup now runs the capacity check before it creates the event store or checks for a stale socket. Before, the check ran after both. When there are several errors, the startup error also says how many more there are and points at emergent validate.

2. --log-stdout / EMERGENT_LOG_STDOUT=1

Without --verbose, the engine logs to ~/.local/share/emergent/<name>/emergent.log. Inside a container that file is invisible to docker logs and is lost along with the container. --log-stdout, or EMERGENT_LOG_STDOUT set to 1, true, yes or on, sends the same info-level lines to stdout. ANSI colour is used only when stdout is a TTY. The default and --verbose are unchanged, and --verbose wins if both are given. The decision is a pure function, logging::log_destination.

3. WAL for the SQLite event store

SqliteEventStore::new now sets PRAGMA journal_mode = WAL and PRAGMA synchronous = NORMAL. In rollback-journal mode, any reader of events.db (sqlite3, a dashboard, a control plane tailing events) held a shared lock, and the engine's inserts failed with database is locked while that lock was held. In WAL mode readers work from a snapshot and never block the writer.

NORMAL is SQLite's recommended setting with WAL:

  • It never corrupts the database.
  • It can lose the last few commits on a power loss or OS crash, but not when the engine itself crashes.
  • That is the durability the engine already offers: the JSON log goes through a BufWriter and is not fsynced per event.

flush() already ran PRAGMA wal_checkpoint(TRUNCATE), so the shutdown path needs no change.

Tests

  • preflight.rs unit tests cover:

    • every error being reported, with codes and paths
    • the first error matching what validate() returns
    • --skip-path-check removing only the path check
    • capacity against the limit
    • wire_format producing a warning
    • unknown-key location (sinks[1].publishes, line and column)
    • table-path resolution
    • the JSON shape
  • logging.rs unit tests cover the default, --verbose, the flag, and truthy and falsey env values.

  • sqlite.rs tests check that:

    • journal_mode is wal and synchronous is 1
    • an external connection holding an open read transaction does not block store(), and it keeps its snapshot until it commits

    Both tests fail without the change: the second one fails with SQLITE_BUSY.

  • A new tests/validate_cli.rs drives the real binary. It checks:

    • exit codes, and that stdout is JSON only
    • unknown_field, path_not_found with and without --skip-path-check, connection_capacity, and config_not_found
    • the human-readable output
    • that startup (with --log-stdout) refuses a config over capacity through the same check, before creating any data directory or socket

cargo fmt --all --check, cargo clippy --workspace --all-targets and cargo test --workspace all pass.

Docs

  • docs/configuration.md: a Validation subsection for validate, a new Logging section, and a note on WAL and backups in the event-store section
  • skills/emergent/references/configuration.md: the new flags and the WAL note
  • docker/README.md: EMERGENT_LOG_STDOUT, and validate under troubleshooting
  • CLAUDE.md: the new modules, and a rule that new pre-spawn checks go in issues/preflight

🤖 Generated with Claude Code

https://claude.ai/code/session_01EuTzw5ZEExrSJe2GDkYr1F

@rrrodzilla rrrodzilla left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared preflight design is sound, and the WAL concurrency test verifies the reader/writer behavior directly. Please address the backup guidance and diagnostic issues below before merging.

Verified locally at bb5520e:

  • cargo nextest run -p emergent-engine: 399 passed, 1 skipped.
  • cargo clippy --workspace --all-targets -- -D warnings: passed.
  • cargo fmt --all --check: passed.

The diagnostic findings below were reproduced using the built CLI.

Comment thread docs/configuration.md Outdated

Both paths support `"auto"` for XDG data directory placement.

The SQLite database is opened in write-ahead-log mode (`journal_mode = WAL`, `synchronous = NORMAL`), so another process can read it while the engine runs, whether `sqlite3` or a dashboard tailing new events, without making the engine's inserts fail with `database is locked`. A reader sees the last committed events. SQLite keeps `events.db-wal` and `events.db-shm` next to the database while it is open; copy all three, or stop the engine, to back it up. `NORMAL` can lose the last few events on a power failure or OS crash, never on an engine crash, and never corrupts the file.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Use a consistent backup procedure for a live database

Copying the database, WAL, and SHM files separately while the engine is writing does not guarantee a consistent snapshot. A write or checkpoint between copies can leave the backup with mismatched contents and potentially corrupt it.

Please recommend SQLite's backup API (including the CLI .backup command), VACUUM INTO, or stopping writes before copying. SQLite documents this risk and the safe alternatives here: https://sqlite.org/howtocorrupt.html#backup_or_restore_while_a_transaction_is_active

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced the "copy all three" advice with consistent-snapshot guidance: sqlite3 events.db ".backup ..." (the online backup API), VACUUM INTO, or stop the engine before copying, with a link to the SQLite page you cited. This was the only place in docs/ and skills/ that gave backup advice. Fixed in d2bbcc1.

🤖 Generated with Claude Code

Comment thread emergent-engine/src/preflight.rs Outdated
let mut counts: std::collections::HashMap<String, usize> = std::collections::HashMap::new();
let mut current: Option<String> = None;

for line in before.lines() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Derive diagnostic paths from TOML structure rather than scanning header-like lines

This scan counts table headers inside multiline strings as real tables. The following config has two sinks, but validate --json reports the unknown field at sinks[2].publishes, a nonexistent third sink, instead of sinks[1].publishes:

[[sinks]]
name = "first"
path = "/bin/sh"
args = ["""
[[sinks]]
"""]
[[sinks]]
name = "second"
path = "/bin/sh"
publishes = ["x"]

This makes the machine-readable location unreliable for a control plane or editor. Please derive the path from parser-aware structure and add coverage for multiline strings. Quoted table names also need normalization: [["sinks"]] currently produces a path containing literal quotes around sinks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

table_path_at now walks toml_edit's parsed document and picks the header table that starts last at or before the error offset, so a [[sinks]] inside a multiline string is not a table and [["sinks"]] becomes sinks[0]. When the text does not parse, path is omitted. Your example now reports sinks[1].publishes, and there are tests for it, for the quoted name, and for the ordinary case. This adds toml_edit as a direct dependency; toml already pulls it in at the same version, so Cargo.lock still has one copy. Fixed in e459834.

🤖 Generated with Claude Code

Comment thread emergent-engine/src/config.rs Outdated

/// Every `[engine].api_allowed_hosts` problem (pure function).
fn allowed_host_issues(&self) -> Vec<ConfigIssue> {
crate::api_host::parse_allowed_hosts(&self.engine.api_allowed_hosts)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: Collect each invalid allowed-host entry

parse_allowed_hosts returns immediately on the first invalid value, so this wrapper does not fulfill the new validation command's promise to report every semantic error in one run.

Reproduced with:

[engine]
api_allowed_hosts = ["https://a.example", "https://b.example"]

The JSON report contains only the error for https://a.example. Please validate each entry and report its index, such as engine.api_allowed_hosts[0] and engine.api_allowed_hosts[1], while preserving startup's first-error behavior.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validate now reports one invalid_api_allowed_host per bad entry at engine.api_allowed_hosts[i], using a new per-entry api_host::parse_allowed_host that parse_allowed_hosts also calls. Startup still refuses on the first bad entry, and a test covers both behaviors. Fixed in 58fe240.

🤖 Generated with Claude Code

timaxorum and others added 4 commits September 29, 2026 22:12
…ent store

- `emergent validate [--json] [--skip-path-check]` runs the engine's own
  startup checks (schema, names, restart policies, topics, api_allowed_hosts,
  primitive paths, IPC connection capacity) without starting anything, and
  reports every problem with a stable code and config path. Startup and
  validate share one code path (`preflight`).
- `--log-stdout` / `EMERGENT_LOG_STDOUT=1` sends info-level logs to stdout
  for containers; the default and `--verbose` are unchanged.
- The SQLite event store opens with journal_mode=WAL and synchronous=NORMAL,
  so an external reader no longer makes inserts fail with SQLITE_BUSY.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuTzw5ZEExrSJe2GDkYr1F
`emergent validate` now reports one invalid_api_allowed_host issue per bad
entry, at `engine.api_allowed_hosts[i]`, instead of stopping at the first.
The per-entry decision is `api_host::parse_allowed_host`, which
`parse_allowed_hosts` now calls; startup still refuses on the first error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuTzw5ZEExrSJe2GDkYr1F
…ture

`table_path_at` scanned header-like lines, so a `[[sinks]]` inside a
multiline string counted as a table and `[["sinks"]]` kept its quotes. It
now walks `toml_edit`'s parsed document and picks the header table that
starts last at or before the error offset, so strings are not tables and
quoted names are unquoted. When the text does not parse, `path` is omitted.

`toml_edit` becomes a direct dependency at the version `toml` already
pulls in, so the lockfile still holds one copy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuTzw5ZEExrSJe2GDkYr1F
Copying `events.db`, `-wal` and `-shm` one by one while the engine writes
can capture a commit or checkpoint between copies and yield a corrupt
backup. Recommend `sqlite3 .backup` (the online backup API), `VACUUM INTO`,
or stopping the engine before copying.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuTzw5ZEExrSJe2GDkYr1F
@timaxorum
timaxorum force-pushed the feat/validate-and-container-logging branch from bb5520e to d2bbcc1 Compare September 30, 2026 03:16
@timaxorum

Copy link
Copy Markdown
Contributor Author

Rebased onto main at v0.15.0 (acton-reactive 10). The one conflict was in main.rs: upstream added apply_engine_ipc_overrides (idle read timeout = 0), and this PR moved IPC config creation into preflight::ipc_config. I moved the override into preflight::apply_engine_ipc_overrides and ipc_config calls it, so startup and validate behave as upstream does. Upstream's IPC tests are unchanged apart from the new call path. fmt, clippy (-D warnings) and the workspace tests pass.

🤖 Generated with Claude Code

@rrrodzilla rrrodzilla left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at d2bbcc1. All three previous findings are resolved:

  • Live SQLite backup guidance now recommends a consistent snapshot through .backup or VACUUM INTO, or stopping the engine before copying.
  • Diagnostic paths come from parsed TOML structure. The multiline-string example reports sinks[1].publishes, and quoted table names report sinks[0].publishes.
  • Every invalid allowed-host entry is reported at its own index, while startup retains first-error behavior.

The rebase onto v0.15.0 preserves the issue #91 idle-read override, separate admission deadline, and regression tests. I found no additional blocking issues in the updated diff.

Verified locally:

  • cargo nextest run -p emergent-engine: 407 passed, 1 skipped.
  • cargo clippy --workspace --all-targets -- -D warnings: passed.
  • cargo fmt --all --check: passed.
  • Direct CLI checks for multiline strings, quoted table names, and multiple invalid hosts: passed.

@rrrodzilla
rrrodzilla merged commit 04a06cb into Govcraft:main Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants