Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ emergent scaffold -t handler -n my_filter -l rust -S timer.tick -p timer.filtere
# Initialize a new config file
emergent init

# Check a config with the startup checks, without starting anything
emergent validate --config ./config/emergent.toml [--json] [--skip-path-check]

# Marketplace commands
emergent marketplace list
emergent marketplace install http-source
Expand Down Expand Up @@ -102,7 +105,9 @@ so a slow-starting subscriber missed the first events (Govcraft/emergent#66).

### Engine Modules

- `config.rs` — TOML config loading, path expansion, validation
- `config.rs` — TOML config loading, path expansion, validation (each rule reports a located `ConfigIssue`)
- `preflight.rs` — the one call startup and `emergent validate` both make before anything is spawned: config issues, connection capacity, warnings, and the `validate --json` report
- `logging.rs` — pure decision for where engine logs go (file, `--verbose`, `--log-stdout` / `EMERGENT_LOG_STDOUT`)
- `process_manager.rs` — Actor-based lifecycle for primitives
- `primitive_actor.rs` — Per-primitive actor (spawns child process, monitors, broadcasts system events, owns the primitive's live state)
- `lifecycle.rs`: pure state machine mapping a lifecycle event to a primitive's next state, pid and error
Expand Down Expand Up @@ -192,6 +197,12 @@ using `[timeouts] admission_timeout_ms` (default 60000) and

Connection limit: every enabled primitive holds one IPC connection for the life of its process. The ceiling comes from acton-reactive, resolved from `$XDG_CONFIG_HOME/acton/ipc.toml` (`[limits] max_connections`) or its own default; `[engine].max_connections` overrides both, and leaving the key out keeps whatever acton resolved. From 0.14.0 the engine refuses to start when that limit cannot cover every enabled primitive plus `RESERVED_IPC_CONNECTIONS` (4: one for a restart overlap, three for CLI and topology-viewer queries), with an error naming both numbers. On 0.10.10 and earlier there was no check, so an oversized topology started with some primitives silently dropped at the accept semaphore while `/api/topology` still reported them running. The decision lives in `emergent-engine/src/config.rs` as `check_connection_capacity`, a pure function.

Pre-flight: startup and `emergent validate` run the same checks through `emergent_engine::preflight::preflight`, after `EmergentConfig::read` (parse and resolve, no validation). Startup refuses on the first error before creating the socket, the log file or the event store; `validate` reports every error, with a stable `code` and a config `path` such as `sinks[0].publishes`, and exits 1. `--skip-path-check` drops only the primitive-path check. A new pre-spawn rule belongs in `EmergentConfig::issues` or `preflight`, never in `main.rs`, so the two cannot drift.

Logging: the engine logs to `~/.local/share/emergent/<name>/emergent.log` by default, to the terminal with `--verbose`, and to stdout without ANSI codes (unless stdout is a TTY) with `--log-stdout` or `EMERGENT_LOG_STDOUT=1`, for containers. The decision is `emergent_engine::logging::log_destination`.

Event store: the SQLite database opens in WAL mode with `synchronous = NORMAL`, so an external reader never blocks an insert. `NORMAL` can lose the last commits on power loss, not on an engine crash; the JSON log is not fsynced per event either.

Marketplace: from 0.14.0 the registry is two files fetched over HTTPS,
`index.toml` and `manifests.toml`, published as assets of the emergent-primitives
release. `[marketplace].registry_url` (in `$XDG_CONFIG_HOME/emergent/marketplace.toml`)
Expand Down
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ rmp-serde = "1"

# Configuration
toml = "0.8"
# The parser `toml` itself uses; kept at its version so the lockfile holds one copy.
toml_edit = "0.22.27"

# Database
rusqlite = { version = "0.32", features = ["bundled"] }
Expand Down
2 changes: 2 additions & 0 deletions docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ docker images emergent:local
| Variable | Default | Description |
|----------|---------|-------------|
| `RUST_LOG` | `info` | Log level (trace, debug, info, warn, error) |
| `EMERGENT_LOG_STDOUT` | unset | Set to `1` to log to stdout, where `docker logs` sees it, instead of `emergent.log` inside the container (same as `--log-stdout`) |

## GitHub Container Registry

Expand All @@ -106,6 +107,7 @@ healthcheck:

**Engine won't start:**
- Verify config file exists at `config/emergent.toml`
- Check it without starting the engine: `docker run --rm -v ./config:/etc/emergent:ro emergent:local validate --config /etc/emergent/emergent.toml --skip-path-check`
- Check logs: `docker-compose logs`

**Socket permission issues:**
Expand Down
41 changes: 40 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ emergent --config /path/to/config.toml
emergent --config ./emergent.toml
```

Use `emergent init` to generate a starter config interactively.
Use `emergent init` to generate a starter config interactively, and `emergent validate` to check one without starting anything (see [Validation](#validation)).

On engine 0.10.10 and earlier an unknown key was ignored without a word, so `retension_days` or a singular `subscribe` loaded cleanly and did nothing. From 0.14.0 an unknown key is a load error that names the key, the keys its table accepts, and the file it came from.

Expand Down Expand Up @@ -421,6 +421,8 @@ retention_days = 30

Both paths support `"auto"` for XDG data directory placement.

The SQLite database is opened in write-ahead-log mode (`journal_mode = WAL`, `synchronous = NORMAL`), so another process can read it while the engine runs, whether `sqlite3` or a dashboard tailing new events, without making the engine's inserts fail with `database is locked`. A reader sees the last committed events. SQLite keeps `events.db-wal` and `events.db-shm` next to the database while it is open. Do not back it up by copying those files while the engine runs: a commit or checkpoint between copies leaves a backup that is inconsistent and can be corrupt ([SQLite: backup while a transaction is active](https://sqlite.org/howtocorrupt.html#backup_or_restore_while_a_transaction_is_active)). Take a consistent snapshot instead, with the online backup API (`sqlite3 events.db ".backup events-backup.db"`) or `sqlite3 events.db "VACUUM INTO 'events-backup.db'"`, or stop the engine first and then copy the database, together with any `-wal` file still beside it. `NORMAL` can lose the last few events on a power failure or OS crash, never on an engine crash, and never corrupts the file.

**Retention:** on engine 0.10.10 and earlier `retention_days` was parsed and never enforced, so both stores grew without bound. From 0.14.0 the engine prunes at startup and once a day afterwards: SQLite rows older than the window are deleted, and `events-YYYY-MM-DD.jsonl` files dated before the window are removed. The day at the edge of the window is kept, and files that are not rotated event logs are never touched. Each pass logs what it removed. `retention_days = 0` disables pruning and keeps every event, which the engine states at startup.

## Sources
Expand Down Expand Up @@ -808,4 +810,41 @@ The engine validates configuration at startup:
- `restart` must be one of `never`, `on-failure` or `always` (from 0.14.0)
- Unknown keys are a load error that names the key and its table (from 0.14.0). That is what rejects `subscribes` on a source and `publishes` on a sink, since neither table has that key

- The IPC connection limit (`[engine].max_connections`, else acton's `ipc.toml`, else its default) must cover every enabled primitive plus 4 reserved connections (from 0.14.0)

`subscribes` and `publishes` may be empty or omitted. The engine does not require them, and an empty `subscribes` on a handler or sink loads and receives nothing.

### Checking a config without starting it

`emergent validate` runs the same checks, through the same code, and reports every problem rather than stopping at the first. It spawns nothing and touches neither the socket nor the event store. It exits `0` when the engine would start and `1` when it would not.

```bash
emergent validate --config ./emergent.toml
emergent validate --config ./emergent.toml --json
emergent validate --config ./emergent.toml --json --skip-path-check
```

`--skip-path-check` leaves out the check that each enabled primitive's `path` exists, for a host (a CI job, a control plane) that validates configs for primitives installed somewhere else. `--json` prints one object on stdout:

```json
{
"ok": false,
"engine_version": "0.14.0",
"errors": [
{
"code": "unknown_field",
"message": "unknown field `publishes` (line 540, column 1)",
"path": "sinks[0].publishes"
}
],
"warnings": []
}
```

`path` locates the problem in the file (`sinks[0]` is the first `[[sinks]]` table) and is omitted when there is nothing to point at. `code` is one of `config_not_found`, `read_error`, `parse_error`, `unknown_field`, `invalid_name`, `duplicate_name`, `invalid_restart_policy`, `invalid_subscription_topic`, `invalid_api_allowed_host`, `path_not_found` and `connection_capacity` for errors, and `wire_format_ignored` for warnings. TOML parsing stops at the first unknown key or syntax error, so a file with several gets one of them reported per run; everything checked after parsing is reported in full.

The connection limit is resolved on the host that runs `validate`, from its own `$XDG_CONFIG_HOME/acton/ipc.toml` when `[engine].max_connections` is not set.

## Logging

A running engine logs to `~/.local/share/emergent/<engine.name>/emergent.log`. `--verbose` sends the same lines to the terminal instead. In a container, or under a supervisor that collects stdout, pass `--log-stdout` or set `EMERGENT_LOG_STDOUT=1` (also `true`, `yes`, `on`): the lines go to stdout, without colour codes unless stdout is a terminal, and no log file is written. `RUST_LOG` sets the level in every mode.
1 change: 1 addition & 0 deletions emergent-engine/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ serde = { workspace = true, features = ["derive"] }
serde_json.workspace = true
rmp-serde.workspace = true
toml.workspace = true
toml_edit.workspace = true
rusqlite.workspace = true
tracing.workspace = true
tracing-subscriber.workspace = true
Expand Down
40 changes: 35 additions & 5 deletions emergent-engine/src/api_host.rs
Original file line number Diff line number Diff line change
Expand Up @@ -215,18 +215,30 @@ pub fn decide_request_host(
pub fn parse_allowed_hosts(values: &[String]) -> Result<Vec<String>, String> {
let mut hosts: Vec<String> = Vec::new();
for value in values {
let Some(host) = parse_host_name(value) else {
return Err(format!(
"Invalid {ALLOWED_HOSTS_KEY} entry \"{value}\": expected a host name with no scheme, port or path, such as app.example"
));
};
let host = parse_allowed_host(value)?;
if !hosts.contains(&host) {
hosts.push(host);
}
}
Ok(hosts)
}

/// Parse one configured name (pure function).
///
/// The per-entry decision [`parse_allowed_hosts`] makes, for callers that
/// report every bad entry rather than only the first.
///
/// # Errors
///
/// Returns the one thing wrong with `value` when it is not a host name.
pub fn parse_allowed_host(value: &str) -> Result<String, String> {
parse_host_name(value).ok_or_else(|| {
format!(
"Invalid {ALLOWED_HOSTS_KEY} entry \"{value}\": expected a host name with no scheme, port or path, such as app.example"
)
})
}

fn parse_host_name(value: &str) -> Option<String> {
if value.is_empty() || value.starts_with('-') {
return None;
Expand Down Expand Up @@ -519,6 +531,24 @@ mod tests {
}
}

#[test]
fn parse_allowed_host_agrees_with_the_list_parser() {
for value in ["app.example", "APP.Example.", "münchen.example", "my_host"] {
assert_eq!(
parse_allowed_host(value).map(|host| vec![host]),
parse_allowed_hosts(&[value.to_owned()]),
"{value}"
);
}
for value in ["", "https://app.example", "app.example:8080", "*"] {
assert_eq!(
parse_allowed_host(value).map(|host| vec![host]),
parse_allowed_hosts(&[value.to_owned()]),
"{value}"
);
}
}

#[test]
fn every_name_that_parses_is_one_decide_host_accepts() {
// The point of storing a name in the browser's spelling: what an
Expand Down
Loading