Repository navigation
chore(release): 0.14.0 for the engine and all four SDKs - #89
Merged
Merged
Conversation
A coding agent left to its defaults wraps a whole workflow in one exec-handler and calls it a pipeline. Ship the skill that teaches the idiom alongside the engine, so the guidance versions with the code it describes.
The stopping rules bound a primitive's size but said nothing about what it is built from, so a topology could pass them with a score.sh behind every block. Add a ladder that prefers marketplace primitives, then bare commands, then one shape-act-shape pipe, with no rung for a script file, plus the tells that a command has become a program and the split each one implies.
The reference predated primitives 0.10.0. Most consequentially it still said a non-zero exit with empty stderr is a silent filter; since 0.10.0 that is an error event unless the code is listed in --silent-exit-codes, and jq select() filters by exiting 0 with no output. Also fixes wrong short flags and env names, documents the missing flags and payload shapes, and records that the publishes array overrides topic flags by position.
Subscriptions are exact-match, unknown keys are ignored, retention_days and wire_format have no effect, and an invalid primitive name hangs startup. The topology-viewer note now says what it renders today: the engine node only.
Wire format is always MessagePack, subscriptions are exact-match, shutdown runs on fixed timers with no SIGKILL, nothing is restarted, and the engine answers only subscription requests.
The Rust run_handler example could not compile when the handler publishes, so it is now the subscribe loop. TypeScript, Go, and the lint guidance are fixed against the real signatures. Rust and TypeScript snippets were compiled.
stream-runner replaces the shell splitter, every shell body holds a single act, the routers are exclusive and exhaustive, and every path ends in the ack. The old config lost the issue title on re-entry, never bounded its loop, and read stdin twice. The new one was run end to end against stubs.
The join is now an accumulator, a router, and a sink. Scripts, shell branches, and wildcard subscriptions are gone, fragments carry their path, and the exit-code, concurrency, and stream-runner claims match primitives 0.12.0. The join, fan-in, reaper, and self-seeding patterns were run live.
The join is three primitives instead of a shell conditional that published errors, shell bodies are TOML literal strings, the one-event and silent-exit claims match the code, and Gate 2 gains a dead-end test covering unrouted events and wildcard subscriptions.
…havior An invalid primitive name aborts the engine and orphans the spawned child, it does not hang. The ignored system.shutdown kind is confirmed at runtime for the Rust, Python and TypeScript SDKs. Known limitations now cite their issues.
The Python client treated any non-empty EMERGENT_UNWRAP_STDOUT as on, so setting it to "false" or "0" to switch unwrapping off enabled it instead. TypeScript accepted only "true". Both now parse the flag through one pure function that accepts "true" and "1", case-insensitively, which is what the Rust and Go SDKs already do. Closes #47
get_topology() in every SDK published system.request.topology and waited for system.response.topology, but the engine only forwarded the request to subscribers and expected a "topology-query handler" that never shipped. Every call timed out after 30 s. The engine now answers the request itself, in the same style as system.request.subscriptions: it replies with system.response.topology, source emergent-engine, the request's correlation_id copied onto the envelope where the SDKs match it, and the primitive list as the payload. The primitive list is built by a new pure function, emergent_engine::topology::build_topology_payload, which the HTTP GET /api/topology handler now uses too, so the two transports cannot report different topologies. Refs #46
EmergentHandler::messages and EmergentSink::messages took a types
argument and threw it away, always subscribing to the engine-configured
list instead. EmergentSink::messages("x", ["timer.tick"]) against a
config with an empty subscribes list received nothing and reported no
error. The Python and TypeScript sinks had the same defect.
The rule is now: the topics you pass win, and an empty list falls back to
the engine's configured subscribes list. The engine is only queried when
the caller requested nothing, so the explicit case also drops a round
trip. The choice lives in two pure functions per SDK, resolve_topics and
needs_configured_topics, each with unit tests, and is documented on every
method that takes the argument.
This is not a breaking change: callers that passed topics matching their
config keep the same behavior, and callers that passed an empty list keep
the config-driven behavior.
The Go SDK has no messages() convenience, so it needed no change.
Refs #46
An agent given only this skill designed an invoice-intake topology and reported where the skill misled it. Each fix below was checked against a live engine before it was written down. - Open every piped shell body with `set -o pipefail`. Without it a failed act in the middle of a pipe left the trailing jq to exit 0, so the handler published nothing (a permanent stall behind a stream-runner) or published a success event with empty content. - Merge named fields from the act instead of `$orig + .`, so untrusted model output cannot overwrite an item's identity. - Add the `act && shape` form of rung 3 for acts with no stdin or stdout, with a tested claim-by-rename example, and say why `&&` is not a branch. - Split the retry pattern into guard, then delay, so an exhausted item escalates at once instead of sleeping its longest sleep first. Note that retried acts must be idempotent. - Add HTTP status as data, so a 4xx is routed to a person and only transport failures and 5xx reach the retry loop. - Say where a stream-runner ack can go and what each placement costs, that a poll needs a seen-set, that exec-source cannot wrap a streaming command, and that `--ack-topic` takes one topic. - Reconcile statements that disagreed: which error topics need a name, the dead-end test versus deliberate "nobody yet" events, the long-timeout tell versus delay handlers, the name test versus router names, and the subscriber test for steps inserted between stages. - Add the requirement test, the one gate that argues for fewer primitives, and stop the straight-line check from inviting invented loops. - Give a judge that reads outside text no tools (`--tools ""`).
…ting The retry handler slept before checking its guard and disagreed with the pattern it was meant to illustrate. It is now two exclusive routers on the failure event followed by a delay whose only act is the pause, with the backoff read from the payload. The feedback edge is renamed for what became true (`issue.backoff-elapsed`). Piped bodies open with `set -o pipefail`, the scoring merge takes three named fields from the model, the judge runs with no tools, and an idle poll publishes no list event. The catalog says which events are unconsumed on purpose, and Gate 2 gains the requirement and dead-end rows. Re-run end to end against stubbed gh, claude, and curl: three issues in, three settled, nothing on exec.error, and the third scoring failure escalated with no pause in front of it.
EmergentSink::messages connected, subscribed, and returned only the stream. The sink was dropped when the function returned, which dropped the last reference to the IPC client, closed the connection, and closed the stream. Callers got a stream that yielded None on the first read instead of messages, with no error. MessageStream can now own the connection that feeds it, and EmergentSink::messages hands its client over before returning. Nothing else changes: the field is private and unused by every other constructor, which still returns the primitive alongside its stream. Verified against a live engine: a sink with an empty configured subscribes list now receives a message published on the topic it asked for, where before the stream closed immediately. The Python and TypeScript sinks are async generators that hold their sink for the life of the iteration, so neither shares this defect. Refs #46
The pub/sub topology section told readers to run a handler that subscribes to system.request.topology, reads the HTTP API, and publishes the response. The engine answers the request itself now, so that handler is dead weight. Removes examples/handlers/topology-api, which also contradicted the engine: it put the correlation id inside the payload as correlationId, while every SDK matches on the envelope's correlation_id. The source and viewer examples still apply and are unchanged; the engine simply plays the part the handler used to. Closes #46
The engine now answers system.request.topology itself and messages() honors its types argument, so the references no longer tell readers to add a responder handler or to expect the argument to be ignored. Each statement names the last release with the old behavior. Refs #46
mypy rejected one variable reused for three request types; each pending lookup now has its own name. The frozen-model tests assert pydantic's ValidationError instead of a blind Exception, the stream tests keep and await the tasks they spawn, and two annotation-only imports move under TYPE_CHECKING. ruff check, ruff format, and mypy are all clean.
The skill's SDK reference described the Rust helper only. It now says that the Python, TypeScript and Go helpers fire the same shutdown signal when the engine closes the connection, what the earlier releases did, and which Sources were exposed. The Go SDK page says the same in short. Refs #83
The startup line and the middleware want the same list, and parsing it twice invited the two to disagree. Share one Arc. Refs #81
sdks/ts has a package.json, which puts Deno in manual node_modules mode. In a clean copy of the tree deno check fails with "Could not find a matching package for 'npm:typeid-js@1.2.0'" until deno install runs; with it, check passes and 106 tests pass. The JSR publish workflow already installs. Refs #70
The API answered any Host header, so a page on any domain could rebind that domain to 127.0.0.1 and read the topology from the browser. It now answers to IP literals, localhost and the names in [engine].api_allowed_hosts, and refuses the rest with 421, over HTTP/1.1 and HTTP/2 alike. Closes #81
Python and TypeScript no longer orphan a newer message stream when an older one closes, and close the stream of a subscribe that throws. run_source in Python, TypeScript and Go now fires its shutdown signal when the engine closes the connection, so a hand-started Source stops instead of publishing into a dead socket. Closes #82 Closes #83
…dules The root package.json puts Deno in manual node_modules mode for every script below it, so on a fresh clone console_color, topology-viewer and topology-api exited 1 on the SDK's npm:typeid-js import. examples/deno.json turns that mode off for the examples, and CI now type-checks all three. Closes #85
Admission walked nowhere: every peer was Unmanaged, so declaration enforcement fell back to the source a client writes itself and a subscribe frame could not be judged at all. The resolver now walks the peer's process ancestry to a pid the engine spawned, which covers a launcher such as `uv` that stays alive as the engine's child while its interpreter opens the socket. The walk is a pure function over the child table and a parent lookup; /proc is the only thing the shell around it touches. A pid is a primitive exactly while the engine holds that child as live: an unreaped child's pid cannot be recycled, and a grandchild pid that is recycled no longer climbs to a recorded child. The module docs make the argument and the table tests cover it, along with the uv chain, deeper wrappers, a foreign process, the engine's own pid, a missing peer pid, the no-ancestry collapse, a cycle, pid 1 and a comm full of parentheses. Connections admitted as a primitive are recorded, and every one of them is revoked when that primitive's child exits. [engine].authenticate_connections says what happens to a peer the engine did not spawn: off admits it, warn says so, strict refuses another user id and still admits a same-user one, because the CLI and a primitive run by hand cannot be told from an impostor at admission. Refs #24
A client feeds one registered stream. A second subscribe() took the registration and left the earlier stream open and unfed, and from then on nothing could end it: not connection loss, not system.shutdown, not close(). An async for over it waited for ever. The stream being replaced is now closed at the moment the new one is registered, so its consumer stops. The engine keeps the earlier subscriptions, so the new stream receives the earlier topics too, which the subscribe docs and the README now say. Refs #86
A client feeds one registered stream. A second subscribe() took the registration and left the earlier stream open and unfed, and from then on nothing could end it: not connection loss, not system.shutdown, not close(). A for await over it waited for ever. The stream being replaced is now closed at the moment the new one is registered, so its consumer stops. The engine keeps the earlier subscriptions, so the new stream receives the earlier topics too, which the subscribe docs and the README now say. Refs #86
A client feeds one registered stream. A second Subscribe took the registration and left the earlier stream open and unfed, and from then on nothing could close it: not connection loss, not system.shutdown, not Close. A range over its channel waited for ever. The stream being replaced is now closed when the new one is registered, after c.mu is released because its onClose callback takes c.mu. The engine keeps the earlier subscriptions, so the new stream receives the earlier topics too, which the Subscribe docs and the README now say. Refs #86
A Rust client hands its push channel to its first stream, so a second stream can never be registered and the first is never orphaned. The second subscribe was refused, but only after the engine had accepted it: the refused topics started arriving on the first stream while the caller held SubscriptionFailed and subscribed_types() did not list them. The refusal now comes first, from a pure check on whether this client has handed out its stream, so the first stream and its subscriptions stay as they were. Tests run both a Handler and a Sink against a real engine. Refs #86
One stream per client in all four SDKs. In Python, TypeScript and Go a second subscribe ends the earlier stream and the new one receives the earlier topics too, because the engine keeps those subscriptions. In Rust the second call is refused before anything is sent. The skill's SDK reference and the Go and Rust SDK pages now say so, with what the earlier releases did. Refs #86
In Python, TypeScript and Go a second subscribe left the first stream open forever; it now ends the stream it replaces. The Rust SDK already refused a second subscribe, but only after sending it, so the refused topics were delivered anyway; it now refuses before anything is sent. Closes #86
…e a socket A primitive that dies with its own socket takes the connection with it, so the sweep on child exit finds nothing left to close and said nothing. The sweep that matters is the other one: a descendant still holding a connection the engine admitted under a name that is now dead. Log held and closed counts at debug for both, so an operator can see the path ran. Refs #24
The engine moves from 0.10.10 to 0.14.0 so the engine and the SDKs share one version. 0.11.0 is skipped because a v0.11.0 tag already exists.
Version said 0.10.5 in Go and __version__ said 0.1.0 in Python while both were released as 0.13.1. Each now has a test that fails when the constant and the released version disagree. Closes #87
…ctions The docs still said the engine checks self-reported names and that issue #24 was open. Rewrite the enforcement sections around the resolver that ships: where a connection's name comes from, why pid reuse is safe, what revocation on child exit is for, what authenticate_connections does and does not buy, and what degrades where process ancestry is unavailable. Refs #24
# Conflicts: # CLAUDE.md # emergent-engine/src/config.rs
The fixes on this branch were described relative to the last published versions, because the release number was not yet decided. It is now 0.14.0 for the engine and all four SDKs, so the text names it. Statements about 0.10.10 and 0.13.1 themselves stay as they are.
The steps still said the Go SDK has no version file and that the engine and the SDKs were on different numbers.
An IPC connection is now named by the spawned child it descends from, read from the peer credentials and /proc, not by the name the client sends. A primitive started through uv or sh is named like a direct spawn, so declaration enforcement applies to it and its tier is released on its subscribe instead of at the 5000 ms deadline. A connection the engine cannot tie to a child gets the protocol topics and no primitive's name. Connections held for a child are revoked when it exits. authenticate_connections = off, warn or strict governs only whether a peer under another uid is admitted. Closes #24 Closes #23
….14.0 Conflicts were the version wording only: the sweep side is kept and the 0.14.0 wording reapplied to it.
t.TempDir() embeds the test and subtest names. Go 1.23, which CI uses, does not shorten them, so three tests built a socket path the kernel refuses with 'bind: invalid argument'. Newer Go releases shorten the name, which is why the suite passed locally.
GitHub creates no push event when more than three tags arrive in one push, so the four SDK tags pushed together started no publish workflow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 0.14.0 for the engine and all four SDKs. This pull request exists so that CI runs on GitHub before any SDK tag is pushed: the publish workflows do not run tests themselves.
Every local gate passes on this branch: fmt, clippy with 0 warnings, 534 Rust tests, doctests, the scaffold compile test,
cargo package -p emergent-client, 398 Python tests with ruff and mypy clean, 113 TypeScript tests, the Go race suite and the Go and Deno examples.Draft release notes follow.
Emergent 0.14.0 (engine and all four SDKs)
This release fixes message loss, silent failures and local exposure across the engine and every SDK. It contains breaking changes in each SDK and new load-time checks in the engine, so read "Before you upgrade" first. The engine jumps from 0.10.10 to 0.14.0 so that the engine and the SDKs share one version from here on.
Before you upgrade
retention_daysis enforced, and it defaults to 30. It was parsed and ignored before. On first start the engine prunes events and rotated JSON logs older than that, whether or not your config sets the key. Setretention_days = 0to keep everything.max_connectionsis checked at load. The engine refuses to start when the limit cannot cover the enabled primitives plus 4.emergent scaffold.prefix.*as an exact name.Engine
Security
localhostand names listed in the new[engine].api_allowed_hosts. Any otherHostgets421 Misdirected Request, over HTTP/1.1 and HTTP/2. This closes DNS rebinding against/api/topology. If you reach the API through a reverse proxy under a host name, list that name.[engine].enforce_declarations = "off" | "warn" | "strict"(defaultoff). Instrictthe broker refuses a publish or subscribe outside the primitive's declaredpublishesandsubscribes.uv runorsh -care named like direct spawns. A process the engine did not start (a CLI query, the topology viewer, a primitive you run by hand) gets the protocol topics and no primitive's name, so withenforce_declarationsatwarnorstricta hand-started primitive is reported or refused. When a primitive exits, connections still held under its name are closed. Ancestry is read from/proc, so wrapped primitives are named on Linux only; elsewhere the engine falls back to the name on the message and says so at start.[engine].authenticate_connections = "off" | "warn" | "strict"(defaultoff).strictrefuses a peer running as another user. It still admits a same-user peer, because the kernel cannot tell a viewer from an impostor at admission.Reliability
startup_ready_timeout_msbounds the wait.shutdown_drain_ms,shutdown_grace_ms), then SIGKILL to the process group. Measured exit: about 130 ms, down from 7 to 14 s.restartpolicy.system.started.*) now route. Overlapping topics deliver one copy. A wildcard anywhere but the end is a load error.publish_ackof asystem.request.*message is acknowledged instead of reported as failed, and the broker no longer sends itself a reply for every fire-and-forget publish./api/topologyand the pub/sub topology answer report live state and pid in a stable order.Marketplace and scaffold
emergent scaffoldpins the SDK version it was built with, fails when it cannot write every file, and rejects names the loader would reject.SDKs (Rust, Python, TypeScript, Go)
Breaking changes
run_handlerpasses the handler to the closure by value (EmergentHandleris nowClone), so the closure can publish through it.ClientErrorgains the variantInvalidSubscriptionTopic, which breaks exhaustive matches.MessageStreamis no longerUnwindSafe. New:PublishStatsandpublish_stats()for refused fire-and-forget publishes, fallibleEmergentMessage::try_newconstructors, thesubscribemodule, andVERSION.publish()throwsPublishError(withcause) when the socket refuses a write. The exported subscription, publish and discovery error classes are now the ones actually thrown.PublishErrororConnectionErrorfrom the SDK, not a rawOSError. The default wire format is MessagePack. Unknown envelope fields are ignored.SubscriptionError,PublishError,DiscoveryErrorandConnectionErrorcarryErrand implementUnwrap. Unkeyed struct literals of these types no longer compile.EMERGENT_UNWRAP_STDOUTis on only fortrueor1, trimmed and case-insensitive.Fixes in every SDK
run_sourcefires its shutdown signal, so a hand-started source exits instead of publishing into a dead socket.subscribe()ends the stream it replaces.MessageStreampush andClosesafe against each other.Examples, CI and the agent skill
examples/deno.json).skills/emergentships a coding-agent skill, verified against the engine, the four SDKs and primitives 0.12.0.