README.md

Tests with evidence

The suite uses Effect v4 and @effect/vitest. Scenarios use layer and it.effect. Effect owns the servers, HTTP clients, actors, browser contexts, concurrency, raw evidence capture, and cleanup. The test runtime uses live clocks because it talks to real processes. It does not import Executor implementations or construct partial application servers.

Playwright is the browser driver, behind an injected Effect adapter. Its test runner and fixture system are not used. React renders evidence only when requested. Normal runs, including CI and failed runs, do not build the viewer, render videos, or generate HTML reports.

Run

The private HTTPS MCP scenario requires openssl on PATH to issue its temporary test certificate.

bun install
bunx playwright install chromium webkit
bun run e2e:prepare
bun run e2e:self-host

e2e:prepare builds the dashboards, app framework, bundled Motel and shared workerd runtime artifact. Run it again after changing these inputs. It caches each build step's outputs in ~/.cache/executor-e2e-prepare (or $EXECUTOR_E2E_CACHE) under a hash of the step's inputs: the files git sees in its packages and their workspace dependencies, the lockfile, patches and root config, and the Node and Bun versions. A step whose inputs have not changed, in any rift, is restored instead of built. scripts/e2e-prepare.ts lists each step's inputs; add a file there when a build starts reading one outside them. CI and --no-cache build every step. The Node server runs current TypeScript source; the workerd host bundle is prebuilt, so a change to the SDK's app runner needs bun run e2e:runtime before a self-host run sees it. bun run e2e:check runs the boundary check and TypeScript check; the root check includes it.

Scenarios on the self-host, local and managed Cloud targets do not install npm packages from the internet. Products install them from the run's loopback registry (support/npm-registry.ts). It serves the staged apps package and only the versions pinned in e2e/npm-packages.json, from archives that e2e:prepare downloads once and checks against their integrity. It refuses any other package with a 404 that names the lock. When a scenario declares a new package, add it to the roots in scripts/e2e-npm-packages.ts, then run node scripts/e2e-npm-packages.ts --update. Self-host and local read a synthetic official app feed and the public app registry from a loopback copy (support/public-catalogs.ts), through EXECUTOR_CATALOG_URL and EXECUTOR_REGISTRY_URL. A scenario that checks interoperation with a real third-party service, such as remoteMcpCloudflare, runs only in Cloud tests on main.

bun run e2e:local
E2E_ROWS=12 bun run e2e:self-host
bun run e2e:self-host --test-name 'password login'

The default data-volume scenario remains 1,000 accounts with four concurrent writers. Smaller runs use the same assertions. --test-name is a regular expression matched against scenario titles, without the enclosing Vitest suite name. Filtered cases are not counted as executed cases in the evidence view.

Native self-host authentication

The ordinary self-host suite uses Bun. To exercise the packaged Go/workerd host's trusted-proxy boundary and public OAuth registration rate limit, prepare its artifacts and run the focused native scenario with the same Effect/Vitest runner:

bun run e2e:prepare
bun run --cwd apps/hosted/self-host package:runtime
mkdir -p .local/native-auth
go build -C apps/hosted/self-host/native -o ../../../../.local/native-auth/executor-host .
bunx vitest run --config e2e/self-host-native.config.ts --testNamePattern 'Native self-host OAuth rate limits'

The same artifacts run the native telemetry scenarios (--testNamePattern 'Native self-host (telemetry ingest|restarts its telemetry)'). The packaged runtime must include motel.capnp. The host rewrites that config into a temporary file and serves the collector on a private Unix socket, recorded at .executor-telemetry in the product data directory. Nothing listens on TCP port 4318. The scenarios check that collector ingest does not delay product health, that a crashed collector restarts on its retained store after the supervisor's one-second backoff, and that Ctrl-C stops it with a draining product without a restart.

This needs Go and the current platform's workerd executable. The scenario owns its listeners, data directories and processes; logs remain in .local/native-auth/ and product data is removed at cleanup. Missing artifacts fail preparation rather than substituting the Bun server. Rebuild both native artifacts after source changes. It is registered separately in the test plan and is not run by the ordinary Bun self-host job or existing release configuration. No CI coverage for this native scenario is claimed until that job is explicitly wired.

Line coverage

E2E_COVERAGE=1 profiles every self-host and local product process. Self-host runs under bun test --coverage through apps/hosted/self-host/scripts/coverage-main.ts, because Bun profiles nothing else; local runs with NODE_V8_COVERAGE. Each scenario starts its own product, so each report under <run>/<target>/coverage/<scenario id>/ belongs to one scenario. Chromium also records each scenario's page scripts on every target; e2e:prepare builds the dashboards with unreferenced source maps, which map that coverage to the dashboards' sources. Managed Cloud's Worker serves every scenario at once, so Cloud counts only browser coverage.

The code each product runs inside workerd is profiled too: the host Worker, and the app framework in every app and data facet Worker. The runner starts workerd with V8's --log-function-events (through ALCHEMY_WORKERD_V8_FLAGS), and support/workerd-coverage.ts reads each isolate's log once the product stops. A function counts as run when V8 compiled it lazily, which only a call does, or logged its first execution: a function that ran counts all its lines, including branches it skipped. Nothing attaches to the product. workerd's inspector stays closed: with it open, an app Worker the runtime unloaded fails to load again under the same name. e2e:prepare writes source maps beside the host bundle (.local/test-runtime/host.json.map) and the framework modules (packages/apps/dist/runtime.json.map, outside the package's files), which map that coverage to packages/. An app's own bundle is not product code and is not counted.

E2E_COVERAGE=1 bun run e2e:self-host --test-name 'password login'
node e2e/coverage.ts collect .local/line-coverage.json .local/e2e/<run>
node e2e/coverage.ts report .local/line-coverage.json

report prints coverage by target and package, and the scenarios that cover no line another scenario does not. On a full run, CI's line-coverage job fails when a target covers fewer lines than e2e/coverage-baseline.json records, less a tolerance for paths that timing decides. Before deleting a scenario as redundant, check that it adds no line; a change that removes product code records a new baseline with node e2e/coverage.ts baseline e2e/coverage-baseline.json <collected files> from the line-coverage-* artifacts of a full run.

Dependency injection

support/platform.ts supplies target configuration and native platform services. support/case.ts composes the Layers used by Effect Vitest:

• Target: exact origin, runtime, data directory and private configuration.
• SessionClients: native Effect HTTP clients with independent cookie jars.
• Actors: fresh owner/admin/member sessions and an organization for each hosted case.
• Api: records each real request while preserving the test trace ID.
• BrowserDriver: scoped Playwright process, shared by a suite.
• Browser: isolated context per case, with an Effect use boundary for SDK calls.
• Evidence: named steps, screenshots, request timings and the final outcome.
• Telemetry: queries spans that actually reached Motel over HTTP.
• McpOAuth: public discovery, registration, PKCE, recorded browser consent, refresh and revocation.
• McpClient: scoped official MCP protocol clients with safe method/revision/trace evidence.
• ClaudeClient: real Claude Code in a Terminal Control PTY, with an injected model API.
• RecordingFocus: one clock and an ordered activity log for every recorded window.
• Terminal: scoped Terminal Control sessions; each use call selects that window in the recording.

Each test yields the services it needs. Native Vitest setup hooks start the isolated server and provision the actors declared in the test plan. Provisioning belongs to the setup deadline; scenario actions retain their full deadline. Scenarios that need the default management app can declare managementApps with the required actor roles. Setup waits for the app and each actor's linked Executor key through public APIs. Scenarios that exercise provisioning progress leave that prerequisite undeclared. Scenarios that need another Testing SDK scenario declare sdkScenarios; setup creates each one in a child scope the test may close. Native cleanup hooks release those fixtures and close the server scope. withCase provides the per-case Layers; @effect/vitest owns suite sharing and test interruption. The scenario deadline is 60 seconds, with separate 60-second setup and cleanup deadlines and no retries. report/lifecycle/ records setup, scenario, cleanup and total elapsed times. The report uses Vitest's final result so a cleanup failure cannot appear as a pass. Effect scopes close browsers, save evidence, and stop child processes. Owned server process groups get a 15-second graceful shutdown window before Effect escalates to SIGKILL. Effect.forEach bounds concurrent writes and waits for interrupted children. Transport failures are typed and do not expose credentials.

e2e:prepare builds .local/test-runtime/host.json once. Each isolated local or self-host process reads that explicit artifact instead of rebuilding the same trusted runtime. Rebuild with e2e:prepare after source changes; scenario data and processes remain isolated. Product listeners use an OS-assigned port.

Local and self-host restart scenarios can advance persisted wall time while their server is stopped through the authenticated runner control API. A runner-owned preload, loaded by Node and Bun, sets wall time for source and installed CLI processes. It is never packaged. The control API's kill ends the product process group with SIGKILL, running none of its shutdown, to model a crash. Sleep timers and duration measurements stay real. This tests minute-based scheduling without adding a minute of sleep to each scenario.

Legacy storage

Some upgrade bugs only arise from rows that an older version wrote and no current public surface can create. A scenario that needs such rows declares legacyStorage: true in test-plan.ts and calls legacyStorage from support/legacy-storage.ts. The runner stops that scenario's product, applies the parameterized SQL to its own PGlite database in one transaction, returns each statement's rows, and leaves the product stopped. serverControl("start") then boots the current server over that state, as an upgrade would. Statement errors roll back the whole write. A write targets the product database unless it names "mcp-auth": Local's separate MCP auth database, where MCP connections and grants live.

Use it only for legacy or upgrade-era data, for reading rows that exist only in storage, and for a storage fault no public surface can cause, such as a trigger that fails one owner's writes to one table. Create everything else through the product, as usual. The boundary check rejects any use outside a declared scenario. Only scenarios and the runner may import the module, and it is the only file allowed a database driver. The control route refuses undeclared scenarios at runtime. Self-host and Local support it. Cloud scenarios must be N/A: they share one Worker and database, which the runner cannot stop, and seeded rows would reach every case.

Each case owns its fixtures. Self-host runs signup and invitations against a new process and PGlite directory. Local uses its own process, database and pairing key. Cloud shares one Worker and database while each case owns a random organization and three synthetic identities. Managed Cloud serves the built site through that Worker, including its static asset rewrites and server-resolved entry pages. It does not start Vite's source development server. A runner-owned loopback process provisions Cloud fixtures. It keeps database and signing credentials in memory. Neither those credentials nor fixture endpoints are installed in the Worker. The test plan declares appOrigin: true for scenarios that use private app URLs. Deployed preparation verifies those HTTPS origins before the scenario deadline; an origin that misses the infrastructure deadline produces a native setup failure for its scenario. Beside organization provisioning, a dedicated organization deploys one small app through POST /apps/deploy until the freshly deployed compiler Worker builds it, within the same infrastructure deadline; otherwise preparation fails with CompilerNotReady. An organization whose actors are not provisioned within their 60-second deadline does the same. Independent scenarios still run. The preparation report retains each unavailable organization and its failure, the number of ready origins, both phase timings, and each origin's probe count and last safe DNS, TLS, or HTTP failure. All origin probes start together. Fallback organization cleanup uses the worker bound and preserves release order within each scenario, including when another organization's cleanup fails.

Files run in parallel with one worker per two CPUs by default, up to 16. Each self-host and local worker runs its own product server and browser. Managed Cloud workers share one local Cloud Worker and collector, so that target runs at most six. Use --workers 1 through --workers 32 to set the bound for every target. A file's cases retain their declared sequence. Interactive recordings use one worker. Filters load only applicable files. Each unattended test has a 60-second timeout, except the 1,000-account self-host inventory case, which has 120 seconds. Cleanup hooks retain a separate 60-second timeout. Interactive inspection has no test timeout. Within a scenario, use Effect.all or Effect.forEach with a concurrency bound when operations are independent. Keep dependent actions ordered.

The self-host load case creates 1,000 accounts through four concurrent API writers. The MCP catalog scale case deploys 29 apps with 7,000 tools and about 46 MB of input schemas, plus nine MCP apps whose server never answers, and bounds execute and search latency. CI gives both cases their own 16-vCPU Linux runner, one after the other, in parallel with the functional suite. The PGlite workload depends on single-thread speed. Running both workloads on one machine can consume its CPU budget and invalidate the load timing. The normal self-host command still includes every applicable case. To reproduce the CI split, use separate machines:

bun run e2e:self-host --test-name '^(?!.*(?:Claude Code connects|concurrent owners and admins save every account|MCP execute over 7,000 tools))'
bun run e2e:self-host --test-name 'concurrent owners and admins save every account'
bun run e2e:self-host --test-name 'MCP execute over 7,000 tools'

Shared SDK and interactive CLI

e2e/sdk/index.ts exports the same operations used by the runner and CLI. startEnvironment, startDeployment, and createScenario require an Effect scope. Closing that scope releases the owned resources, including after failure or interruption. runSuite and runDeployedSuite own their complete run scopes. withHostedCase composes actors inside the case's evidence and cleanup lifetime. Tests that start signed out use withCase.

Start a foreground environment in one terminal. Keep it running while exploring:

bun run testing start --target self-host --handle .local/testing.json

Then use another terminal:

bun run testing create --handle .local/testing.json --label 'Account picker'
# Copy the returned id into SCENARIO_ID. It is an identifier, not a credential.
bun run testing seed --handle .local/testing.json --id "$SCENARIO_ID" --preset populated
bun run testing request --handle .local/testing.json --id "$SCENARIO_ID" --role member --path /api/viewer
bun run testing open --handle .local/testing.json --id "$SCENARIO_ID" --role owner
bun run testing list --handle .local/testing.json
bun run testing remove --handle .local/testing.json --id "$SCENARIO_ID"
bun run testing stop --handle .local/testing.json

The handle is a private file with a generated loopback capability. Do not commit or share it. stop waits for cleanup. Ctrl-C also closes the environment scope. An abrupt machine or process kill cannot run finalizers; temporary deployed stages additionally have a lease for scheduled cleanup. Cleanup failures remain failures and retain logs. Restarting an environment never adopts another run's scenario identities.

Targets are local, self-host, cloud (managed local Worker), and deployed. Local has no organizations, so it supports paired requests and browser use but not organization populations. Deployed environments need the same explicit credential launcher as e2e:deployed. The CLI stays alive to own their teardown. request accepts --method and a JSON --body. All requests stay on the scenario's product origin and use its own role session. start --headless keeps browser operations unattended. open owns an authenticated browser and records screenshots, requests and video with the normal evidence adapter.

The populated recipe creates eight apps, 32 accounts, two groups, and 1,000 app records. large creates 24 apps, 200 accounts, and 10,000 records. Each recipe owns a private GitHub emulator instance and real issued tokens. App queries make authenticated HTTP calls to its private repository. Names and data are repeatable; identities and emulator instances are unique. Override volumes with:

bun run testing seed --handle .local/testing.json --id "$SCENARIO_ID" \
  --shape '{"seed":7,"apps":3,"accounts":12,"records":500}'

Repeating the same successful recipe returns its receipt. A different recipe or a failed seed requires a fresh scenario. Empty-state tests start empty. Other tests can opt into realistic data through the same operation:

import { seedOrganization, populations } from "../sdk/index.ts";
// Inside withHostedCase, where Api and Actors are already provided:
const data = yield * seedOrganization(populations.populated);
// Use data.apps, data.accounts and data.groups in product API/UI assertions.

Turn an exploration into a committed test in e2e/tests/ and register its title and targets in test-plan.ts. Use the same SDK recipe and role API operations, then add assertions for the behavior observed. testing-sdk.spec.ts proves parallel populated organizations, equal resource names, real provider requests, cross-organization denial and survival after another scenario fails. testing-cli.spec.ts exercises the actual CLI through creation, role requests, seeding, browser opening and complete teardown.

Targets and shared behavior

The self-host release-image check runs against a prebuilt Docker image, outside the source-server targets. It covers first-admin setup, an npm-dependent app, encrypted account access, and retained login/app execution after replacing the container with the same volume. It also compiles an authored frontend with Tailwind and checks stored app data before and after replacement. It runs with explicit settings, local defaults, and Railway's domain/port variables plus a root-owned volume. It also checks non-root execution, generated key permissions, and refusal to replace missing keys. The explicit-settings scenario also checks allowed internal imports, preserved Host headers, and DNS rejection before any connection, including redirected imports. It clones and pushes app source through the image's Git HTTP server, then checks the committed files through the workspace API before and after replacement.

A separate case serves the image at a tailnet-style origin. It creates a Docker network in 100.64.0.0/10, gives the container a fixed address there and maps nexus.example.ts.net to that address inside the container. BETTER_AUTH_URL uses that name, and EXECUTOR_APPS_ALLOW_PRIVATE_FETCH is unset. After first-admin setup, an API key calls the built-in Executor app through /mcp. An authored app then checks that Executor refuses the container's private address by name, and that the image's public-only network refuses it behind a public name mapped to it, which no name check catches. This fork's self-host serves an organization-only registry from its own database and reads no remote catalog, so the case does not run upstream's remote-registry failure checks; the release case instead checks that a publication survives a restart and that /api/registry/* is not served. The runner reaches the server through a port published on 127.0.0.1 in the range 4431-4439. It sends each request with the tailnet Host header through node:http, because Node's fetch replaces that header. This works with Docker Desktop, OrbStack and Linux Docker. The case removes its container, network and anonymous volume:

EXECUTOR_E2E_DOCKER_IMAGE=<image-tag> EXECUTOR_E2E_DOCKER_ARCH=arm64 \
EXECUTOR_E2E_DOCKER_VERSION=<commit-sha> \
  bunx --no-install vitest run --config e2e/docker-release.config.ts

Use amd64 when checking that image architecture. The scenario creates and removes its own container and volume. It does not publish the image. Build with --build-arg EXECUTOR_BUILD_VERSION=<commit-sha>. The image embeds this identity in both dashboard assets and the server environment. The release check requires the same version in a delivered server trace before and after restart. To check an upgrade, pull the previous image and set EXECUTOR_E2E_DOCKER_PREVIOUS_IMAGE=<previous-tag>. The scenario creates data with that image, replaces it with EXECUTOR_E2E_DOCKER_IMAGE, and checks retained login, encrypted credentials, app data, frontend availability and execution. It reads the previous build version from the image and checks the new version after replacement.

The same config runs docker-oauth-renewal.spec.ts against the image. It shares the runner's network so the container reaches a loopback token endpoint that rotates refresh tokens. A slow renewal holds its claim while other calls wait, a renewal survives its caller disconnecting, and docker kill mid-renewal followed by an immediate docker start recovers the grant within an execute deadline.

Command	Target	Current coverage
bun run e2e:self-host	Fresh Node/PGlite self-host	Shared hosted scenario, password login, account volume, Motel
bun run e2e:local	Fresh Node/PGlite local product	Pairing, replay rejection, dashboard access and reload
bun run e2e:cloud	Managed local Cloud Worker + Postgres, or explicit attached URL	Cloud onboarding, shared hosted behavior and MCP
bun run e2e:parity	Self-host and Cloudflare	Identical role/account and MCP scenarios on both
bun run e2e:all	All three	All applicable tests in one combined report

With no E2E_CLOUD_URL, the Cloud target starts the real Alchemy Worker and a throwaway Postgres container on fresh ports. It provisions its external services through emulators.dev and generates its own temporary database/auth values. It starts with a sealed environment and an empty Alchemy profile directory. No Cloudflare, PlanetScale, Google, GitHub, Context.dev, or 1Password credentials are needed. Docker must be running; Bun, Playwright Chromium and ffmpeg are normal tool prerequisites.

The run waits for the Worker's /health to answer 200. It fails at once when Alchemy reports that a resource could not start, and otherwise after ten minutes. Each probe has its own ten-second limit. The Worker's output is in the run directory's cloud/cloud.log.

The run removes its Postgres container when it ends, including after a failed or interrupted start; under load Docker can finish creating a container long after the run has stopped waiting for it. Each container is labelled with executor.e2e.role=cloud-postgres and the process ID of the run that owns it, with the process namespace that ID belongs to (executor.e2e.pid-namespace: the boot and PID namespace on Linux, the boot session on macOS). A run killed before its cleanup leaves the container behind; the next Cloud run in the same namespace removes containers whose owning process has exited and that are more than five minutes old. Containers of live runs, of other namespaces (such as another devcontainer sharing the Docker socket) and unlabelled containers are never touched. A run that cannot determine its namespace records none and removes nothing.

The disposable Postgres server allows 512 connections. The local Worker connects directly, so concurrent requests and background jobs cannot share a pooler's backend connections. PostgreSQL's default 100 slots can reject parallel scenario startup. This capacity setting applies only to the managed test container.

Setting E2E_CLOUD_URL explicitly attaches to that server instead. A failed attached target stays failed; it does not fall back to a local instance. The report identifies the origin, managed/attached mode and local Worker runtime. Attached role tests require the runner-owned E2E_FIXTURES loopback capability; attached onboarding uses the stage's generated emulator fixture. Use bun run e2e:deployed to let the runner own deployment and teardown automatically.

tests/hosted-shared.spec.ts contains one Effect program with no target branches in its assertions or UI steps: signed-out rejection, owner/admin/member permissions, deploying an app, account connection, tool discovery/invocation and both dashboard views. A scoped finalizer deletes only its created app/account, including on failure.

The SDK isolation scenario also runs on the disposable Cloud stage. Deployed runs use a scoped Axiom reader for delivered telemetry. This is correctness coverage; it does not establish Cloud load capacity.

Neon stages connect to Neon's pooled endpoint and PlanetScale stages to their branch's PgBouncer, both directly over verified TLS. Realistic concurrent CI coverage of that path is tracked in #508.

MCP server scenarios

tests/claude-mcp.spec.ts drives the actual interactive Claude Code application. It starts with an unauthenticated MCP URL, enters /mcp, selects Authenticate, and follows the browser request opened by Claude. Self-host signs in through the password form. Cloud uses an already signed-in synthetic browser session. Both approve the organization, return to Claude's own loopback callback, verify its successful connection, and invoke a tool in that same terminal session. A random receipt absent from the prompt proves a real invocation.

The client's configured OS browser handler forwards its actual authorization URL to the recorded browser. Claude performs discovery, client registration, PKCE, the code exchange, and credential storage. The test never supplies an MCP token or constructs the authorization request for this scenario.

tests/mcp-server.spec.ts separately checks the public OAuth/MCP protocol with the official SDK client: anonymous rejection, browser consent, discovery, tool execution, refresh and revocation. Both scenarios run unchanged on self-host and Cloudflare, with target-specific browser sign-in supplied by an injected adapter.

Install Claude Code and configure its model API explicitly. We use VibeProxy's Anthropic endpoint. The runner never falls back to a personal Claude OAuth login. Supply E2E_CLAUDE_BASE_URL and E2E_CLAUDE_API_KEY through the environment or a credential launcher; never put a real key in a command argument. Optional E2E_CLAUDE_COMMAND defaults to claude; E2E_CLAUDE_MODEL defaults to claude-sonnet-4-6. The interactive tool response has a 90-second wait limit.

# With model API variables supplied by your private launcher:
bun run e2e:self-host --test-name 'Claude Code connects'
# Supply the Cloud environment attachment for both hosted targets:
bun run e2e:parity --test-name 'Claude Code connects'

Claude runs interactively with --bare, an explicit VibeProxy model API, no built-in tools, and its own temporary configuration directory. First-run UI choices and MCP authentication are driven through Terminal Control. Its private configuration and cached credentials are removed on exit, and the test revokes its server grant. One recording follows terminal → browser → terminal from the actual driver calls. Original clips, the activity timeline and the edit plan remain in supporting evidence. Recorded test and request durations do not change. OAuth cases discard Playwright's network trace because it contains credentials; video, screenshots, sanitized navigation, protocol metadata and request timings remain. Protocol revisions in mcp-*.json describe the official SDK client. claude-client.json records the actual CLI version, model, permission mode and model API.

tests/local-claude-mcp.spec.ts runs the same Claude Code journey against Local. Local has no organization, so its consent adapter pairs the synthetic operator through /auth/pair and /auth/exchange, approves the client on Local's /mcp/authorize page and revokes the grant afterwards. CI excludes both Claude scenarios because it holds no model API key:

bun run e2e:local --test-name 'Claude Code connects'

Executor's native tool-policy approval and browser tool approval remain separate coverage. The SDK protocol scenario remains hosted-only. App and grant cleanup uses public endpoints. Anonymous OAuth client registrations remain on the dedicated stage because the product has no public deletion flow; cleanup evidence calls this out. No active grants are deliberately retained.

Evidence

The runner prints the saved run directory and an explicit render command. It keeps raw case results, screenshots, browser/terminal captures, traces, Vitest JSON results, and a combined evidence.json manifest. No rendering runs automatically, even on failure. Test failures and empty filters still fail the test command.

When evidence needs review, render the retained run, then serve the report on loopback. Rendering needs Playwright Chromium installed, plus ffmpeg and ffprobe on PATH. The command builds the viewer assets. It does not start servers, deploy, provision accounts, or need credentials. It can run after the test environment is destroyed.

bun run e2e:render --directory .local/e2e/<run>
bun run e2e:report --directory .local/e2e/<run>/report

Each case's telemetry.json holds the delivered spans of its five slowest requests and its last five browser traces. A failed case also keeps the trace of every request it sent: from the test, its background fibers and its cleanup, answered or not. A case fails when its body fails or when its own cleanup does, so a finalizer that fails after the body passed fails the evidence too. The flush waits up to five seconds for the server to export the spans that answered those requests. Each entry's kept lists why it was kept.

The failure traces have a budget of 64 MiB of compact JSON (FailureTraceBudget). The largest case, the 1,000-account inventory load, sends about 4,000 requests whose traces take 47 MB; most cases take under 4 MB. Past the budget, the oldest traces are left out. The newest request, every request that never answered and the five slowest are always kept, so the budget is soft: those traces are kept even when they alone exceed it.

Exporting and reading traces stops after 25 seconds, so the evidence finishes inside the 60-second cleanup hook and the product still stops. The evidence writes result.json and trace-ids.json before it reads any trace. trace-ids.json lists every request newest first with why its trace is kept, every answered request's trace, the failure traces kept, the dropped ones and the unfetched ones, and its state: collecting until the end, then complete or partial when the deadline stopped the reads. telemetry.json is written a batch at a time, one compact entry per line: first the newest request, then the other traces always kept, then the rest newest first. A trace not read before the deadline has an entry with only an error. unanswered-requests.json lists every request that never answered. failure-evidence.spec.ts checks this with cases that fail on purpose.

For CI failures, download and extract the evidence artifact, then pass the extracted run directory (the one containing evidence.json and target folders) to e2e:render. Keep the target folders together. Rendering uses relative paths, so it does not depend on the CI workspace path. Use the checkout at the run's recorded commit. The SDK also exports renderSuiteEvidence; SDK callers must build the viewer assets with bun run e2e:viewer:build first.

All assets and media use relative links. The React viewer opens with a searchable results list, status/target filters and 50-row pages. Each shared scenario appears once, with a separate result and duration for each target. Selecting it leads with its recording or request evidence. Target controls switch between that scenario's recordings without duplicating it in the sidebar. The target's exact URL remains visible. Only the selected test loads media. The viewer follows the system's light or dark theme, including native controls. While dragging the seek bar, a thumbnail strip and timestamp appear above it. Releasing the scrubber hides the strip. It overlays the footage so the scrubber does not move. The exporter generates the eight-frame overview from the final composed video, so it follows the same edit as playback. It loads only for the selected test. Generation happens only during the explicit render command. Existing reports without thumbnails retain the plain seek bar.

The player includes keyboard seeking, five-second back/forward buttons, playback speed and fullscreen controls. Controls stay below the footage. The report server supports byte-range requests for seeking without downloading the entire recording first.

Local and CI runs use no artificial action or reading delays. Raw evidence is still captured. For a deliberately slower recording, set E2E_RECORDING_PACE_MS (0–3000). Playwright instruments individual actions, including multiple actions inside one Browser.use call. The shared driver owns reading pauses; tests do not add sleeps.

# Full-speed run with normal evidence capture
bun run e2e:cloud --test-name 'Cloud onboarding'
# Deliberately slower recording for review
E2E_RECORDING_PACE_MS=500 bun run e2e:cloud --test-name 'Cloud onboarding'

Reading pauses last twice the configured action delay. This only changes capture drivers; API calls, server configuration and evidence encoding are never paced. recording-pacing.json records the setting. Test durations include these deliberate waits, so use unpaced runs for timing comparisons. Request and server span durations still measure the actual requests. Cancellation interrupts pending reading pauses.

Recording focus is automatic in the driver adapters. Browser operations, checkpoints and actor changes select the browser; opening or using a terminal selects that terminal. Overlapping calls use the newest call's window. An older call completing does not change focus, and saving artifacts during cleanup does not select a window. The exporter follows recording-timeline.json, so a test can switch between windows any number of times without phase markers. It trims idle tails and leading blank terminal startup, holds short terminal results for reading, and retains the full source captures. Terminal Control exports include startup so their time axes stay aligned with the browser capture.

Tests save raw terminal captures and close their live sessions. Terminal video encoding, browser address-bar rendering, and composition run only when requested with e2e:render. The render command reports processing time separately. Each recording has an evidence-processing.json attachment. Those times are excluded from the test duration; raw capture and normal cleanup still belong to the test's resource scope. Export failures fail the render command and retain the raw captures for diagnosis. They do not change the saved test result.

N/A means the test is explicitly irrelevant to that target, with the reason available on hover. Not run means the test is relevant but has no result, or requires infrastructure that was not available. For example, cloud scale stays Not run until dedicated capacity exists. Recorded skipped results remain skipped. Applicability comes from the scenario plan used by the runner and is saved in each report; missing evidence alone never becomes N/A.

Each case saves steps, request status/timings, trace IDs, latency percentiles, checkpoints, a Playwright trace and a video when it opens a page. Failed and interrupted cases retain evidence. Native Vitest JSON results remain linked from the report. A failing target makes the test command fail after saving raw results.

On explicit rendering, an Effect operation adds a 72px address bar with 28px text above the recorded page. Query strings/fragments are omitted. Playwright renders the bar images and scoped ffmpeg processes compose the MP4; the tested page is never modified. Raw video and trace artifacts remain in the case directory.

The server log and complete Motel database remain under each target's run folder. Saved trace samples are not a full trace archive. The telemetry test checks actual server spans, status and the presence of SQL spans; client spans cannot satisfy it. While running, data/diagnostics/collector.json identifies the collector query URL.

Managed Cloud's collector is four Motels behind one router (serveShardedOtlpCollector in support/otlp-collector.ts). One Motel stores about 1,500 spans a second, and six workers provisioning their organizations export more than that; a single Motel refused thousands of exports a run, which the product dropped once its three-second export budget ran out. The router splits each export by trace ID, sends a trace's reads to its Motel, and merges searches across all four. Each Motel keeps its database in data/diagnostics/motel-<n> and its log in collector-<n>.log; GET /api/ingest on the router lists each one's refused exports.

All runs use synthetic identities. Private run directories and session files are ignored by Git. They can still contain session cookies and should not be published.

State storyboard

Use E2E_UI_OBSERVE=1 bun run e2e:cloud --test-name 'Cloud onboarding' to capture loading and error states as direct screenshots. The viewer supports arrow-key stepping, journey selection, previous-frame overlays, side-by-side comparison and movement highlights. Browser layout-shift events after recent input are retained.

This opt-in managed Cloud capture pauses API requests and driver actions for screenshots. Its durations include those holds; normal runs do not. Every observed candidate is retained, with an explicit status if it changed before capture. See the storyboard guide for the protocol, comparison rules and coverage limits.

Pause, take over, resume

bun run e2e:inspect --test-name 'hosted roles'

The headed browser pauses at named checkpoints. Use that same browser normally, then press Resume in Playwright Inspector. Server state, cookies, recording and telemetry remain live. Failed interactive cases pause before teardown; cancellation closes the owned scopes. Interactive cases are annotated as manual intervention and do not count as unattended passes. Process-crash restoration is not implemented.

Cloud actors

Cloud onboarding with emulators.dev

Run the complete onboarding slice without environment files or account secrets:

bun run e2e:cloud --test-name 'Cloud onboarding'

This is the existing E2E suite and evidence pipeline. The runner starts a real local Cloud Worker plus Postgres and creates isolated Google, GitHub, Resend, Context.dev and Autumn emulator instances. Company lookup has deterministic matched and unmatched domains; the Google case verifies the actual suggested company name before editing it. No real company-lookup key is used.

Each onboarding case begins signed out. Google and GitHub complete OAuth token exchanges; email retrieves its actual delivered code from the mail emulator. Passkeys use browser WebAuthn and real server registration/assertion endpoints with a virtual authenticator. They do not exercise a native password-manager sheet. Ordinary role fixtures use the runner-owned fixture process, which keeps setup traffic out of the email sign-in rate limit. Onboarding cases never use those prepared sessions.

The local test origin is http://localhost:<port>. Browsers treat localhost as a secure context for WebAuthn, so the run needs no installed certificate or key. No TLS verification is disabled. Real external services still use verified HTTPS. The Worker is the real Cloud entry point; only external service configuration and resource lifetimes vary. No alternate auth server is constructed.

The runner ignores inherited infrastructure credentials, sets CI=true, and uses an empty per-run ALCHEMY_HOME. Alchemy's local Worker, R2 and Hyperdrive providers use a fixed local account in CI and never resolve cloud credentials. Generated test credentials are ephemeral, not personal or production secrets. On completion the runner stops the Worker, removes its Postgres container, removes the emulator credential file and resets its external emulator instances. Recordings remain in the report.

These Cloud-only scenarios have explicit N/A reasons on self-host and Local. Their recordings cover provider selection, company loading, edited team confirmation/retry, passkey enrollment and returning sign-in after enrollment or Not now. The loading-state case delays the original network request; it neither replaces the response nor presents that delay as server latency.

For an independently deployed E2E stage, provision its external configuration:

node e2e/create-emulators.ts --origin https://e2e-your-stage.executor.engineering \
  --output /absolute/private/path/onboarding-emulators.json

The deployer supplies that file's services object as EXECUTOR_EMULATORS when updating the dedicated stage. The mode is permitted only for loopback Cloud dev or test-e2e-* stages; production origins reject it. It does not resolve real social-provider, email, billing, company-lookup or OAuth proxy credentials. Google signature, issuer, audience and nonce verification remain enabled.

E2E_CLOUD_URL=https://e2e-your-stage.executor.engineering \
E2E_EMULATORS=/absolute/private/path/onboarding-emulators.json \
  bun run e2e:cloud --test-name 'Cloud onboarding'

That optional file contains generated emulator capabilities, not infrastructure credentials. It is created with mode 0600 and never overwritten. Keep it private. Provider emulation does not claim coverage of Google's/GitHub's live UI or native Cloudflare mail delivery. The separate interactive Claude journey still takes its model-endpoint configuration documented above; onboarding does not need it.

Signed-in actors for other hosted tests

Use bun run e2e:deployed --test-name 'Testing SDK|hosted roles' --workers 2. The runner deploys one dedicated test-e2e-* stage for the whole suite. After migrations, a local Alchemy command transfers its database and signing settings to the runner's authenticated fixture process. The setup validates the exact stage origin, branch, database, and username. Every case then receives new identities with one-hour sessions. Tests never receive infrastructure credentials.

Cloud cleanup restores cleanup authority for only its reserved synthetic organization, calls the real product removal endpoint, waits for removal, then deletes its synthetic users. This also handles partially failed provisioning and membership changes. The enclosing deployment scope removes its stage even when setup or tests fail. Existing stages are outside that scope.

For an anonymous check, E2E_CLOUD_URL alone is sufficient with --test-name 'cloud endpoint'. Do not pass an old shared actor file to a suite.

Workflow durability during a host deployment

The confirmed-write timeout scenario first commits a control mutation. Its second mutation inserts and reads a row inside the transaction, then starts a separate workflow as a durable observation before waiting beyond its timeout. The scenario requires that observation and checks that the row remains absent after the authored body would otherwise have returned.

The sleep scenario defaults to one second. On a dedicated deployed stage, set E2E_WORKFLOW_HOLD_MS=180000 to provide a three-minute host deployment window. Interactive mode disables the normal 60-second test limit for this manual check:

# Supply the running environment attachment as described above.
E2E_INTERACTIVE=1 E2E_WORKFLOW_HOLD_MS=180000 bun run e2e:cloud --test-name 'workflow sleep preserves'

Wait for Workflow sleep window and inspect the native Cloudflare instance to confirm an unfinished sleep before deploying the same stage through Alchemy. The case saves sleeping-workflow.json with the run ID. Capture the Worker deployment and instance version before and after deployment, while the sleep is still pending. After completion, completed-workflow.json identifies the original run and a fresh run. Require a changed Worker deployment before the sleep deadline, an unfinished sleep after that deployment, and successful completion of both runs. The HTTP assertions require exactly one mutation before and after each sleep. Record native workflow IDs separately: the workflow versionId stayed unchanged across the verified Worker redeployment and cannot be used as its Worker code version.

The ordinary attached suite does not redeploy its host during a scenario. A passing sleep scenario alone does not prove a host deployment overlapped it; retain the provider timestamps and version evidence with the report. The runner only forwards the bounded hold duration, never deployment credentials, into the test process.

Existing failure

The original 1,000-account run preserved every account and selection but lost later server telemetry before it reached Motel. That delivery assertion remains enabled; this runner migration does not relax its timeout or reduce the default data volume.

Driver references

Anomaly's Terminal Control, installed as @kitlangton/terminal-control, supplies the Claude PTY and terminal recording. Its bundled native binary exports MP4 through ffmpeg. Browser Control was reviewed for existing-profile browser adoption and human handoff; browser tests currently use isolated Playwright contexts.

OAuth URL policy regression

bun run e2e:self-host --test-name 'OAuth setup honors host URL policy' exercises the public hosted account routes. The managed self-host uses a named .localhost callback with a static query parameter and one explicit HTTP origin exception. The scenario checks HTTPS, the permitted HTTP origin and a denied different port. It starts authorization with a synthetic manual client; it does not contact a live provider. SDK protocol tests separately cover discovery, registration, code exchange, refresh and callback parameter tampering through the production transport seam.

Authored app observability

app query traces connect browser, streamed host work, runtime and React commits uses a real app, checks two results while its stream is open, then closes it and requires a complete parent graph. It checks linked source maps without embedded source text and preserved drafts. Separate scenarios check warm query timing, a failed subscription followed by a linked retry, and live access revocation. Each scenario owns its app and organization. The retry scenario requires both the failed attempt and its native link to reach the collector.

Self-host reads actual delivery through Motel. For a dedicated deployed Cloud stage, bind E2E_AXIOM_TOKEN through the credential launcher and set E2E_AXIOM_DATASET=executor-next-test-traces together with the normal attached stage URL and private synthetic actor file. The query adapter reads only the validated trace ID in the current run's time window. Personal Axiom tokens also require E2E_AXIOM_ORG_ID; dataset-scoped API tokens do not. Partial or truncated results fail; missing parents are never replaced by synthetic success records.

Framework authoring and optimistic UI

framework discovery deploys its checked example with optimistic updates and rollback reads the built-in app through MCP, checks native and imported tool output signatures, follows a pinned skill topic, and deploys the example returned by framework.describe. It holds the real write and reconciliation read at the browser boundary, checks the optimistic row and draft, rejects a later write, then retries and reloads persisted data. Run it with bun run e2e:self-host --test-name 'framework discovery deploys'.

bun run e2e:local --test-name 'local MCP skills' checks local framework queries and topic routing alongside configured copies and pinned versions. Package tests cover queued writes, argument variants, read failures, unmounts and disposal.

closing an app warns about queued optimistic deletes until writes settle holds writes and reconciliation reads while deleting three stored rows. It checks the native close-tab warning for active and queued writes, dismissal, failure cleanup, and safe closing after the last acknowledgement. Read-only reconciliation must not retain the warning. The final API read verifies that all three deletions persisted after the tab closed.

Real Cloudflare environments

bun run e2e:deployed creates one disposable Neon-backed Cloudflare environment, runs the ordinary Cloud scenarios, and destroys it even on failure. Supply the staging credentials through the approved launcher. Use --database planetscale for the release checks, or --test-name '<scenario>' for focused verification. Alchemy disables Better Auth rate limits on automated test-e2e-* stages so parallel scenarios do not share the runner IP's request allowance. Set TEST_STAGE_AUTH_RATE_LIMIT=true when deploying a stage for focused rate-limit checks. Managed local Cloud does the same: the runner starts it with TEST_STAGE_AUTH_RATE_LIMIT=false, which Cloud honors only under alchemy dev on a loopback origin. bun run e2e:cloud --auth-rate-limit starts it with the limit on and runs only the scenarios whose Cloud plan declares runtime: "rate-limited"; CI runs them in their own step. Likewise bun run e2e:cloud --rollback starts it with Cloud's rollback switch on (CLOUD_BROWSER_ORIGIN=deployment, sign-in on the deployment origin) and runs only the scenarios declaring runtime: "rolled-back". Managed local Cloud runs with Better Auth's OAuth proxy on, as the proxy's production (its edge), as production does. bun run e2e:cloud --oauth-proxy-preview also starts a second local Cloud as that production and tests a stage that signs in through it, running only the scenarios declaring runtime: "oauth-proxy-preview". Production (v2), ordinary previews and every other deployed stage always retain rate limits, whatever their configuration; bun run auth:rate-limit (part of check) proves it. Local and Cloud CI jobs use 16-vCPU runners. Self-host functional tests use a 12-vCPU Mac; the 1,000-account test runs independently on a 6-vCPU Mac. The same tests run on both providers. Add new Cloud scenarios normally in test-plan.ts; no deployment fixture belongs in a scenario.

The deployed runner provisions the selected actor fixtures together before starting Vitest. Each run gives each scenario a new 128-bit identity. The runner then checks the actual HTTPS domains, including normal certificate verification. Certificate issuance belongs to environment preparation, with a five-minute limit, and is reported separately from test time. Per-scenario setup, assertions, and cleanup retain their separate 60-second deadlines. The suite owns all prepared organizations, including those whose tests never start after a failure.

Only scenarios declaring runtime: "managed", "rate-limited" or "rolled-back" require the local Cloud target (for example, local telemetry collectors). Their deployed report says N/A with the reason. They run only in the Cloud jobs of local CI, so each must match a job pattern in ci-selection.ts; the select job fails on a scheduled scenario that no CI job runs. Claude Code's model-dependent scenario is excluded by the deployed runner's default filter. A filter that executes no scenarios is a failure. See test stages for retained previews, shared infrastructure, background pause/resume, and cleanup.

Scenarios declaring runtime: "attached" require a deployed Cloud target. The compiler memory scenario uses large, pinned npm dependencies to exceed the real compiler Worker's memory limit. It checks the typed error, the unchanged active version, and a subsequent successful build. Local workerd does not enforce that memory limit, so its report marks this scenario N/A. Run it alone with bun run e2e:deployed --test-name 'Cloud compiler memory failures' --workers 1. The default deployed filter excludes it because exhausting the shared compiler can interrupt other scenarios' builds.

The deployed job runs after the functional checks on main. PRs run the emulated Cloud target. Main and manual deployed CI jobs share one non-cancelling concurrency group; scenario workers remain parallel within each job. Agents can still run targeted disposable deployments through this CLI. The job retains raw evidence and destroys its own environment.

Installed CLI artifact

To verify the installed CLI artifact through the same local scenarios, set EXECUTOR_E2E_LOCAL_ENTRY to the absolute path of the installed executor executable (executor.exe on Windows), from the platform archive bun run release:cli builds, and run bun run e2e:local. The harness starts that entry from its install directory, with an isolated data directory and synthetic secrets. Pairing, dashboard loading and app deploy/call use real HTTP requests against the installed package. A JavaScript entry, such as the source bin.ts, runs under Node instead.

The release executable runs no preload, so the local CLI owns the runner's two process hooks: EXECUTOR_TEST_CLOCK_OFFSET_MS with EXECUTOR_TEST_CLOCK_CONTROL=stdin for its wall clock and EXECUTOR_TEST_STOP_FD=3 for stop requests, honoured only with EXECUTOR_ENVIRONMENT=e2e (apps/local/server/src/implementation/e2e-process.ts). Scenarios that replace a package the executable loads from disk, such as the OS credential store binding, start it from a shadow copy of its install (support/stand-in-package.ts).

The first-launch key scenarios also run against an installed entry:

EXECUTOR_E2E_LOCAL_ENTRY=/path/to/node_modules/executor/executor \
  bunx vitest run --config e2e/local-bootstrap.config.ts

The OS credential scenario uses the real store and removes only its own entry. The key file and denied-access scenarios never touch the real store. They start the CLI with a stand-in keyring module that reproduces the package's errors: an absent store, a cancelled or dismissed prompt, and a store that grants access. Only an absent store may fall back to keys.json. The key storage scenario covers EXECUTOR_KEY_STORAGE: file on a new or denied-pending directory, no-ops on matching directories, refusals on mismatched ones, and invalid values. The pair scenario pairs with a running server whose key is in the stand-in store, and refuses denied, absent, missing and unfinished keys without writing anything. On Linux outside a D-Bus session, set EXECUTOR_E2E_CREDENTIAL_STORE=absent to use the real missing Secret Service for the key file scenario instead; release CI runs both.

The npm launcher scenario unpacks the launcher and native archives as npm lays them out, then starts --version through every launcher entry point, including Executor 1's bin/executor, which stale command shims still start. It checks that the launcher passes on exit codes and SIGTERM, explains a missing platform package, and, with EXECUTOR_E2E_OLD_NODE, starts on the oldest Node its package accepts:

bun run release:wrapper
EXECUTOR_E2E_LAUNCHER_ARCHIVE=.local/releases/<version>/wrapper/executor-<version>.tgz \
EXECUTOR_E2E_RUNTIME_ARCHIVE=/path/to/executor-<version>-<platform>-<arch>.tgz \
  bunx vitest run --config e2e/npm-launcher.config.ts

Two more release suites take the executable or its archive. standalone-lockdown.config.ts starts the executable with every Bun override a local process could set, from BUN_OPTIONS to a bunfig.toml in its working directory, and requires that none takes effect. installer.config.ts runs apps/docs/public/install.sh (or install.ps1) against a local copy of a release download:

EXECUTOR_E2E_LOCAL_ENTRY=/path/to/executor \
  bunx vitest run --config e2e/standalone-lockdown.config.ts
EXECUTOR_E2E_RUNTIME_ARCHIVE=/path/to/executor-<version>-<platform>-<arch>.tgz \
  bunx vitest run --config e2e/installer.config.ts

The desktop artifact smoke uses the packaged executable, synthetic secrets and a fresh profile/data directory. It deploys a dependency-using app, calls it and closes the app. The background server keeps running; the CLI the desktop staged reports and stops it. The scenario then calls the retained app after the desktop starts the server again:

EXECUTOR_E2E_DESKTOP_EXECUTABLE='/path/to/Executor Preview.app/Contents/MacOS/Executor Preview' \
  bunx vitest run --config e2e/desktop-release.config.ts

Verify a locally built image without pushing it:

EXECUTOR_E2E_DOCKER_IMAGE=executor-next-release:preview \
EXECUTOR_E2E_DOCKER_ARCH=arm64 \
  bunx vitest run --config e2e/docker-release.config.ts

Use amd64 on an amd64 runner. The test checks the actual architecture, uses the normal image command, and removes its own synthetic container and volume.

CI runs a separate cleanup step even after cancellation. It reads only the stages created in that job, skips confirmed destruction, and retries incomplete teardown. The registry lease remains the fallback if the entire runner is lost.