Optimized Evaluations

[DerDennisOP]

Full overhaul of Gradient's evaluation workers into a native, parallel, cache-controlled evaluator.

Today every evaluation runs a subprocess EvalWorker/EvalWorkerPool driving a single NixEvaluator over the Nix C API (nix-bindings), with a tryEval-based wildcard resolver, no parallelism across workers, and no control over the Nix eval-cache, it lives per-worker (embedded SQLite) and is thrown away on recycle, so every evaluation re-evaluates from cold and big evals blow up memory.

We want:

• Full control over the evaluation cache.
• The ability to split one flake's evaluation across multiple workers, with the eval cache cached into the Gradient instance so workers share it instead of each re-evaluating the same flake from scratch.
• A native parallel evaluator (no external binary) that isolates per-attribute failures and reclaims memory without restarting the whole process.
• Native evaluation stats/progress (no frontend in this issue, storage + proto only).

This is a from-scratch overhaul: expect breaking changes across gradient-proto, the database, and the API. Optimal results matter more than compatibility.

Builds on / references:

• Wildcard flake resolver: tryEval doesn't catch 'attribute missing' errors during evaluation #139: builtins.tryEval doesn't catch attribute missing / type / infinite-recursion / abort / I/O failures; a native per-attribute evaluator isolates these.
• Support nix-eval-jobs NotAShelf/nix-bindings#39: native parallel-evaluator primitives (ParallelEvalBuilder, per-thread EvalState, shared work queue, Receiver<JobResult>, memory-threshold restart, GC roots) and the C API expansion (libexpr-c: expose essential evaluator functions for derivation tree inspection NixOS/nix#15842 / C++ shim in nix-bindings-sys).
• Architectural models to reimplement natively in Rust: NixOS/nix-eval-jobs (parallel attrset traversal, --check-cache-status, --max-memory-size, GC roots) and NotAShelf/nix-evaluator-stats (eval stats from NIX_SHOW_STATS / NIX_COUNT_CALLS).

Detail per area in the comments below.

[DerDennisOP]

1. Native parallel evaluator (retire the subprocess pool)

Replace the per-eval subprocess model: EvalWorker, EvalWorkerPool, PooledEvalWorker, run_eval_worker, and the line-delimited EvalRequest/EvalResponse/ResolvedItem protocol, with a native multi-threaded evaluator in gradient-worker built directly on the nix-bindings C API, following the ParallelEvalBuilder pattern from NotAShelf/nix-bindings#39:

• A pool of threads, each owning its own EvalState (the C API state objects are independent and thread-safe across instances), instead of one NixEvaluator per subprocess.
• A shared concurrent work queue of attribute paths; results streamed over a Rust channel (Receiver<JobResult>) rather than one-request/one-response JSON round trips.
• Per-thread memory accounting (e.g. getrusage) with restart above a threshold, replacing the coarse whole-subprocess evaluations_served / max_evaluations_per_worker recycle. Boehm GC heap is reclaimed per worker thread rather than by killing the process.
• GC-root registration per evaluated derivation (nix_store_add_perm_root) so eval roots survive concurrent garbage collection.
• Keep DerivationResolver as the seam (list_flake_derivations, resolve_derivation_paths, get_derivation, get_features) but its production impl becomes the native pool; WorkerPoolResolver is reworked or retired.

Targets the "memory explodes / restart after every evaluation / no parallelism" trifecta from nix-bindings#39. A warm EvalState can preload common nixpkgs library state so repeat evals don't re-thunk the basics.

[DerDennisOP]

2. Per-attribute failure isolation + thunk probing (#139)

Retire the tryEval-based wildcard BFS (the resolve / safeAttrNames / safeGet / isDrv Nix functions). Native traversal via nix_get_attr / nix_get_type evaluates each attribute path independently, so an attribute missing, type error, infinite recursion, abort, or builtin I/O failure on one attr produces an error JobResult for that attr and does not abort the whole evaluation, the core of #139, which builtins.tryEval cannot deliver (it only catches throw / failed assert).

• Probe nix_get_type before forcing a value so known-non-derivation thunks aren't forced unnecessarily (the "know if something is a thunk/drv before evaluating into it" ask from nix-bindings#39).
• DiscoveredDerivation and the EvalResult variant of JobUpdateKind carry per-attr errors instead of silently dropping branches; evaluation_message rows keep the nix-eval:<attr> source granularity.
• Removes the bumped 64 MiB eval stack as a correctness crutch, deep builtins.match / builtins.split recursion is bounded per worker thread, not papered over process-wide.

[DerDennisOP]

3. Gradient-controlled, shareable evaluation cache

Take full ownership of the Nix eval-cache. Today it is the per-worker embedded SQLite cache: unshared, and discarded when a worker is recycled, there is no cross-eval or cross-worker memoization, so identical source + wildcards are re-evaluated from cold every time.

Make the eval cache a first-class Gradient artifact:

• Keyed by flake source + lock (nixpkgs rev + resolved inputs). Workers pull the warm cache from the Gradient instance before evaluating and push their updated cache back after, so a flake evaluated once is not re-evaluated cold on the next worker/eval.
• Integrate a cache-status probe (nix-eval-jobs' --check-cache-status: Local / Cached / NotBuilt) so DiscoveredDerivation reports whether outputs are already substitutable, feeding the substituted decision without a separate pass.
• New gradient-proto push/pull messages plus DB entities: an eval-cache store, and a cache-status column on derivation. The existing FlakeSource::Cached { store_path } archived-source path generalizes into this mechanism.

This shared, instance-resident cache is the prerequisite for splitting one evaluation across workers (comment 4) and for preloading common library state (comment 1).

[DerDennisOP]

4. Splitting one evaluation across multiple workers

Today WorkerPoolResolver round-robins attribute resolution within a single worker's local pool. We want one flake's attribute-path space sharded across multiple eval-capable workers (worker_registration.enable_eval, GradientCapabilities.eval).

The shared eval cache (comment 3) is the enabler: each worker pulls the warm cache, evaluates only its shard, and streams partial EvalResult batches back, no redundant cold flake evaluation per worker.

• FlakeJob gains shard / partition addressing so dispatch can fan a single evaluation out to N workers.
• The scheduler's handle_eval_result merges shards into one evaluation with its derivation / derivation_dependency / entry_point rows, deduping across shards.
• Completion is reconciled across all shards before the evaluation leaves the eval phases, extending the EvaluationStatus machine (Fetching / EvaluatingFlake / EvaluatingDerivation) and the per-phase phase_event rows and timing columns (eval_flake_started_at, eval_drv_started_at) to a multi-worker fan-out/fan-in.

[DerDennisOP]

5. Native evaluation stats & progress

Reimplement NotAShelf/nix-evaluator-stats' parser natively in Rust, reading the evaluator statistics directly through the nix-bindings C API rather than scraping NIX_SHOW_STATS / NIX_COUNT_CALLS env-var JSON: cpuTime, nrExprs, nrThunks, nrAvoided, GC cycles, env/values sizes, symbols, etc.

• Gradient records true eval cost per evaluation and surfaces real progress (done / total attributes) instead of the content-free BoardEvent::EvaluationProgress ping.
• Stream per-shard progress over a new JobUpdateKind; persist a per-evaluation stats row alongside the existing evaluation timing columns and phase_events.
• No frontend in this issue: storage + proto only; the frontend consumes it later.

[DerDennisOP]

6. Proto / DB / API surface & sequencing

Expected breaking surface (optimal > compatible):

• gradient-proto / gradient-types: rework FlakeJob (shard addressing, eval-cache pull/push), JobUpdateKind (per-attr errors, progress, stats), DiscoveredDerivation (cache-status), and the EvalServer / EvalClient traits (start_eval, abort_eval, on_eval_result).
• Database (gradient-entity / gradient-db / gradient-migration): new eval-cache store and eval-stats entities; cache-status and stats columns; phase/timing extensions on evaluation for multi-worker sharding.
• API: eval-cache management and eval-stats read endpoints, documented in the OpenAPI spec.

Sequencing / dependencies:

1. nix-bindings#39 primitives (libexpr-c: expose essential evaluator functions for derivation tree inspection NixOS/nix#15842 C API expansion / C++ shim).
2. Native parallel evaluator (comment 1).
3. Per-attribute failure isolation, Wildcard flake resolver: tryEval doesn't catch 'attribute missing' errors during evaluation #139 (comment 2).
4. Shared, instance-resident eval cache (comment 3).
5. Cross-worker evaluation split (comment 4).
6. Native eval stats (comment 5).

Reference implementations to mirror natively: NixOS/nix-eval-jobs (parallelism, cache-status, GC roots) and NotAShelf/nix-evaluator-stats (stats parsing).

[DerDennisOP]

Implementation

I read the nix-eval-jobs source. Almost everything good about it comes from process isolation, not from threads. The native-thread sketch in NotAShelf/nix-bindings#39 is appealing, but shows the thread model silently drops the three guarantees we actually want. We already have a process model (the EvalWorker subprocess), the overhaul should likely keep process boundaries and parallelise across worker processes, not collapse to a shared-process thread pool. Concretely:

A. Memory reclamation is process-scoped. nix-eval-jobs sets GC_DONT_GC=1, it disables the Boehm GC entirely and "does garbage collection by killing forks". Each worker checks getrusage(RUSAGE_SELF).ru_maxrss after every attr and, above maxMemorySize, tells the parent to kill+respawn it so the OS reclaims everything. In a single-process thread pool sharing one Boehm heap you cannot do this: RSS is process-wide (there is no per-thread RSS; RUSAGE_THREAD is CPU-only), so you can't attribute or reclaim one thread's memory. Either GC stays on (fragile, and we already know Boehm GC fights Tokio's signal handling, hence the blocking-context rule around NixEvaluator), or you keep process restarts. This is the #1 pain ("memory explodes") and it is solved by process death, not by parallelism.

B. The uncatchable failures from #139 are caught by the process dying, not by try/catch. attribute missing / type errors / throw / assert surface as catchable EvalError (the C API nix_err_msg path). But infinite recursion (stack overflow), abort, SIGSEGV/SIGBUS do not, nix-eval-jobs marks StackOverflowError .fatal and relies on the parent (handleBrokenWorkerPipe) detecting the worker crash, recording it as that attr's error, and respawning. In a thread pool, one attr's stack overflow or abort takes down all in-flight evaluations. So the #139 fix (comment 2) is only partially achievable with catchable C-API errors; full isolation needs a process boundary per risky unit of work.

C. Stacks. They couldn't use std::thread because it won't set stack size; default stacks (musl 128 KiB, macOS 512 KiB) overflow immediately, and Nix's 64 MiB main-thread stack doesn't propagate. Every evaluator thread/process needs an explicit large stack (we already bump to 64 MiB). N parallel evaluators × 64 MiB, platform-sensitive.

D. The shared eval-cache (our headline goal) has real concurrency + correctness hazards.

• The Nix eval-cache is SQLite. nix-eval-jobs pre-creates the eval-store once before spawning workers because concurrent creation races into "database is busy" / "schema is corrupt" (their issue Minor Frontend Issues #401). Our "cache into the Gradient instance, shared across workers" must serialise schema creation and use WAL / single-writer discipline, N workers writing one cache DB will contend.
• It is only valid for pure evaluation. Flake mode forces pureEval = true; --impure disables it. The cache key must encode purity + the full resolved lock (nixpkgs rev + all inputs); impure evals must not be shared.
• Memoising the attr→value mapping is not enough on its own: the .drv files it points at are store paths that must exist on the consuming worker. Sharing eval results across workers means shipping the eval-cache DB and the .drv input closure (or pulling it from our cache). Budget for that in the proto push/pull (comment 3).

E. Other concrete gotchas to bake into the design:

• One-time global init before fan-out: curl_global_init must run before any fork (an objc/curl quirk crashes otherwise); GC and the Nix subsystems (nix_libutil_init / nix_libstore_init / nix_libexpr_init) have ordering requirements. Init once, then spawn.
• IFD: they set builders = "" during eval because import-from-derivation + the build hook causes "unexpected EOF". IFD silently triggers builds mid-eval, we must control builder settings during evaluation.
• recurseIntoAttrs: descent is gated on the recurseForDerivations attribute. Blindly walking every attrset evaluates the world; our wildcard resolver must map onto recurseIntoAttrs semantics (and --force-recurse only when explicitly asked).
• GC roots need a writable LocalFSStore: addPermRoot / nix_store_add_perm_root is skipped in readOnlyMode and doesn't apply to a remote/binary-cache store, so the worker that registers roots needs a local writable store.
• --no-instantiate = readOnlyMode: faster, but writes no .drv and disables GC roots + derivation reads (so cacheStatus can't read drvs). We need the .drv to build, so the build-feeding eval can't use it; a discovery-only/cacheStatus pass could.
• Dynamic work set: the attr queue is grown during traversal (discovering nested attrs enqueues them) and each job re-resolves via findAlongAttrPath + autoCallFunction against a per-worker root value. Our cross-worker sharding (comment 4) must shard a traversal-discovered, not a static, work set and the expensive flake/nixpkgs base eval currently happens once per worker process, which is exactly what the shared cache must amortise.
• cacheStatus batching: nix-eval-jobs itself flags per-drv substituter status queries as a likely bottleneck ("TODO: batch these"). If we adopt --check-cache-status semantics, batch the substituter queries.

Bottom line: parallelism is the easy part; memory reclamation, crash isolation, and a correctly-shared eval cache are the hard parts, and the first two are process-level properties. A hybrid (a pool of bounded-lifetime worker processes, each possibly multi-threaded internally, with RSS-threshold respawn and a pre-created shared cache) keeps the nix-eval-jobs guarantees while adding the cross-worker sharing we want.

[DerDennisOP]

North Star architecture (the target we're building toward)

The asymmetric advantage over nix-eval-jobs is that Gradient has a database and a fleet: nix-eval-jobs is stateless per run, so it re-evaluates everything every time; Gradient can remember forever. The optimal solution is therefore not "a faster evaluator" but turning evaluation into fleet-wide persistent memoization where we almost never evaluate the same (flake, lock, attr) twice. Four levers, ordered by impact-per-effort. We're committing to all four; a true parallel evaluator (Tvix/Snix-class) is explicitly out of scope but the DerivationResolver seam keeps the door open.

1. Evaluation-as-memoization
(flake_fingerprint, attr_path) → { drvPath, inputDrvs, system, required_features, cacheStatus } is pure and immutable — the locked flake fully determines it. flake_fingerprint = hash of the locked-source store path + lock contents. Store it canonically in the instance. This is nearly what derivation, derivation_dependency, and entry_point already hold, just keyed per-fingerprint instead of per-evaluation. On a new evaluation we only evaluate the attr paths not already memoized for that fingerprint; everything else is a DB lookup. Typical CI commits touch a tiny attr delta, so most evaluations collapse to lookup + evaluate-the-delta, and the content-addressed derivation table already dedups overlap across orgs on the same nixpkgs rev. This is Gradient-native and is where the 10–100× lives.

2. Warm snapshot-fork workers
Evaluate the heavy base (nixpkgs/stdenv per target system) once into a warm eval state, then fork() per-shard workers that inherit the warm heap copy-on-write. Each fork is warm at ~zero cost, isolated (own address space → keeps the #139 crash isolation), and reclaimable (kill it → OS frees memory, with GC_DONT_GC). RSS-threshold respawn re-forks from the warm snapshot via getrusage(RUSAGE_SELF). This replaces the per-eval EvalWorker cold restart and beats nix-eval-jobs, which forks but never pre-warms before forking. NixEvaluator becomes the warm-snapshot owner; the fork pool sits behind DerivationResolver.

3. Gradient-resident shared eval-cache (needs nix-bindings / C-API work).
Expose the Nix eval-cache fingerprint + extract/inject so the canonical eval-cache for a (flake, lock) lives in the instance, ships to any worker, and grows incrementally as workers stream cache deltas back (new gradient-proto push/pull messages; the existing FlakeSource::Cached { store_path } generalizes into this). Two layers stack: the DB memo (#1) answers "have we seen this attr"; the eval-cache gives warm in-process re-eval of the remaining delta. Combined with #2 and cross-worker sharding (comment 4), redundant base evaluation drops to zero and the delta scales linearly across hosts. Mind the eval-cache hazards already noted: pre-create the SQLite schema once, single-writer/WAL, pure-eval-only validity, and ship the .drv closure with the cache.

4. Expand the C API / nix-bindings (co-drive upstream with the maintainer).
The enablers worth contributing, since NotAShelf/nix-bindings#39 + NixOS/nix#15842 are already in flight:

• Ergonomic locked-flake evaluation (an InstallableFlake equivalent) so flake eval + input overrides are first-class in the C API rather than the current hand-rolled builtins.getFlake path.
• Eval-cache extract / inject / fingerprint (the primitive 3. depends on).
• Thunk-type peek without forcing: nix_get_type` on an unforced value, so known-non-derivation thunks are never forced (long-standing ask in nix-bindings#39).
• Precise per-eval heap / memory stats read straight from the C API, which is the native evaluation-stats from comment 5, for free.
• Batched substituter cache-status (nix-eval-jobs flags the per-drv query as a bottleneck).

End-to-end target: a commit arrives → diff the lock/fingerprint → ~95% of attrs are already memoized (#1) → shard the remaining delta across N warm-forked workers (#2) pulling the shared resident eval-cache (#3) → near-instant evaluation that scales with the fleet, with crash + memory isolation intact. #1 carries the order-of-magnitude and needs nothing upstream; #3/#4 make the remaining delta cheap; the DerivationResolver trait stays the single swap seam so the evaluator backend (subprocess pool → warm-fork pool → future parallel evaluator) can change without touching the scheduler.

[kraftnix]

I was about to post an issue asking about performance due to evaluations taking for a single repo (even after a successful previous build) taking longer than an entire run of nix-fast-build to eval + build the repo.

The performance in my experience so far is that the full pipeline run is at least one order of magnitude slower than nix-fast-build, sometimes two, especially when nix-fast-build has an eval + build cached.

Looking forward to this.

[DerDennisOP]

Yes, this is the last missing step for Gradient, to be faster than any other CI system out there. I've been working on this for the past 2 Month now, even talking to the Snix maintainer to find the bottlenecks and looking for solutions. I think I have done all the ground work now to implement it into gradient in the next few days.