RAII things - #759
Open
cds-amal wants to merge 7 commits into
Open
Conversation
cds-amal
force-pushed
the
spike/storage-backend
branch
from
August 15, 2026 18:10
cd8a29c to
52a396e
Compare
Process-wide atomic counters for pool and connection events, a snapshot/since API, and an ignored census_workload test that builds and drops ten surfnets per phase and prints the movement. Nothing is connected yet; the counters read zero until a backend routes its pool construction and connections through them. Closes are counted by Drop on a connection wrapper around the r2d2 manager, because CustomizeConnection::on_release fires only for broken or reaped connections, not at pool drop, and undercounts. Co-authored-by: Claude <noreply@anthropic.com>
SQLite pools now build through CountingSqliteManager, and the build sites
in get_or_create_shared_pool report creation and cache reuse.
census_workload, ten surfnets built and dropped per phase:
on-disk: pools 10 (+180 reuses), conns opened 100, closed 0,
100 still live after the last drop
in-memory: pools 190, conns opened 1,900, closed 1,900,
peak 190 live during one surfnet
Connections owned by the process-global SHARED_POOLS survive every
surfnet drop. The in-memory line prices the 19 isolated pools each
surfnet builds.
Co-authored-by: Claude <noreply@anthropic.com>
…ions Each surfnet now opens one StorageBackend from its database URL and mints all of its kv stores from it. The SQLite pool moves off the process-global SHARED_POOLS map onto the backend, so connections live exactly as long as the surfnet that opened them; the checkpoint-dedup set and the per-store shutdown fan-out fall away with it. PostgreSQL keeps its process-level pool cache behind the backend, where pooling actually amortizes a network connection. In-memory SQLite now shares one database per surfnet instead of building 19 isolated pools, matching the on-disk layout of distinct tables in one database. The counting manager remains the pool's connection type, so connection opens and closes stay counted; the census build-site hook is reconnected in the next commit. Co-authored-by: Claude <noreply@anthropic.com>
SqliteBackend::open reports pool creation, and a new assertion test,
dropping_backend_closes_connections, fails if any connection opened by a
backend survives its drop.
census_workload, same workload as the baseline measurement two commits
back:
on-disk: pools 10, conns opened 100, closed 100, live 0
in-memory: pools 10, conns opened 100, closed 100, live 0
Ten dropped surfnets leave 100 connections live on the baseline and none
here; in-memory construction opens 190 connections per surfnet there and
10 here.
Co-authored-by: Claude <noreply@anthropic.com>
Counting stays always on: the counters are a handful of relaxed atomics bumped at pool and connection lifecycle events, which are rare next to queries, so there is nothing worth stripping from a release build. What was missing was a way to see the movement outside a test harness. Each counter now emits a debug line under this module's log target, so RUST_LOG=surfpool_core::storage::census=debug turns on reporting in any build, including shipped binaries. That matters for field diagnosis: when someone hits descriptor exhaustion, the ask is an env var and a re-run, not a from-source build. It also keeps a single pool type in every configuration, so tests exercise exactly what ships. The census tests initialize env_logger (is_test, same pattern as the integration suite) so harness runs show the per-event narrative next to the snapshot deltas. Note the event ordering in the logs: "pool created" appears after its connections open because r2d2 fills the pool eagerly inside build(), and we count the pool only once the build succeeds.
The census did its job: the baseline and fix measurement commits carry its numbers, and an external fd sampler reproduces the on-disk result from outside the process (peak 434 SQLite descriptors on the shared-pool baseline against 23 with backend-owned pools, over the same workload of 31 on-disk tests). The evidence for the refactor no longer needs counters in the tree, and keeping them would mean a counting wrapper inside the shipped pool type forever, a larger maintenance surface than a merged fix should carry. SqliteBackend builds on a plain ConnectionManager again, so the pooled connection loses one deref hop and the pool type is identical in every configuration. The pre-removal tip is tagged census-archive; the counters, the Drop-based close accounting with its r2d2 on_release subtlety, and the workload harness live there if this leak class ever needs exact instrumentation again.
cds-amal
force-pushed
the
spike/storage-backend
branch
from
August 15, 2026 22:30
52a396e to
de71c83
Compare
cds-amal
marked this pull request as ready for review
August 16, 2026 01:46
Contributor
Greptile SummaryThe PR replaces per-store database construction with a surfnet-owned
Confidence Score: 5/5The PR appears safe to merge; no concrete changed-code-triggered failure remains. The storage lifecycle refactor consistently routes store creation through a surfnet-owned backend, preserves PostgreSQL sharing, and keeps bundle sandbox pool clones aligned with the lifetimes of their overlay stores. Important Files Changed
Reviews (1): Last reviewed commit: "Remove the census now that its measureme..." | Re-trigger Greptile |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes a SQLite connection leak that presents with exhausted file-descriptors over the course of multiple surfnet lifetimes, when the number of file-descriptors open crosses the system limit and file opens fail. This happens on stock linux machines under
ulimit -Sn 1024.Every SQLite store builds or borrows its pool from a process-global cache. A process accumulates connections for each surfnet because its associated connections are owned by the process-global cache. Further, every in-memory SQLite backed surfnet built 19 separate pools; one per store, each a cache miss (19 x 10).
The fix is to let
StorageBackendgive each surfnet an owner for its database connections, specifically one pool, built when the surfnet's storage initializes, shared by every store the surfnet opens. Stores hold pool clones so the pool-clones are bound to the store's lifetime, and when theStorageBackendgoes away, it will release the one pool it owns; RAII ftw.PostgreSQL holds a lease on the process-level pool for its database URL. Unlike SQLite, the pool stays shared across surfnets, since pooling exists to amortize the network connection and the server caps total sessions.
Measurements
Counter movement for ten surfnets built and dropped per phase, recorded by a temporary in-process census that this branch carried for the measurement and removes before merge (the history has it, the squash lands only the fix):
The suite-level effect:
cargo test -p surfpool-core --lib --features ignore_tests_ciunderulimit -Sn 1024, same machine, same session,both builds:
into that API, the ~5x drop was enough for this PR.
Verifying from outside the process
The on-disk result reproduces externally, with no code from this branch: the workload is 31 pre-existing tests that each build and drop an on-disk surfnet.
On main the count climbs a staircase to 434 and holds it until process exit; on this branch it peaks at 23 (one live surfnet's worth) and returns to the floor between tests. In-memory databases hold no descriptors, so that half of the improvement is visible only in the counter table above.