feat(config): auto-tune pool size for async workers#114
Conversation
Async workers use fibers that share connections (only one fiber runs at a time per reactor thread), so they need far fewer database connections than thread-based workers. The pool size auto-tuning now contributes 3 connections per async worker (reactor + polling + headroom) instead of N connections per N fibers. Example impact: - Thread worker (50 threads): 50 connections - Async worker (50 fibers): 3 connections - Mixed (50 async + 5 thread): 8 connections This matches Solid Queue's approach where Rails 7.2+ async workers use 3-5 connections regardless of fiber capacity.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughIntroduces a fixed async connection constant and async-mode detection into pool-size auto-tuning. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@lib/pgbus/configuration.rb`:
- Around line 641-645: The method async_execution_mode?(entry) currently reads
execution_mode only from the entry and returns false if missing, ignoring the
global fallback; change it to call execution_mode_for(entry) to obtain the
effective mode (which honors global config), then normalize via
ExecutionPools.normalize_mode(...) and compare to :async so async pool sizing
uses the same logic as the rest of the configuration API (ensure you still
handle nil safely before normalization).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: 3d412c00-3055-4aff-9994-29c4eb499ee6
📒 Files selected for processing (2)
lib/pgbus/configuration.rbspec/pgbus/configuration_spec.rb
async_execution_mode? only checked per-entry execution_mode and returned false when unset, ignoring the global config.execution_mode fallback. A user setting config.execution_mode = :async globally (without per-worker overrides) would still get N connections per worker instead of ASYNC_POOL_CONNECTIONS (3). Reuse execution_mode_for(entry) which already handles the global fallback. Added test for the global path.
The allow_files filter still attributed gem-internal retained objects to lib/pgbus/ paths on Ruby 3.3 (MemoryProfiler tracks allocation source, not retention cause). Replace with a two-pass approach: 1. First pass: MemoryProfiler.report captures any one-time lazy initialization (gem globals, JSON caches, connection_pool singletons) 2. GC.start to collect the first-pass artifacts 3. Second pass: measures only steady-state behavior — should retain 0 This is immune to Ruby version differences in GC timing, gem loading order, and MemoryProfiler attribution heuristics.
Summary
Auto-tune the PGMQ connection pool size for async workers. Fibers share connections (only one runs at a time per reactor thread), so async workers need 3 connections instead of N-per-fiber.
The constant
ASYNC_POOL_CONNECTIONS = 3covers: one for the reactor's serial execution, one for polling, one for headroom. This matches Solid Queue's approach where Rails 7.2+ async workers use 3-5 connections regardless of fiber capacity.Explicit
pool_sizestill overrides auto-tuning for advanced cases.Test plan
:fiberalias works (normalizes to async)Summary by CodeRabbit
New Features
Tests