chore(infra): derive DAO API database memory settings from cgroup limit - #2099
chore(infra): derive DAO API database memory settings from cgroup limit#2099pikonha wants to merge 3 commits into
Conversation
All 14 DAO databases run the stock postgres-ssl image with no tuning, and every one of them carries a hand-applied `max_connections = 500` in postgresql.auto.conf — undocumented, invisible outside the volume, and applied across four different memory tiers (2/4/6/8 GB). 500 backends is roughly 2.5 GB of unreclaimable memory. On the 2 GB tier (nouns-db, shutter-db) that is an OOM the container cannot survive, and PostgreSQLHighConnections cannot catch it: it fires at 80% sustained for 10m, which the OOM beats. Wrap the stock image so the entrypoint reads /sys/fs/cgroup/memory.max and appends the derived values as -c flags. Command-line options outrank postgresql.auto.conf, so this neutralises the existing 500 without writing to any volume, and reverts by pointing a service back at the stock image. 2 GB -> max_connections=75 effective_cache_size=1.3 GB 4 GB -> max_connections=80 effective_cache_size=2.6 GB 6 GB -> max_connections=120 effective_cache_size=3.9 GB 8 GB -> max_connections=160 effective_cache_size=5.2 GB shared_buffers is deliberately left at its 128 MB default: raising it converts reclaimable page cache into unreclaimable shmem, which would spend safety margin on the two tiers that have the least of it. Separate change. Verify with: bash infra/dao-api-db/entrypoint.dao-api-db.sh --self-test Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
|
🚅 Deployed to the anticapture-pr-2099 environment in anticapture-infra
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 074b1e2c74
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # The base image's ENTRYPOINT (wrapper.sh) sets up SSL certs and pgbackrest, so | ||
| # we don't replace it — entrypoint.dao-api-db.sh derives a couple of settings from | ||
| # the container's memory limit and then execs wrapper.sh with them appended. | ||
| COPY entrypoint.dao-api-db.sh /entrypoint.sh |
There was a problem hiding this comment.
With the repository-root build context selected by dao-api-db.railway.toml, this source resolves to /entrypoint.dao-api-db.sh, but a repo-wide search shows the file exists only under infra/dao-api-db/; the image build therefore fails before deployment. Docker's COPY documentation specifies that local sources are resolved relative to the root of the build context, so this must copy infra/dao-api-db/entrypoint.dao-api-db.sh (or explicitly change the build context).
Useful? React with 👍 / 👎.
The fleet is not on one major: production runs 17 Postgres services across 16 (x8), 17 (x4) and 18 (x5). A hardcoded :16 base would start against a newer PGDATA and exit with "database files are incompatible with server", crashlooping 9 of 17 databases. Take the major from a PG_MAJOR build arg, which Railway populates from the service's variables. No default on purpose: an unset PG_MAJOR fails the build and leaves the running container serving, whereas a wrong default would build cleanly and then crashloop. Booted and verified on all three majors at a 2 GB limit — each reports max_connections=75, effective_cache_size=1331MB, ssl=on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🔍 Vercel preview: https://anticapture-ii0rl6kms-ful.vercel.app |
|
Reviewed this while scoping the Q3 infra-cost KR. The core mechanism is sound — 1. No
|
| service | tier | new cap | ceiling demand |
|---|---|---|---|
| shutter-db | 2 GB | 75 | ~90 (offchain → 3 API pools) |
| ens-db | 4 GB | 80 | ~90 (offchain → 3 API pools) |
| compound-db | 4 GB | 80 | ~90 |
| nouns-db | 2 GB | 75 | ~80 |
Pools are lazy and idle connections get reaped at node-postgres' default 10s, so this may well never bind in practice. But shutter and ens have the least headroom and would fail as FATAL: sorry, too many clients already under a burst, which is a worse failure shape than the slow degradation the floor was chosen to avoid. Confirming against real pg_stat_activity counts before rollout would settle it.
4. Merging this does not change anything by itself
None of the *-db services are in the list above — the DAO databases are Railway-managed and not repo-connected, which is also why no DB deploy check appears on this PR. Each of the 17 needs to be repointed at infra/dao-api-db/Dockerfile.dao-api-db with PG_MAJOR set, individually, in the UI. Worth stating in the PR body as an explicit rollout step so it doesn't read as merge-and-done.
Related, for sizing: since shared_buffers deliberately stays at 128 MB, the entire memory effect comes from shedding backends. At the ~5 MB/backend implied by this PR's own "500 backends is roughly 2.5 GB":
saved ≈ (actual_connections − new_cap) × 5 MB × 17 databases
Railway currently bills memory at roughly $10/GB/month, and the memory line is 74% of the bill (~90 GB resident against 1.42 vCPU average — the signature of idle backends holding RAM without consuming CPU). So if the fleet really is sitting near 500, this is worth a few hundred a month; if it's near 90, it's worth about $13. That difference is exactly what pg_stat_activity per database would tell us, which loops back to getting the exporter deployed first.
None of this argues against the change — 500 across a 2 GB tier is an OOM waiting to happen and the direction is right. Mostly it's the ordering: measure, deploy the exporter, then repoint.
2 participants
Why
All DAO databases run the stock
postgres-sslimage with zero tuning, and every one sampled carries a hand-appliedmax_connections = 500inpostgresql.auto.conf— undocumented, invisible outside the volume, applied across four different memory tiers.memory.maxauto.confmax_connections = 500max_connections = 500max_connections = 500max_connections = 500max_connections = 500max_connections = 500500 backends is ~2.5 GB of unreclaimable memory. On the 2 GB tier that is an OOM the container cannot survive.
PostgreSQLHighConnectionscannot catch it either — it fires at 80% sustainedfor: 10m, which the OOM beats.What
Wrap the stock image. The entrypoint reads
/sys/fs/cgroup/memory.maxand appends derived values as-cflags.max_connectionseffective_cache_sizeDerived rather than hardcoded because the fleet spans four tiers off one image, and a Railway plan change moves a service between tiers without touching the repo.
-cflags outrankpostgresql.auto.conf, so this neutralises the existing500without writing to any volume, and reverts by pointing a service back at the stock image.shared_buffersdeliberately stays at its 128 MB default — raising it converts reclaimable page cache into unreclaimable shmem, spending safety margin on the two tiers with the least of it. Separate change.PG_MAJORProduction runs 17 Postgres services across three majors: 16 (×8), 17 (×4), 18 (×5). The base image version comes from a
PG_MAJORbuild arg, which Railway populates from the service's variables.There is no default on purpose. An unset
PG_MAJORfails the build, leaving the running container serving. A wrong default would build cleanly and then crashloop ondatabase files are incompatible with server.Verified locally
--self-testpasses 8 cases: 4 fleet tiers, both clamp boundaries, 2 fallback pathsPGDATA; each reportsmax_connections=75,effective_cache_size=1331MB,ssl=onat a 2 GB limitPG_MAJORfails the build, as designed75) and 6 GiB (120)ssl = on— the entrypoint wrap does not break the base image's cert setupmax_connections = 500written into a realauto.confand restarted:SHOW max_connections→75,source→command lineNot in this PR
shared_buffersat 128 MB fleet-wideREINDEX DATABASEbeforeREFRESH COLLATION VERSIONpostgres-exporter'sDATA_SOURCE_NAMEholds plaintext superuser credentials for 14 production databases in one variable🤖 Generated with Claude Code