api: move tiredsolid scoreboard feed config to postgres - #701
api: move tiredsolid scoreboard feed config to postgres#701armcconnell wants to merge 10 commits into
Conversation
loadHyperliquidScoreboardEntries swallowed every Postgres error and returned an empty set, so a DB blip looked identical to "zero feeds configured". FetchHyperliquidScoreboardData then returned a valid empty response, which the background refresher and page-cache worker happily cached over the last-good payload — blanking the public scoreboard silently and indefinitely. Now a genuine load failure (query, scan, or rows.Err) propagates as an error so callers skip the cache write; zero configured rows still returns the clean empty response unchanged. Also reject tob_* feeds in the loader: the allow-list clause assumes DoubleZero's own tob_ feeds are never configured entries, and a mistaken tob_ row would broaden it to match races against unconfigured competitors, leaking their feed ids into the public payload.
label flows unvalidated into the public scoreboard JSON payload. Add a CHECK constraint so it can't be empty or unbounded. Edited in place since this migration has never been deployed.
With zero configured feeds the page rendered the hero card and gauge showing 0 races / 0.0% win rate as if it were real data. Between deploy and manual per-environment seeding, that publishes a misleading headline instead of nothing. Show a neutral empty state in its place.
ben-dz
left a comment
ben-dz
left a comment
There was a problem hiding this comment.
Two mediums worth addressing before or shortly after merge: the live request path now returns a 500 during a Postgres outage where it previously served from ClickHouse (config was in code, and the page cache is also in Postgres, so there is no fallback), and the DZ-lost arm of the unconfigured-feed exclusion clause has no test. Two lows: the 500 body now carries raw Postgres error text (swap in the existing internalError() helper), and the new table should attach the existing updated_at trigger. The sanitizer, allow-list clause semantics, and last-good cache preservation on config-load failure all check out.
- Low — api/handlers/hyperliquid_scoreboard.go:736: the 500 body returns raw err.Error(), and this PR routes the first Postgres-sourced errors into it — pgx connect errors embed internal detail (failed to connect to host=… user=… database=…), so during a Pg incident an uncached request (e.g. ?symbol=BTC) discloses infra names to anonymous clients. The repo already has internalError() (api/handlers/errors.go:45) for exactly this; swap it in.
| // rather than scanning for feeds we would then discard. | ||
| entries, err := a.loadHyperliquidScoreboardEntries(ctx) | ||
| if err != nil { | ||
| return nil, err |
There was a problem hiding this comment.
The live request path now hard-fails during a Postgres outage where it previously served. Before this PR a Pg outage made readPageCache miss and the handler still served the scoreboard live from ClickHouse (config was in code); now the live path dies here with a 500 — and there is no cached fallback either, because the page cache also lives in Postgres. Other pages degrade to live-serving during a Pg outage; this one now 500s. The propagate-don't-blank choice is right for the refreshers, but if scoreboard uptime through Pg maintenance matters, consider a process-local last-good copy of hyperliquidEntries (e.g. an atomic.Pointer on API, updated on every successful load) as a fallback when the load fails.
| insertEntry(t, api, "feed_a_bbo", "Feed A", 1, true) | ||
|
|
||
| insertPairwise(t, api, "tyo-rec1", "tyo", "BTC", 10, 1, "tob_gcp_tyo_hl_mainnet1", "feed_a_bbo", 1.0) | ||
| insertPairwise(t, api, "tyo-rec1", "tyo", "ETH", 20, 2, "tob_gcp_tyo_hl_mainnet1", "feed_unlisted_bbo", 1.0) |
There was a problem hiding this comment.
The DZ-lost arm of the unconfigured-feed exclusion is unpinned: this test (and TobFeedEntryIgnored) only inserts winner=tob_, loser=feed_unlisted_bbo. The reverse row (feed=feed_unlisted_bbo, loser_feed=tob_ — an unconfigured competitor beating DZ) exercises the loser_feed IN (…) side of the OR and is excluded by the code but asserted nowhere. One extra insertPairwise pins the clause's other arm.
| label TEXT NOT NULL CHECK (length(label) BETWEEN 1 AND 64), | ||
| display_order INT NOT NULL DEFAULT 0, | ||
| enabled BOOLEAN NOT NULL DEFAULT TRUE, | ||
| updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() |
There was a problem hiding this comment.
updated_at freshness relies on operators remembering updated_at = NOW() in the runbook UPDATE. The repo's update_updated_at_column() trigger function (migrations 00001/00003) is attached to sibling tables for this; attach it here so a hand-typed UPDATE can't leave a stale timestamp.
3 participants
Summary
feed,label,display_order,enabled). Adding, removing, or reordering a feed is now a row change instead of a deploy.The migration ships the table empty on purpose — rows are environment config, seeded out of band, so values never live in the repo. Until an environment is seeded its scoreboard renders empty. Changes take effect on the next cache refresh (~60s for the 1h view, ~10min for 24h/7d), no restart.
Testing Verification
go test ./api/handlers/green, 0 failures.