Skip to content

feat(indexes): stake address log for O(page) account address queries - #1225

Closed
slowbackspace wants to merge 2 commits into
fix/minibf-account-addressesfrom
feat/minibf-stake-address-log
Closed

feat(indexes): stake address log for O(page) account address queries#1225
slowbackspace wants to merge 2 commits into
fix/minibf-account-addressesfrom
feat/minibf-stake-address-log

Conversation

@slowbackspace

Copy link
Copy Markdown
Contributor

Stacked on #1221 (the base branch). Follow-up to the performance concern flagged there.

Problem

With the correct first-appearance ordering from #1221, GET /accounts/{stake_address}/addresses?order=desc must scan the account's entire archive history before it can reverse the list. Measured on a fully synced mainnet snapshot: 305 seconds (cold and warm — the cost is CPU-bound block decoding) and ~274MB transient RSS for an exchange account with 400k+ addresses. Any desc page on such an account pays the full scan, which is also a DoS shape on public nodes.

Design

A stake address log: each (stake credential, address) pair stored once, at its first on-chain appearance, ordered by (slot, tx order, output order). Both asc and desc become a single page read (desc is a reverse range read — Blockfrost defines it as the exact reverse of asc).

  • fjall (the supported index backend) gets a new stake-log keyspace with two entry shapes: a membership entry per pair (the write-path probe and the undo key) and an ordered entry (the page read). The write batch cannot read its own pending inserts, so the writer keeps a batch-local seen-set — one writer can span many blocks in WAL catch-up.
  • memory implements the same semantics, so ToyDomain endpoint tests exercise the log. redb3 and noop answer None (redb3 is deprecated for index stores).
  • Undo correctness: rollback does not re-run index_block, so compute_undo derives the appearances from the same stake_appearances_from_block function the apply path uses. A pair is removed only when the undone block is its stored first appearance; a block that merely repeated an address leaves the log untouched.
  • Rollout: a ready marker gates reads. Genesis bootstrap sets it, so any store synced (or mithril-bootstrapped) from scratch serves the log; existing stores answer None and the endpoint transparently falls back to the archive scan from fix(minibf): fix ordering and 404 edge cases in account addresses #1221 until they resync. No doctor/backfill job — resync is the rollout path.
  • Full address bytes are stored (not just the payment part) so pointer addresses round-trip. Estimated mainnet size: 5–15GB against a 340GB store.

Testing

  • Backend conformance test in tests/index_roundtrip.rs runs against fjall and memory: marker gating, first-appearance dedup (including inside one multi-block writer), ordered paging from both ends, offset windows, stake isolation, and undo-of-first-appearance-only.
  • Memory-store unit tests cover the same semantics at the store level.
  • Endpoint equivalence test: two identical synthetic chains, one serving from the log and one from the archive-scan fallback, must return identical responses for every query shape. Verified non-vacuous by flipping the order flag and watching the order tests fail.
  • Full workspace suite passes; clippy clean; nightly fmt applied.

Not yet done: an end-to-end run against a freshly synced network store (my local preview/mainnet stores predate the log, so they exercise only the fallback). Happy to re-sync preview and attach numbers if you want them before undrafting.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Aug 18, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1cff66db-8ce6-4911-93aa-8819fd7e03ea

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

The /accounts/{stake_address}/addresses endpoint diverged from
Blockfrost in two cases:

- order=desc sorted addresses by their latest on-chain appearance.
  Blockfrost returns the exact reverse of the asc list, which orders
  addresses by first appearance. Reused addresses came out in the
  wrong position.

- Accounts that only appear inside pool registrations (reward account
  or pool owner) returned 404. Blockfrost knows these credentials and
  returns an empty list.

Fixes #1140
The /accounts/{stake_address}/addresses endpoint scans every archive
block that touches the account. With correct first-appearance ordering,
a desc request must scan the account's full history: 305 seconds
measured on mainnet for an exchange account with 400k+ addresses.

The stake address log stores each (stake credential, address) pair
once, at its first on-chain appearance, ordered by slot, transaction
order, and output order. Both orders become one page read.

- fjall: new stake-log keyspace with membership entries (the write
  probe and undo key) and ordered entries (the page read).
- memory: same semantics, so ToyDomain tests exercise the log.
- redb3 and noop answer None; redb3 is deprecated for index stores.
- The apply path emits appearances from index_block. The undo path
  derives them from the same function, so a rollback removes exactly
  what apply inserted, and only when the undone block was the pair's
  first appearance.
- A ready marker gates reads. Genesis bootstrap sets it, so stores
  synced from scratch serve the log. Existing stores answer None and
  the endpoint falls back to the archive scan until a resync.
@slowbackspace

Copy link
Copy Markdown
Contributor Author

Applied into #1221 as commit 568d3ba. That PR now carries both the fixes and the stake address log.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant