Pace long received-log downloads to avoid monopolizing validator storage

[ndr-ds]

Supersedes #6631 — same change, rebased onto current main (the original branched 2026-07-28 and was 81 commits behind), plus the threshold turned into a flag.

Motivation

Client::get_received_log_from_validator pages a validator's received_log in 20 000-entry pages with no pacing at all. On the validator side each page becomes received_log.read(start..end), which reads one storage row per entry — so a long backlog walk saturates the validator's storage and degrades read latency for every other client of that validator.

Measured on testnet_conway: statement-class IOPS go from ~100/s to ~5 500/s and read p99 from 6 ms to 4.3 s, for 20–60 minutes, which then stalls chains across the whole PM fleet.

Pacing existed in the original design (#4706, as sync_sleep_ms) and was removed wholesale by #4708.

Proposal

After the first few full pages — by then it is clearly a backlog walk, not a routine sync — sleep between pages for as long as the previous page took to serve.

That is self-tuning in a useful direction: it caps the walk at roughly half the validator's serving capacity for this client, and automatically backs off further when the validator is already slow. Short syncs never reach the threshold and are unaffected.

Beyond the original PR

The original hardcoded RECEIVED_LOG_PAGES_BEFORE_PACING = 5. It is now --received-log-pages-before-pacing, defaulting to the same 5 via DEFAULT_RECEIVED_LOG_PAGES_BEFORE_PACING, threaded through ClientOptions → chain_client::Options like the other sync tunables (--sender-certificate-download-batch-size, --max-concurrent-batch-downloads). 0 paces from the first page.

The right value depends on backlog size and validator capacity, neither of which is known at compile time — and if a storm is in progress, being able to lower it without a release is the point.

Test Plan

• cargo check -p linera-client passes (the flag threads through linera-core and linera-client).
• CLI.md regenerated with cargo run --bin linera -- help-markdown, so check-outdated-cli-md stays green.

Honest limits. There is no automated test: asserting "it slept between pages" needs a fake clock plus a validator stub that serves ≥ 6 full 20 000-entry pages, which the current test harness does not provide. The change is also a mitigation, not a fix, and the arithmetic should be read before merging:

• it halves the peak rate but roughly doubles the wall-clock of a long walk (same total I/O, spread thinner);
• for the measured 3.87 M-entry backlog that is ~194 pages, so ~19 min becomes ~38 min at about half the amplitude.

Whether that trade is worth it depends on whether the pain is peak latency (it helps) or total storm duration (it does not). It composes well with #6830, which keeps backlogs small enough that pacing rarely triggers at all.

Release Plan

• These changes should be backported to the latest testnet branch, then
  • be released in a validator hotfix.

Links

• reviewer checklist
• Supersedes Pace long received-log downloads to avoid monopolizing validator storage #6631.
• Related: Periodically re-run the background received-certificate sync to keep trackers fresh #6830 keeps the backlog small (the real fix for the restart case); Log progress of large received-certificate syncs at info level #6831 makes a long walk visible at info; Order LogView index keys so a range of entries is contiguous in storage #6829 makes each entry of any walk cheap by ordering LogView keys.