Leios: omnibus for production-grade EB diffusion

[nfrisby]

Closes all of input-output-hk/ouroboros-leios#1074 subissues.

Also introduces committee-via-forecasting infrastructure that I suspect Issue input-output-hk/ouroboros-leios#1073 should use for validating votes.

This development lead to these two upstream PRs:

• PR BlockFetchConsensusInterface: add matchedBlock ouroboros-network#5430 so the predecessor's slot and LedgerView directly accompanies the block when it arrives
• PR [Leios Prototype] Add Dijkstra forecasting for Leios cardano-ledger#6064 so forecasting can provide the Leios committee and proto params and the TICKF rule puts the correct epoch's Leios committee in that forecast

---

When this PR merges, we'll need to add a few small commits to downstream repos:

• cardano-api needs d9bf166c7 - (HEAD -> nfrisby/integrate-PraosWithLeios, origin/nfrisby/integrate-PraosWithLeios) Consensus: change Dijkstra to PraosWithLeios
• cardano-node needs addf5beb1 - (HEAD -> nfrisby/leios-issue-1074-omnibus, origin/nfrisby/leios-issue-1074-omnibus) Add ChainDB tracers for forgetting CertRBs at start-up (and its handful of predecessors)
• ouroboros-leios needs 99179913e - (HEAD -> nfrisby/integrate-LeiosMinOfferLead, origin/nfrisby/integrate-LeiosMinOfferLead) Add appropriate LeiosMinOfferLead to devnets' config files

---

Another consequence of merging this PR is that the subsequent prototype will have a different codec for pre-Dijkstra eras than prior prototypes: this PR changes the prototype to (correctly) use the unchanged Praos header codecs for pre-Dijkstra and only introduce the Leios fields as of Dijkstra. Our testnet does not start in Dijkstra. So shipping this PR requires respinning the testnet's chain, and requires all nodes to update their header codecs in the same way.

---

Overview of major changes, roughly in commit order:

• Issue Validate certificate in a CertRB _before_ ChainSel begins selecting it input-output-hk/ouroboros-leios#1071:

  • Replace data Praos with data BasePraos and type Praos = BasePraos PextNone to make room for type PraosWithLeios = BasePraos PextLeios. Integrate/dispatch to the new upstream HeaderBody types accordingly.
  • We add the predecessor's slotno and its LedgerView to BlockToAdd, so we can know which LeiosPoint the block certifies and which Leios committee to use when validating a CertRB's cert in ChainSel's first pass (before we have its predecessor's ledger state).
  • That slotno is always available: easy in the forge, but because of the ChainSync->BlockFetch flow we needed to add the MatchedBlock annotation. In order to convey that info from ChainSync to BlockFetch, we added hwtPredecessorSlot and hwtLedgerViewOfPredecessor (which makes HeaderWithTime a misnomer, but I left that as a TODO to fix, for now).
  • ChainSel now only calls setTentativeHeader (ie "block diffusion pipelining") if the block it's trying to select just arrived (ie arrived "in order" and after the EB it certifies, if any) (see TentativeHeaderPermission)
  • And that's all enough to let ChainSel validate a CertRB's on arrival, before it's attempting to select that CertRB. (see precheckLeiosCert). Because the corresponding LedgerView is immediately available (see above), that certificate can always be validated.

• Issue Bound bookkeeping of EB offers that arrive via ChainSync input-output-hk/ouroboros-leios#1069

  • Fetching an EB is justified by elections and certifications, never by a peer's offer. Offers only say which peers can serve what.
  • Nothing lists an EB on the strength of a claim we have not yet verified; the candidates we pursue are bounded by elections rather than by peers (thanks to Issue 1071).

• Issue Enable the recovery path in LeiosFetch logic input-output-hk/ouroboros-leios#1070

  • ValidClaims records which EB each claim announced, so isCertifiedEb can answer whether an announcement is certified, and
  • focusCertifiedEb moves a verified certificate's election onto the EB it names — which is what lists that second body to fetch, if we had been mislead by an equivocation. We also disconnect from peers that claim committee equivocation; those claims arrive over ChainSync rather than LeiosNotify.
  • This work introduced the node-vs-environment Leios test harness, which invokes the actual initNodeKernel. Some of the subsequent enrich it in order to also test their cases while increasing the node-under-test's fidelity.
  • For Issue 1071, we needed to refine its behavior during node startup.
    • See the valid_claims_startup.md.
    • At startup a CertRB already in the VolatileDB never passes through chainSelAddBlock again, so nothing hands its certificate a ledger view and its claim stays unverified for the run — which closes the Recovery Path for the EB it certifies. We now forget every CertRB not on the selection after initial chain selection: unknown but still stored, so re-adding it writes nothing and re-admits it, and the ordinary path then supplies the view.

• Issue Require announcements precede offers and tighten LeiosFetch logic input-output-hk/ouroboros-leios#1082

  • Offers become PeerOffer, with the body and the closure offered independently. A peer may offer the parts of only an EB it announced to us, at most once per part, and never for an EB older than our immutable tip; each violation costs it the connection.
  • This node holds itself to the same rules — it offers only what it actually sent announcements for and LeiosMinOfferLead sets how much younger than our immutable tip an EB must be to be relayed. (The Recovery Path catches, after the fact, any offers that these rules prevent from happening over LeiosNotify.)

• Issue BUG: the prototype relays EBs that lied about their tx's sizes input-output-hk/ouroboros-leios#1115

  • If an announcement gives the wrong size for an EB, then its closure is no longer called complete. Nobody is disconnected — the issuer isn't a peer and the relayer may be honest.
  • Because the node never concludes it has the EB's closure, it never relays the closure and it never votes for the evil announcement.

• Issue Disconnect if a peer requests an EB body or EB closure that we don't have input-output-hk/ouroboros-leios#1083 and Issue Abort a LeiosFetch request processing if it turns out to be too many bytes input-output-hk/ouroboros-leios#1124

  • The LeiosFetch server disconnects on requests it cannot answer instead of replying with whatever it found.
  • batchRetrieveTxs takes a byte budget, so the LeiosFetch server aborts if it realizes a request is for more bytes than an honest client would put in a single request.
  • The honest client gained a pipelining depth of 150.
  • maxLeiosEbBytesSize/maxLeiosTxsRequestBytesSize were corrected to be the design ceilings,
    dropping maxTxsPerEb from 71,428 to 14,979.

• Issue Integrate LeiosFetch with Genesis's ChainSync Jumping (CSJ) input-output-hk/ouroboros-leios#1072

  • Record offers implied by a CSJ Jumper accepting a jump via the new CSJ.leiosJumpAcceptedCallback config field.
  • Added maxAcceptedJumpSlot field to Leios.PeerVars to limit this processing to just the headers that weren't also part of previous jumps.

[nfrisby]

It's in draft because of unacceptable commit names and no tests.

[nfrisby]

Self-review pass

[nfrisby]

Maybe the two methods above this one could be retired, since we can compose this new one with protocolLedgerView?

[ch1bo]

Yes, retired; see the thread below.

[nfrisby]

I didn't parameterize this over pext because:

• I know it occurs in very many places.
• We can use KnownPraosExtension pext as a small workaround to recover the Signable (KES c) HeaderBody dict for some pext in the few places we need it.

[nfrisby]

This was merely floated out of its where clause so it could be called from a second location.

[nfrisby]

TODO is this impl too allocation-heavy?

[nfrisby]

My latest pushes are just trying to fix all the CI failures. I'm only compiling with 9.12 -Werror locally.

[nfrisby]

There are a couple testing modules where I haven't yet combed through the comments Claude wrote, but otherwise, I've reviewed every line (and trimmed down a lot of comment, pushed back on extra complexity, etc).

[nfrisby]

For what it's worth, Claude breaks down the diff's line counts as in the table below.

My highlights:

• 38% of the added lines are are the new node-vs-environment test, which I'm delighted to have now.
• LeiosDB is only touched because that's currently where the detection logic is for when we have acquired an EB's entire closure---if that moved into the LeiosFetch decision logic (... which I think @ch1bo's branch might be doing?), then this PR wouldn't alter the LeiosDB at all. (edit: lookupTrustedEbClosure is just a rename to emphasize that it is called by code that trusts this EB and its closure (and their size!) was already validated before this function is called.)

  ┌──────────────────────────────────────────────────┬──────┬─────┬───────┬───────┐
  │                    component                     │  +   │  −  │ churn │ files │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ Praos / BasePraos protocol                       │ 1016 │ 711 │  1727 │    10 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ ChainSel + ChainDB + LedgerDB                    │ 1047 │ 345 │  1392 │    24 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ node-vs-env tests (RecoveryPath)                 │ 1288 │   0 │  1288 │     1 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ LeiosTestBlock (test lib)                        │ 1233 │   0 │  1233 │     1 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ node-vs-env harness (Environment, NodeUnderTest) │ 1043 │   0 │  1043 │     2 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ Recovery Path / ValidClaims                      │  979 │   0 │   979 │     4 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ LeiosFetch decision logic (LeiosDemoLogic)       │  620 │ 148 │   768 │     5 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ Shelley/Cardano ledger integration               │  453 │ 275 │   728 │    29 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ Leios types (LeiosDemoTypes)                     │  588 │  54 │   642 │     1 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ LeiosDb                                          │  431 │ 131 │   562 │     5 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ Node wiring (NodeKernel/NodeToNode/Node)         │  404 │  68 │   472 │     4 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ docs + changelog + cabal + nix                   │  238 │  18 │   256 │     9 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ other                                            │  106 │  22 │   128 │     7 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ other tests                                      │   94 │  26 │   120 │     9 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ LeiosFetch protocol (client/server)              │   61 │   9 │    70 │     1 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ ChainSync + CSJ                                  │   49 │  10 │    59 │     6 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ LeiosNotify protocol (client/server)             │   18 │   1 │    19 │     1 │
  ├──────────────────────────────────────────────────┼──────┼─────┼───────┼───────┤
  │ Leios voting                                     │   13 │   3 │    16 │     2 │
  └──────────────────────────────────────────────────┴──────┴─────┴───────┴───────┘

  The headline: tests and test infrastructure are 3,684 added lines — RecoveryPath.hs, LeiosTestBlock.hs, the harness, plus 120 in other
  tests. That's 38% of the additions and essentially all of it new files, which is why the number looks alarming without being alarming.

  The next largest, Praos/BasePraos at 1,727, is the most deletion-heavy bucket and is mostly not new code: Praos/Header.hs (−338) and
  Praos/VRF.hs (−139) are deleted outright, having moved upstream, so 477 of the 1,821 deletions are that one migration.

  ChainSel + ChainDB + LedgerDB at 1,392 over 24 files is the diffuse one — the Predecessor plumbing reaching through ChainSel, Forker,
  HeaderValidation, the ChainDB API and the LedgerDB.

[nfrisby]

Just now added the commit Config: allow default leiosMinOfferLead to be overriden
0a211e1.

I anticipated the leiosMinOfferLead value needing to be tweaked on devnets while implementing the behavior, but then I was still mystified why my proto-devnet nodes weren't sending any offers. Thankfully, Claude connected the dots between their absence and the comments quickly. New insight = the restriction bites even harder than anticipated at the Origin of a new chain. The fix was to add the ouroboros-leios commit that overrides the config on the proto-devnet with something tolerable: 30 seconds. (See downstream commit mentioned in PR description.)

[nfrisby]

I added this warning to the PR description:

Another consequence of merging this PR is that the subsequent prototype will have a different codec for pre-Dijkstra eras than prior prototypes: this PR changes the prototype to (correctly) use the unchanged Praos header codecs for pre-Dijkstra and only introduce the Leios fields as of Dijkstra. Our testnet does not start in Dijkstra. So shipping this PR requires respinning the testnet's chain, and requires all nodes to update their header codecs in the same way.

(edit: I kludged the local build of cardano-ledger so that I could run my local build of this PR against the testnet. It's currently syncing.)

[ch1bo]

Rebased and had a 🧞 fix several things:

• Repeated offers (pgrange's late-join failure): a body arriving twice before its write settles gets written twice, so the server offered it twice and the honest peer disconnected. The notify server now offers each part of an EB at most once per peer.
• Crossing the hard fork into Dijkstra: a view forecast from a Conway state got made-up Leios values, including a max EB size of 0. Your testnet exception had been hiding this. Without it, a node whose selection or immutable tip is still in Conway rejects every announcing Dijkstra header, and the LeiosNotify path disconnects honest peers. I fixed the Conway→Dijkstra forecast to use Dijkstra's real parameters. The new ThreadNet test forks at epoch 1; it fails without the fix and passes with it.
• Startup forgetting of certifying blocks: it also dropped the certificate of the block at the immutable tip, which breaks moving its EB into immutable storage after a restart. Fixed (one line, no dedicated test).

[ch1bo]

Rebased onto leios-prototype and addressed my remaining comments. I have six commits on top, ready to push to this branch:

• consensus-test: build with GHC 9.6 — this is the 9.6.7 CI failure (-Wno-x-partial), plus a foldl' import 9.6 would have hit next
• Put orphaned haddocks back on their declarations
• Leios: derive the committee from the ledger view only
• LeiosNotify server: offer each part of an EB at most once per peer — fixes the "late join" failure @pgrange reported
• Forecast Dijkstra views from Conway with Dijkstra's Leios parameters
• ChainDB: keep the immutable tip's cert when forgetting CertRBs at start-up

Must: the cross-era forecast. Forecasting Dijkstra from a Conway state made up the Leios fields, including a max EB size of 0. The testnet kludge masked it. Without the kludge, a node whose selection or immutable tip is still in Conway rejects every announcing Dijkstra header and drops honest peers via LeiosNotify. No Leios ThreadNet crossed that boundary (they all fork at epoch 0), so there is now one that forks at epoch 1. It fails without the fix. This matters for the musashi respin and other hard-forks.

Could you also update the PR description? It still says batchRetrieveTxs takes a byte budget, but we dropped that.

TODO Downstream: LeiosMissingThreshold is gone, so the cardano-node commit needs to drop its two clauses in Tracers/ChainDB.hs.

[ch1bo]

Dropped. It was also hiding a real bug: the Dijkstra view forecast from Conway had a max EB size of 0. That's fixed in "Forecast Dijkstra views from Conway with Dijkstra's Leios parameters" (see the review summary).

[ch1bo]

Fixed, and three more of the same kind (praosSharedBlockForging, test_multipleSlotsSameHash and the LedgerSupportsProtocol class).

[ch1bo]

Done. It now lives next to its only caller in Protocol.Leios and takes the PraosWithLeios ledger view. certificationGapOfPeriods stays the shared core with minCertificationGap.

[ch1bo]

Yes, retired; see the thread below.

[ch1bo]

Should: mark the body write as in flight, so a second arrival is not novel.

novel only turns false once the async settle runs acquireEbBody. A second delivery of the same body in that window gets written again, and each writeEbBody notifies AcquiredEb. That is what made the server repeat its offer in the "late join" failure. I guarded the server side (offer each part once per peer), but the double write is still there. On SQLite it also resets missingTxCount and can re-notify AcquiredEbTxs for a complete closure (sql_init_missing_tx_count has no IS NULL guard).

The comment above ("keyed off the lock, two peers delivering the same body cannot both write it") and the one on acquireEbBody ("a concurrent offer may redundantly re-fetch") contradict each other. Which one do we want?

[ch1bo]

Merging despite the "late join" threadnet tests timing out. I saw various seeds succeed and they might have regressed in their overall lifetime. But also don't want to hold up integration into the prototype any longer - we need to soak it before we can cut a release again.