Leios: omnibus for production-grade EB diffusion

cabal.project

@@ -56,12 +56,12 @@ allow-older:
 constraints:
   , any.crypton < 1.1
 
--- Points to ouroboros-network/leios-prototype
+-- Points to ouroboros-network/leios-prototype + matchBlock commit
 source-repository-package
   type: git
   location: https://github.com/IntersectMBO/ouroboros-network
-  tag: 4b3ab7664f609a1aee0f0c24dcfcfd0ab899fc42
-  --sha256: sha256-fIqpI2whstetodpiONmUDVgnhTpzXeq1/gmzuHItK5A=
+  tag: a48ea7a88cd8cc0349013c3b9f99fd3444f26c3b
+  --sha256: sha256-VY0/lP3VFMKwQyCTE7PKzZyBjsR1ZSF77ftwEhn2U/U=
   subdir:
     ouroboros-network
     cardano-diffusion
@@ -80,8 +80,8 @@ source-repository-package
 source-repository-package
   type: git
   location: https://github.com/IntersectMBO/cardano-ledger
-  tag: d93c654699c8f10c883a625168f65dd387abc695
-  --sha256: sha256-iRshuIFnyh6a+NfxXmkfbOzbbMm5YUcezwHyRQwp7dg=
+  tag: f832964ff2d928be03b81db757711226b0ada703
+  --sha256: sha256-M4qjoSjiw3bPsa+N1dBb1DU46//Dl991WzwxisZ4B90=
   subdir:
     libs/cardano-data
     libs/cardano-ledger-api

changelog.d/20260918_170000_nick.frisby_validated_header_annotations.md

@@ -0,0 +1,29 @@
+<!--
+A new scriv changelog fragment.
+
+Uncomment the section that is right (remove the HTML comment wrapper).
+For top level release notes, leave all the headers commented out.
+-->
+
+### Breaking
+
+- Removed `mkHeaderWithTime` and `mkHeadersWithTime`. They rebuilt the annotations after the fact from an near-enough ledger states, which the new annotation can't tolerate. The two components that validate now emit `HeaderWithTime`s directly: the ChainSync client, and ChainSel by way of the LedgerDB.
+
+- `ValidateResult`'s `ValidateSuccessful` and `ValidateLedgerError` now carry the validated headers of the blocks that were successfully applied, which isn't necessarily all of the blocks that were given in the `ValidateLedgerError` case.
+
+- `SuccessForkerAction` gained a block type parameter and receives that fragment, this being the only point at which it and the `Forker` are both in hand.
+
+- `applyBlock` returns the block it applied alongside the resulting state, and `applyThenPush` returns that block's header and the state it pushed, so that a caller which passed a reference need not resolve it a second time.
+
+- `ChainSel`'s `ValidationResult`, `chainSelection` and `initialChainSelection` traffic in `ChainDiff (HeaderWithTime blk)` and `AnchoredFragment (HeaderWithTime blk)` rather than plain headers.
+
+<!--
+
+### Non-Breaking
+
+
+### Patch
+
+- A bullet item for the Patch category.
+
+-->

changelog.d/20260918_193000_nick.frisby_leios_cert_predecessor_view.md

@@ -0,0 +1,34 @@
+<!--
+A new scriv changelog fragment.
+
+Uncomment the section that is right (remove the HTML comment wrapper).
+For top level release notes, leave all the headers commented out.
+-->
+
+### Breaking
+
+- `HeaderWithTime` gained `hwtLedgerViewOfPredecessor`, the ledger view at the slot of the header's predecessor.
+
+- `HeaderStateWithTime` gained `hswtLedgerView`, the ledger view at the tip of its `HeaderState`, likewise mandatory.
+
+- `mkHeaderStateWithTimeFromSummary` takes the corresponding `LedgerState`.
+
+- The ChainDB's add API takes a `Predecessor blk`: either `NoPredecessor`, or the predecessor's slot together with the ledger view at that slot.
+  `trivialPredecessor` builds one for blocks whose `LedgerView` is `()`, which is the only way to supply a view without having genuinely calculated one.
+
+- `MatchedBlock`'s `matchedBlockPredecessorSlot` became `matchedBlockPredecessor`, carrying what the matched header records about the block's predecessor.
+
+- `InitChainDB`'s `addBlock` became `addTheFirstEbb :: blk -> m ()`. That interface only ever seeds an empty chain — both implementations add a single genesis EBB — so it no longer accepts anything about a predecessor.
+
+### Non-Breaking
+
+- Added `ledgerViewOfTip`, the ledger view at a `LedgerState`'s own tip slot. It asks the state's own forecast for the slot that forecast is anchored at, which every implementation answers by projection, so it costs no TICKF and no ticked state.
+
+- `precheckLeiosCert` now returns `Either LeiosExtValidationError ()`: it can no longer fail to reach a verdict.
+
+<!--
+### Patch
+
+- A bullet item for the Patch category.
+
+-->

changelog.d/20260921_100250_nick.frisby_leios_forget_certrbs_at_startup.md

@@ -0,0 +1,23 @@
+<!--
+A new scriv changelog fragment.
+
+Uncomment the section that is right (remove the HTML comment wrapper).
+For top level release notes, leave all the headers commented out.
+-->
+
+### Breaking
+
+- The `VolatileDB` gained `forgetLeiosCertsAtStartUpExcept`. It forgets every block carrying a Leios certificate but the given ones: a forgotten block is stored but unknown, so `getBlockInfo`, `filterByPredecessor` and `getBlockComponent` all answer as though the DB does not hold it, while `putBlock` still recognises its hash and so writes nothing if the block is added again, which also re-admits it.
+
+- The `VolatileDB`'s `TraceEvent` gained `ForgotLeiosCertBlocks` and `ReadmittedForgottenBlock`.
+
+### Non-Breaking
+
+- After initial chain selection, the ChainDB now forgets the cert-carrying blocks that are not on that selection, so they are re-fetched and re-added if some chain it would select turns out to need them. That is what lets their certificates be verified at all: such a block would otherwise never pass through `chainSelAddBlock` again, so nothing would hand it the ledger view its certificate must be checked against, and its claim would stay unverified for the rest of the run — closing the Recovery Path for the endorser block it certifies. See `docs/website/contents/explanations/valid_claims_startup.md`.
+
+<!--
+### Patch
+
+- A bullet item for the Patch category.
+
+-->

changelog.d/20260922_124500_nick.frisby_leios_valid_claims_internal.md

@@ -0,0 +1,25 @@
+<!--
+A new scriv changelog fragment.
+
+Uncomment the section that is right (remove the HTML comment wrapper).
+For top level release notes, leave all the headers commented out.
+-->
+
+### Breaking
+
+- The `ChainDB` gained `getLeiosValidClaims`, which reads the current ValidClaims value with a fingerprint that changes whenever a claim is added.
+
+- `ValidClaims` now records the endorser block each claim announced, so `insertValidClaim` takes it and `isCertifiedEb` answers whether an endorser block's announcement is certified.
+
+- `leiosFetchLogicIteration` takes a predicate deciding which endorser blocks may be fetched at all. An offer it rejects is skipped rather than dropped, so it is reconsidered when the predicate later admits it.
+
+### Non-Breaking
+
+- The Leios fetch logic now only fetches an endorser block once some verified claim announced it, and the node wakes that logic when a claim arrives after the offer did. This is deliberately stricter than the Recovery Path rule it stands in for --- notably it ignores offers heralded only by LeiosNotify --- and is expected to be weakened.
+
+<!--
+### Patch
+
+- A bullet item for the Patch category.
+
+-->

changelog.d/20260924_120000_nick.frisby_leios_offer_validation.md

@@ -0,0 +1,42 @@
+<!--
+A new scriv changelog fragment.
+
+Uncomment the section that is right (remove the HTML comment wrapper).
+For top level release notes, leave all the headers commented out.
+-->
+
+### Breaking
+
+- A peer's offers are now `PeerOffer`, recording the size it offered an EB's body at (if it offered the body) and whether it offered the closure. Now that they're independent offers, `AlsoOfferedTxsClosure` is accordingly `WhetherTxsClosureOffered`, with `TxsClosureOffered` and `TxsClosureNotOffered`.
+
+- `recordEbBodyOffer` takes no `LeiosOutstanding`: it writes only to the offering peer's own record. It gained two siblings: `recordEbClosureOffer`, and `recordCertRbOffer` for a CertRB roll-forward, which offers both the body and the closure. None of the three makes the node start tracking an endorser block, since only an announcement justifies that, not an offer.
+
+- `ValidClaims`' `certifiedEbs` maps each election to the whole `AnnouncementFields` rather than just its `EbHash`. That is what lets `focusCertifiedEb` --- which moves a verified certificate's election onto the EB it names --- also start tracking that endorser block, via the new `trackCertifiedEb`.
+
+- `LeiosNotifyPeerState` holds what one LeiosNotify client remembers of its peer --- its latest-pruned slot, its announcements and its offers. `checkLeiosBlockOffer` and `checkLeiosClosureOffer` accept or reject an offer against it, reporting `ExnLeiosInvalidOffer`, and `pruneLeiosNotifyPeerStateToImmTip` prunes it, replacing `prunePeerStateToImmTip`.
+
+- `NodeKernelArgs` and `NodeKernel` gained `LeiosMinOfferLead`, how much younger than the node's own immutable tip an EB must be for it to be offered (aka relayed) when acquired.
+
+- The LeiosNotify server decides whether a peer hears about what this node stored, consulting `Leios.leiosOfferRelayDecision` as it turns a `LeiosEbNotification` into an offer. The LeiosDb reports what it stored and has no opinion about relaying, so none of this reaches its API.
+
+### Non-Breaking
+
+- A peer may only offer an EB it has announced to us over LeiosNotify, may offer its body once and its closure once, and may not offer one older than our immutable tip. Each of those costs it the connection, which is what stops an offer from being an independent way to make us track an EB, unconstrained by valid elections/announcements. The three checks are consistent by construction: the announcements that pruning to the immutable tip drops are exactly those whose offers that same tip rejects as too old.
+
+- The body offer and the closure offer are independent: either may arrive first, or alone. So a LeiosNotify server never has to synthesise a body offer ahead of a closure offer, and neither `MsgLeiosBlockTxsOffer` nor `AcquiredEbTxs` need a size argument. (If we were to add size argument, then closure offers could again imply body offers. Note that lacking credits is the _only_ reason the body might not have been offered to the peer before the closure would be.)
+
+- This node likewise offers an EB to a peer only if it sent a matching announcement to that peer, since that is what it demands of its own upstream peers. The LeiosNotify server records what it has enqueued to each peer as it enqueues it, so a peer that joins after an announcement was relayed --- or whose queue had no credits when it was --- is not then offered what it was never given announcements for.
+
+- Neither a LeiosNotify offer nor a CertRB's MsgRollForward makes the node start tracking an endorser block any more. Only two things do: an election's first-seen announcement (via LeiosNotify, or via ChainSync of the announcing block), and the verification of a certificate. So nothing is tracked on the strength of a claim we have not yet verified, and what we are willing to fetch is bounded by elections and certifications rather than by what peers say. Offers only say which peers can serve what.
+
+- The announcements on the chain the node has already selected at startup are deliberately not replayed either, though those blocks never roll forward and so are never announced to it again. Between them those two listing paths still reach every EB it could come to need: a CertRB on that chain has its EB already, since it could not have been selected otherwise, and one that is not on that chain is re-fetched if some chain the node would select needs it, at which point verifying its certificate sets the focus and lists it. The only cost is that the `LeiosTxCache` has no entry for those EBs, so their txs read as absent and a later EB that shares one re-fetches it; acceptable consequence of a restart.
+
+- The fetch logic asks a peer for an EB only at the size that peer offered it at. A peer that offered another size holds a different EB, so asking it would only earn a body that fails the arrival size check; its offer is kept rather than dropped, since a later focus change could make it the one we want.
+
+- A node offers an EB only if it was still at least `leiosMinOfferLead` younger than the node's immutable tip, and disconnects a peer that offers one below the local immutable tip. The default is an hour's worth of current mainnet slots, which should almost always cover the gap between two honest node's imm tip's ages. A value approaching the distance between a node's immutable tip and its selection suppresses every offer, which is why the test networks need to configure their own, smaller value.
+
+- A node offers each part of an EB at most once per connection, even when its LeiosDb reports the same acquisition twice. A repeat would make an honest peer disconnect it with `ExnLeiosRepeatedOffer`.
+
+### Patch
+
+- `Cardano.Tools.DBAnalyser.Leios` now imports the `AnnouncementFields` accessors it uses, so `unstable-cardano-tools` builds again.

changelog.d/20260925_150000_nick.frisby_leios_fetch_server_bounds.md

@@ -0,0 +1,52 @@
+<!--
+A new scriv changelog fragment.
+
+Uncomment the section that is right (remove the HTML comment wrapper).
+For top level release notes, leave all the headers commented out.
+-->
+
+### Breaking
+
+- `lookupEbClosure` is now `lookupTrustedEbClosure`, to put its prerequisite in its name: the caller must already have established that the endorser block is trustworthy. A certificate does that for ChainSel applying a cert-RB and for the forge, and the LeiosDb having called the closure complete does it for voting.
+
+- `maxLeiosEbBytesSize` and `maxLeiosTxsRequestBytesSize` replace `maxMsgLeiosBlockBytesSize` and `msgLeiosBlockFramingSize`. They are the design ceilings the logic enforces --- an endorser block's transaction references, and the transactions one `MsgLeiosBlockTxs` carries --- and are unrelated quantities that happen to be close in size. `byteLimitsLeiosFetch` allows `addSafetyMargin` above each, which is what pays for the framing; nothing else may spend that slack as capacity.
+
+  `maxLeiosEbBytesSize` ought to be 500 KiB and is 512 KiB, under protest. `exampleDijkstraGenesis` in cardano-ledger's testlib sets `maxEndorserBlockReferencesSize` to 512 KiB, which is wrong, and `guardLeiosWireLimit` refuses to run with a protocol parameter above our ceiling --- so the Leios ThreadNet, which hard-forks the nodes into that genesis, cannot start until either the ledger is corrected or we accommodate it. Carrying the inflated value keeps this change local; the constant has a TODO. Real deployments are unaffected: the testnet and proto-devnet genesis files both set the parameter to 100,000.
+
+- `leiosFetchClientPeerPipelined` takes a pipelining depth. At that depth it waits for a response instead of pipelining another request. It had no depth limit before, and sent as fast as the fetch logic supplied jobs.
+
+  The outstanding-bytes budget was the only brake, and it does not bound the pipelining depth. An endorser block missing one small transaction costs a whole request and almost no bytes, so many such blocks mean many outstanding requests while staying well within budget.
+
+  `Ouroboros.Consensus.Network.NodeToNode` now exports `leiosFetchPipelineDepth`, which is 150. An honest client never needs more than 140. `maxRequestedBytesSizePerBigLedgerPeer` allows five endorser blocks of the largest permitted size; each is 192 jobs of `maxJobBytesSize`; and one request holds 7 of those, because an eighth would exceed `maxRequestBytesSize` and a job may not span two requests.
+
+  This is the depth the node uses as a client. How many requests an incoming peer may leave outstanding is bounded by the ingress queue's size rather than by a depth, which suffices while the LeiosFetch server is not LookAhead-enabled (see PR https://github.com/IntersectMBO/typed-protocols/pull/93).
+
+- Thus, `maxTxsPerEb` falls from 71,428 to 14,979, since it now derives from the design ceiling rather than from `largeByteLimit`. It sizes the two buffers a LeiosFetch server allocates per inbound peer, which go from 1.09 MiB to 234 KiB of pointer array between them; it is also the largest bitmap word index that server will accept, 1,116 before and 234 now, and a factor of `worstCaseCacheTxCount`.
+
+### Non-Breaking
+
+- Test coverage for Issue https://github.com/input-output-hk/ouroboros-leios/issues/1115: an endorser block whose body misstates the size of a transaction the node already holds never has its closure called complete, so it is neither voted for nor offered onward. The behaviour itself is the LeiosDb's, which refuses the transaction at write time --- a row is pre-allocated as `zeroblob` of the size the body declared, and every fill is guarded on that length, including a cross-EB fill, which is checked against the destination body's own claim. What is new here is a node-versus-environment test that exercises it end to end, in both arrival orders.
+
+- The LeiosFetch server refuses a request it cannot answer, rather than replying with whatever it found, and the refusal costs the asking peer its connection (`ExnLeiosInvalidRequest`). Asked for an endorser block whose body it does not hold, it used to build one out of the nothing it found and send that --- a reply the requester cannot tell from a real one until it hashes it. Asked for offsets an endorser block does not have, it used to answer with fewer transactions than were requested, which is itself a protocol violation.
+
+  It does not check that it ever *offered* what was asked for. That would need per-peer state the LeiosFetch server does not keep, and refusing something we can serve buys nothing. One consequence is not yet handled: an honest peer that asks for something pruned between the offer and the request also loses the connection.
+
+- There is nobody to disconnect: the issuer is not a peer we are connected to, and the peer that relayed the body may be honest. The sanction is that the endorser block is never certified. Its `missingTxCount` stays positive forever, which is the (buried) signal that something was refused.
+
+### Patch
+
+- `unstable-consensus-testlib` gained `mkLeiosTestEbClaiming`, which builds an endorser block stating a size other than a transaction's own --- what an adversary would do, and otherwise unrepresentable.
+
+- Four fixes to the Leios node-versus-environment harness, each of which was quietly making the node under test behave unlike a real node.
+
+  It ran `nullLeiosTxCache`, whose lookups always miss, so the node re-fetched every transaction it already held. Any test about the node *not* fetching was vacuous. It now runs the reference cache.
+
+  Its downstream LeiosNotify client was the non-pipelined one, so it kept no requests outstanding and the node --- by design --- dropped the offers it had no credit to send. A test could conclude the node never made an offer it had in fact made. It now pipelines to `leiosNotifyPipelineDepth`.
+
+  Its LeiosFetch server ignored the request's bitmaps and replied with the endorser block's whole closure. That is a protocol violation, and for a large closure it also breached the message size limit, killing the connection for a reason unrelated to whatever was under test. It now serves exactly the offsets asked for. A peer planted with an empty closure now stalls on a closure request --- a peer that holds the body and withholds the transactions, which is the one thing no peer can be caught at --- rather than answering with fewer transactions than were asked for.
+
+  The harness also gained the other direction of LeiosFetch, so a test can have a downstream peer request something of the node's own fetch server, including things no honest peer would ask for.
+
+- `Test.LeiosDemoDb`'s notification tests inserted four-byte transactions under hashes their endorser block claimed were 100-odd bytes. Hash-only matching made that harmless; it is not harmless any more, and one of those tests hung rather than failed. They now insert transactions of the size the endorser block claims.
+
+- `Test.LeiosDemoDb` also gained parity coverage, run against both the in-memory and the SQLite backends: that a body misstating a size never has its closure announced. The two backends reach that answer by different means --- `zeroblob` pre-allocation plus a `length` guard in SQL, against a declared-size check in the in-memory accept filter --- so nothing but a shared test says they agree.