diff --git a/changelog.d/20261007_120000_damian.nadales_eb_forge_interface.md b/changelog.d/20261007_120000_damian.nadales_eb_forge_interface.md index 36671ac5b0..13b75a2e85 100644 --- a/changelog.d/20261007_120000_damian.nadales_eb_forge_interface.md +++ b/changelog.d/20261007_120000_damian.nadales_eb_forge_interface.md @@ -8,14 +8,14 @@ For top level release notes, leave all the headers commented out. ### Breaking - `TxLimits` has the new superclass constraint `TxEbMeasureByteSize (TxEbMeasure blk)`, and `CanHardFork` has `TxEbMeasureByteSize (HardForkTxEbMeasure xs)`. The new class `TxEbMeasureByteSize` has one method, `txEbMeasureByteSize`: the size of the transaction references that an endorser-block measure counts, without the CBOR framing. `TxMeasure blk` and `IgnoringOverflow ByteSize32` return zero, and `DijkstraEbMeasure` returns its `txReferencesSize`. A block with a new endorser-block measure type must give an instance. The EB-forged machine trace reports this size as `txReferencesSize`. -- `forgeBlock` returns the new record `ForgeBlockResult`. Its field `rankingBlock` is the forged block, and its field `endorserBlock` is the `Maybe LeiosEb` that the block announces. Only Dijkstra (`Praos2`) builds an endorser block, and only from a non-empty `fbEbTxs`. Every other era returns `Nothing`. +- `forgeBlock` returns the new record `ForgeBlockResult`. Its field `rankingBlock` is the forged block, and its field `endorserBlock` is the `Maybe ForgedEndorserBlock` that the block announces. Only Dijkstra (`Praos2`) builds an endorser block, and only from a non-empty `fbEbTxs`. Every other era returns `Nothing`. - `forgeShelleyBlock` returns a `ForgeBlockResult`, and requires `Foldable (WhenLeios proto)`. If the protocol has Leios and `fbEbTxs` is not empty, the header announces the endorser block that `forgeShelleyBlock` returns. - `praos2SharedBlockForging` requires `LedgerSupportsProtocol (ShelleyBlock (Praos2 c) era)`. Its `forgeBlock` asserts that the endorser block it returns is not larger than the `llvMaxEbBodySize` of the `plvLeios` ledger view. - `ForgeBlockArgs` has a new field, `fbEbTxs`: the transactions for the endorser block. `snapshotPartitionWithInitialPayload` selects them after the ranking-block transactions in `fbTxs`. -- `Ouroboros.Consensus.NodeKernel.Forge.forge` takes a new argument. It calls this argument with each forged endorser block after `forgeBlock` and before it adds the ranking block to the ChainDB. The node kernel and db-synthesizer pass a function that does nothing. +- `Ouroboros.Consensus.NodeKernel.Forge.forge` takes a new argument. It calls this argument with each forged endorser block after `forgeBlock` and before it adds the ranking block to the ChainDB. The node kernel passes `storeForgedEb`, and db-synthesizer passes a function that does nothing. - `TraceForgeEvent` has a new constructor, `TraceForgedEndorserBlock`. `forge` traces it after `TraceForgedBlock` when `forgeBlock` returns an endorser block. The `tracing` library renders it under the namespace `ForgedEndorserBlock`, with the hash of the ranking block that announces the endorser block, the endorser block hash, the number of references, the size in bytes, and the selected transactions' measure (transaction bytes, execution units, and reference-script bytes). ### Non-Breaking - `Ouroboros.Consensus.NodeKernel.Forge.forge` gives `ebCapacityTxMeasure` to `snapshotPartitionWithInitialPayload`. If the mempool has more transactions than the Dijkstra ranking block takes, the node forges an endorser block with the next transactions. -- New functions in `Ouroboros.Consensus.Leios.Types`: `leiosEbFromTxBytes` builds an endorser block from the bytes of its transactions, and `hashLeiosEb` gives the hash that a header announces for an endorser block. +- New in `Ouroboros.Consensus.Leios.Types`: the record `ForgedEndorserBlock` holds an endorser block and the bytes of its transactions. `forgeLeiosEb` builds a `ForgedEndorserBlock` from the bytes of the transactions. diff --git a/changelog.d/20261022_120000_damian.nadales_eb_forge_store.md b/changelog.d/20261022_120000_damian.nadales_eb_forge_store.md new file mode 100644 index 0000000000..8c1fda57cd --- /dev/null +++ b/changelog.d/20261022_120000_damian.nadales_eb_forge_store.md @@ -0,0 +1,18 @@ + + +### Breaking + +- `ShelleyBasedEra` has a new method, `getLeiosEraDict`. It returns what the Leios protocol needs to know about the era. Only Dijkstra supports Leios, so only Dijkstra returns `Just`. +- `ShelleyBasedEra` loses the method `blockBodyLeiosCert` and `defaultBlockBodyLeiosCert`. `blockMatchesHeader` and `blockLeiosCert` read the certificate with `leiosCertBlockBodyL` from `getLeiosEraDict`. +- `serializeTxForLeios` from `Ouroboros.Consensus.Shelley.Ledger.Forge` needs `SafeToHash (Tx TopTx era)`. It uses `toCBORForBlockInclusion`, which keeps the `is_valid` flag. + +### Non-Breaking + +- The node kernel writes each forged endorser block to the LeiosDB. The forge does not await the writes. On SQLite, it blocks while the write queue is full, and if a write fails, the node stops. +- The depth of the LeiosDB SQLite write queue counts three jobs for the forge: the point, the body and the transactions of a forged endorser block. +- `Ouroboros.Consensus.Storage.LeiosDB.API` exports `storeForgedEb`, which writes a forged endorser block to the LeiosDB. The node kernel gives it to `forge`. diff --git a/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Eras.hs b/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Eras.hs index e3f225f1c5..fdfe94a721 100644 --- a/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Eras.hs +++ b/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Eras.hs @@ -19,6 +19,7 @@ module Ouroboros.Consensus.Shelley.Eras -- * Shelley-based era , ConwayEraGovDict (..) + , LeiosEraDict (..) , ShelleyBasedEra (..) -- * Convenience functions @@ -29,7 +30,6 @@ module Ouroboros.Consensus.Shelley.Eras ) where import Cardano.Binary (FromCBOR, ToCBOR) -import Cardano.Crypto.Leios (LeiosCert) import Cardano.Ledger.Allegra (AllegraEra) import Cardano.Ledger.Allegra.Translation () import Cardano.Ledger.Alonzo (AlonzoEra, ApplyTxError (AlonzoApplyTxError)) @@ -48,7 +48,7 @@ import qualified Cardano.Ledger.Conway.Rules as SL ) import qualified Cardano.Ledger.Conway.State as CG import Cardano.Ledger.Dijkstra (ApplyTxError (DijkstraApplyTxError), DijkstraEra) -import Cardano.Ledger.Dijkstra.BlockBody (leiosCertBlockBodyL) +import Cardano.Ledger.Dijkstra.BlockBody (DijkstraEraBlockBody) import qualified Cardano.Ledger.Dijkstra.Rules as Dijkstra import qualified Cardano.Ledger.Dijkstra.Rules as SL ( DijkstraLedgerPredFailure (..) @@ -137,6 +137,9 @@ class -- | Whether the era has an instance of 'CG.ConwayEraGov' getConwayEraGovDict :: proxy era -> Maybe (ConwayEraGovDict era) + -- | The 'LeiosEraDict' of the era, if the era supports Leios. + getLeiosEraDict :: proxy era -> Maybe (LeiosEraDict era) + mkEraMkMempoolApplyTxError :: proxy era -> Maybe (Text -> SL.ApplyTxError era) @@ -146,15 +149,14 @@ class getShelleyEraPerasRoundLength :: proxy era -> PerasEnabled PerasRoundLength getShelleyEraPerasRoundLength _ = NoPerasEnabled - -- | The Leios certificate this block body carries, if any. - -- - -- Eras that don't support Leios define this as - -- 'defaultBlockBodyLeiosCert'. - blockBodyLeiosCert :: Core.BlockBody era -> StrictMaybe LeiosCert - data ConwayEraGovDict era where ConwayEraGovDict :: (CG.ConwayEraGov era, CG.ConwayEraCertState era) => ConwayEraGovDict era +-- | What the Leios protocol needs to know about an era. Only an era that +-- supports Leios has one. +data LeiosEraDict era where + LeiosEraDict :: (SafeToHash (Core.Tx TopTx era), DijkstraEraBlockBody era) => LeiosEraDict era + isBeforeConway :: forall era. L.Era era => Proxy era -> Bool isBeforeConway _ = L.eraProtVerLow @era < L.eraProtVerLow @L.ConwayEra @@ -184,69 +186,71 @@ defaultApplyShelleyBasedTx globals ledgerEnv mempoolState _wti tx = defaultGetConwayEraGovDict :: proxy era -> Maybe (ConwayEraGovDict era) defaultGetConwayEraGovDict _ = Nothing -defaultBlockBodyLeiosCert :: Core.BlockBody era -> StrictMaybe LeiosCert -defaultBlockBodyLeiosCert _ = SNothing +defaultGetLeiosEraDict :: proxy era -> Maybe (LeiosEraDict era) +defaultGetLeiosEraDict _ = Nothing instance ShelleyBasedEra ShelleyEra where applyShelleyBasedTx = defaultApplyShelleyBasedTx getConwayEraGovDict = defaultGetConwayEraGovDict - mkEraMkMempoolApplyTxError _prx = Nothing + getLeiosEraDict = defaultGetLeiosEraDict - blockBodyLeiosCert = defaultBlockBodyLeiosCert + mkEraMkMempoolApplyTxError _prx = Nothing instance ShelleyBasedEra AllegraEra where applyShelleyBasedTx = defaultApplyShelleyBasedTx getConwayEraGovDict = defaultGetConwayEraGovDict - mkEraMkMempoolApplyTxError _prx = Nothing + getLeiosEraDict = defaultGetLeiosEraDict - blockBodyLeiosCert = defaultBlockBodyLeiosCert + mkEraMkMempoolApplyTxError _prx = Nothing instance ShelleyBasedEra MaryEra where applyShelleyBasedTx = defaultApplyShelleyBasedTx getConwayEraGovDict = defaultGetConwayEraGovDict - mkEraMkMempoolApplyTxError _prx = Nothing + getLeiosEraDict = defaultGetLeiosEraDict - blockBodyLeiosCert = defaultBlockBodyLeiosCert + mkEraMkMempoolApplyTxError _prx = Nothing instance ShelleyBasedEra AlonzoEra where applyShelleyBasedTx = applyAlonzoBasedTx getConwayEraGovDict = defaultGetConwayEraGovDict - mkEraMkMempoolApplyTxError _prx = Nothing + getLeiosEraDict = defaultGetLeiosEraDict - blockBodyLeiosCert = defaultBlockBodyLeiosCert + mkEraMkMempoolApplyTxError _prx = Nothing instance ShelleyBasedEra BabbageEra where applyShelleyBasedTx = applyAlonzoBasedTx getConwayEraGovDict = defaultGetConwayEraGovDict - mkEraMkMempoolApplyTxError _prx = Nothing + getLeiosEraDict = defaultGetLeiosEraDict - blockBodyLeiosCert = defaultBlockBodyLeiosCert + mkEraMkMempoolApplyTxError _prx = Nothing instance ShelleyBasedEra ConwayEra where applyShelleyBasedTx = applyAlonzoBasedTx getConwayEraGovDict _ = Just ConwayEraGovDict + getLeiosEraDict = defaultGetLeiosEraDict + mkEraMkMempoolApplyTxError _prx = Just $ \txt -> ConwayApplyTxError (NE.singleton (Conway.ConwayMempoolFailure txt)) - blockBodyLeiosCert = defaultBlockBodyLeiosCert - instance ShelleyBasedEra DijkstraEra where applyShelleyBasedTx = applyAlonzoBasedTx getConwayEraGovDict _ = Just ConwayEraGovDict + getLeiosEraDict _ = Just LeiosEraDict + -- TODO we'll need to change the mini protocol (backwards-incompatibly?) to -- use MempoolFailure type family instead of just PredicateFailure type -- family @@ -254,8 +258,6 @@ instance ShelleyBasedEra DijkstraEra where getShelleyEraPerasRoundLength _ = dijkstraPerasRoundLength - blockBodyLeiosCert bb = bb ^. leiosCertBlockBodyL - applyAlonzoBasedTx :: forall era. ( AlonzoEraTx era diff --git a/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Ledger/Block.hs b/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Ledger/Block.hs index 66205664ee..bdb27bb126 100644 --- a/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Ledger/Block.hs +++ b/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Ledger/Block.hs @@ -40,6 +40,7 @@ module Ouroboros.Consensus.Shelley.Ledger.Block ) where import qualified Cardano.Crypto.Hash as Crypto +import Cardano.Crypto.Leios (LeiosCert) import Cardano.Ledger.Binary ( Annotator (..) , DecCBOR (..) @@ -55,17 +56,19 @@ import Cardano.Ledger.Core as SL , toEraCBOR ) import qualified Cardano.Ledger.Core as SL - ( TranslationContext + ( BlockBody + , TranslationContext , hashBlockBody , txSeqBlockBodyL ) +import Cardano.Ledger.Dijkstra.BlockBody (leiosCertBlockBodyL) import Cardano.Ledger.Hashes (HASH) import qualified Cardano.Ledger.Shelley.API as SL import Cardano.Protocol.Crypto (Crypto) import qualified Cardano.Protocol.TPraos.BlockHeader as SL import qualified Data.ByteString.Lazy as Lazy import Data.Coerce (coerce) -import Data.Maybe.Strict (isSJust, strictMaybeToMaybe) +import Data.Maybe.Strict (StrictMaybe (..), isSJust, strictMaybeToMaybe) import Data.Typeable (Typeable) import GHC.Generics (Generic) import Lens.Micro ((^.)) @@ -233,14 +236,21 @@ instance ShelleyCompatible proto era => GetHeader (ShelleyBlock proto era) where where ShelleyHeader{shelleyHeaderRaw = shelleyHdr} = hdr ShelleyBlock{shelleyBlockRaw = SL.Block _ blockBody} = blk - bodyContainsCert = isSJust (blockBodyLeiosCert blockBody) + bodyContainsCert = isSJust (bodyLeiosCert blockBody) bodyContainsTxs = not (null (blockBody ^. SL.txSeqBlockBodyL)) headerIsEBB = const Nothing instance ShelleyCompatible proto era => BlockSupportsLeios (ShelleyBlock proto era) where blockLeiosCert ShelleyBlock{shelleyBlockRaw = SL.Block _ blockBody} = - strictMaybeToMaybe (blockBodyLeiosCert blockBody) + strictMaybeToMaybe (bodyLeiosCert blockBody) + +-- | The Leios certificate of a block body. Only an era with a 'LeiosEraDict' +-- has a certificate field. +bodyLeiosCert :: forall era. ShelleyBasedEra era => SL.BlockBody era -> StrictMaybe LeiosCert +bodyLeiosCert blockBody = case getLeiosEraDict (Proxy @era) of + Just LeiosEraDict -> blockBody ^. leiosCertBlockBodyL + Nothing -> SNothing mkShelleyHeader :: ShelleyCompatible proto era => diff --git a/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Ledger/Forge.hs b/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Ledger/Forge.hs index 0f3155fdb1..ef927e129d 100644 --- a/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Ledger/Forge.hs +++ b/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Ledger/Forge.hs @@ -4,7 +4,7 @@ {-# LANGUAGE TypeApplications #-} module Ouroboros.Consensus.Shelley.Ledger.Forge - ( ebTxBytes + ( serializeTxForLeios , forgeShelleyBlock ) where @@ -18,7 +18,8 @@ import qualified Cardano.Ledger.Core as SL , mkBasicBlockBody , txSeqBlockBodyL ) -import Cardano.Ledger.Hashes (unsafeMakeSafeHash) +import Cardano.Ledger.Dijkstra.Tx (toCBORForBlockInclusion) +import Cardano.Ledger.Hashes (SafeToHash, unsafeMakeSafeHash) import qualified Cardano.Ledger.Shelley.API as SL (Block (..), extractValidatedTx) import qualified Cardano.Protocol.TPraos.BlockHeader as SL import Control.Exception @@ -33,14 +34,15 @@ import Ouroboros.Consensus.Config import Ouroboros.Consensus.Ledger.Abstract import Ouroboros.Consensus.Ledger.SupportsMempool import Ouroboros.Consensus.Leios.Types - ( EndorserBlock + ( ForgedEndorserBlock (..) , encodeEndorserBlockSize - , endorserBlockFromTxBytes , endorserBlockHash + , forgeLeiosEb ) import Ouroboros.Consensus.Protocol.Abstract (CanBeLeader) import Ouroboros.Consensus.Protocol.Ledger.HotKey (HotKey) import Ouroboros.Consensus.Protocol.Praos.Common (WhenLeios, pureLeiosOnly) +import Ouroboros.Consensus.Shelley.Eras (LeiosEraDict (..), ShelleyBasedEra (getLeiosEraDict)) import Ouroboros.Consensus.Shelley.Ledger.Block import Ouroboros.Consensus.Shelley.Ledger.Config ( shelleyProtocolVersion @@ -98,14 +100,16 @@ forgeShelleyBlock -- Only a protocol with Leios has a value in 'WhenLeios', so the other -- protocols build no endorser block. CIP-0164 does not allow an empty - -- endorser block. - forgedEb :: WhenLeios proto (Maybe EndorserBlock) + -- endorser block. A protocol with Leios on an era without 'LeiosEraDict' + -- is a bug in how the eras are put together, so we fail loudly. + forgedEb :: WhenLeios proto (Maybe ForgedEndorserBlock) forgedEb = - pureLeiosOnly @proto $ - endorserBlockFromTxBytes . fmap (ebTxBytes . extractTx) <$> nonEmpty fbEbTxs + pureLeiosOnly @proto $ case getLeiosEraDict (Proxy @era) of + Just LeiosEraDict -> forgeLeiosEb . fmap (serializeTxForLeios . extractTx) <$> nonEmpty fbEbTxs + Nothing -> error "forgeShelleyBlock: a protocol with Leios needs an era with a LeiosEraDict" -- TODO Forging does not yet certify endorser blocks. - leiosFields = (,) False . maybeToStrictMaybe . fmap announce <$> forgedEb + leiosFields = (,) False . maybeToStrictMaybe . fmap (announce . forgedEbBody) <$> forgedEb announce eb = EbReferencesAnnouncement @@ -132,9 +136,14 @@ forgeShelleyBlock -- | The bytes of a transaction that an endorser block references. -- --- The length of these bytes must not exceed 'txByteSizeWithOverhead'. --- Otherwise the endorser block can exceed the size that --- 'txEbMeasureDijkstra' reserved for it, and header validation rejects the --- block with 'LeiosEbTooBig'. -ebTxBytes :: forall era. Core.EraTx era => Core.Tx Core.TopTx era -> ByteString -ebTxBytes = serialize' (SL.eraProtVerLow @era) +-- These bytes must keep the @is_valid@ flag, because a peer cannot get it +-- from anywhere else. 'EncCBOR' drops the flag in Dijkstra, so we use the +-- block-body encoding. +-- +-- Their length must not exceed 'txByteSizeWithOverhead'. Otherwise the +-- endorser block can exceed the size that 'txEbMeasureDijkstra' reserved +-- for it, and header validation rejects the block with 'LeiosEbTooBig'. +serializeTxForLeios :: + forall era. + (Core.EraTx era, SafeToHash (Core.Tx Core.TopTx era)) => Core.Tx Core.TopTx era -> ByteString +serializeTxForLeios = serialize' (SL.eraProtVerLow @era) . toCBORForBlockInclusion diff --git a/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Node/Praos.hs b/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Node/Praos.hs index 59bce4d6ea..b199ee8fa5 100644 --- a/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Node/Praos.hs +++ b/ouroboros-consensus-cardano/src/shelley/Ouroboros/Consensus/Shelley/Node/Praos.hs @@ -22,7 +22,7 @@ import Ouroboros.Consensus.Block import Ouroboros.Consensus.Config (configConsensus, configLedger) import Ouroboros.Consensus.Ledger.SupportsProtocol (LedgerSupportsProtocol (..)) import Ouroboros.Consensus.Leios.Protocol (LeiosLedgerView (..)) -import Ouroboros.Consensus.Leios.Types (encodeEndorserBlockSize) +import Ouroboros.Consensus.Leios.Types (ForgedEndorserBlock (..), encodeEndorserBlockSize) import Ouroboros.Consensus.Protocol.Abstract (CanBeLeader, ConsensusConfig) import qualified Ouroboros.Consensus.Protocol.Ledger.HotKey as HotKey import Ouroboros.Consensus.Protocol.Praos @@ -142,8 +142,8 @@ praos2SharedBlockForging hotKey slotToPeriod credentials = basePraosSharedBlockForging (praosParams . leiosPraosConfig) hotKey slotToPeriod credentials checkEbSize ForgeBlockArgs{fbConfig, fbCurrentTickedLedgerState} forged@ForgeBlockResult{endorserBlock} = - -- If a ledger change makes the 'serialize'' bytes of a transaction longer - -- than its 'sizeTxF' plus 'perTxOverhead', the endorser block can exceed + -- If a ledger change makes the 'serializeTxForLeios' bytes of a transaction + -- longer than its 'sizeTxF' plus 'perTxOverhead', the endorser block can exceed -- the size that 'txEbMeasureDijkstra' reserved. Header validation then -- rejects the block with 'LeiosEbTooBig'. The assert makes a test fail at -- the forge instead. @@ -151,4 +151,4 @@ praos2SharedBlockForging hotKey slotToPeriod credentials = where Praos2HasLeios leiosLedgerView = plvLeios $ protocolLedgerView (configLedger fbConfig) fbCurrentTickedLedgerState - fits eb = encodeEndorserBlockSize eb <= llvMaxEbBodySize leiosLedgerView + fits eb = encodeEndorserBlockSize (forgedEbBody eb) <= llvMaxEbBodySize leiosLedgerView diff --git a/ouroboros-consensus-cardano/test/shelley-test/Test/Consensus/Shelley/EndorserBlock.hs b/ouroboros-consensus-cardano/test/shelley-test/Test/Consensus/Shelley/EndorserBlock.hs index 7e81dae983..410c5c1f63 100644 --- a/ouroboros-consensus-cardano/test/shelley-test/Test/Consensus/Shelley/EndorserBlock.hs +++ b/ouroboros-consensus-cardano/test/shelley-test/Test/Consensus/Shelley/EndorserBlock.hs @@ -6,6 +6,7 @@ import Cardano.Binary (ToCBOR (..)) import Cardano.Crypto.Hash (Hash (UnsafeHash)) import Cardano.Ledger.Core (TopTx, Tx) import Cardano.Ledger.Dijkstra (DijkstraEra) +import Cardano.Ledger.Hashes (extractHash, hashAnnotated) import qualified Codec.CBOR.Encoding as CBOR import Codec.CBOR.Write (toStrictByteString) import qualified Data.ByteString as BS @@ -16,12 +17,13 @@ import Ouroboros.Consensus.Leios.Types , EndorserBlock , TxHash , encodeEndorserBlock + , endorserBlockFromTxBytes , endorserBlockReferences , mkEndorserBlock , referencesCapacity , sizeEndorserBlockReference ) -import Ouroboros.Consensus.Shelley.Ledger.Forge (ebTxBytes) +import Ouroboros.Consensus.Shelley.Ledger.Forge (serializeTxForLeios) import Ouroboros.Consensus.Shelley.Ledger.Mempool (txByteSizeWithOverhead) import Test.Cardano.Ledger.Dijkstra.Arbitrary () import Test.Tasty (TestTree, testGroup) @@ -44,6 +46,9 @@ tests = "the tx size a forged endorser block records does not exceed the tx size the mempool records" -- Few runs: the generator of Dijkstra transactions is slow. (withNumTests 10 prop_dijkstraEbTxSizeBounded) + , testProperty + "a forged endorser block references a Dijkstra transaction by its hashAnnotated" + (withNumTests 10 prop_dijkstraEbTxHashIsLedgerHash) ] genTxHash :: Gen TxHash @@ -138,16 +143,27 @@ prop_referencesCapacityFloorsAtZero = , (1, chooseEnum (maxFraming + 1, maxBound)) ] --- | The length of the 'ebTxBytes' of a Dijkstra transaction never exceeds --- its 'txByteSizeWithOverhead', which 'txEbMeasureDijkstra' reserves for the --- reference. +-- | The length of the 'serializeTxForLeios' bytes of a Dijkstra transaction +-- never exceeds its 'txByteSizeWithOverhead', which 'txEbMeasureDijkstra' +-- reserves for the reference. -- --- The ledger computes 'sizeTxF' from one encoding and 'serialize'' uses --- another, so a ledger change can make the two disagree. +-- The ledger computes 'sizeTxF' from one encoding and 'serializeTxForLeios' +-- uses another, so a ledger change can make the two disagree. prop_dijkstraEbTxSizeBounded :: Tx TopTx DijkstraEra -> Property prop_dijkstraEbTxSizeBounded tx = counterexample ("EB tx size: " <> show ebTxSize <> ", mempool tx size: " <> show mempoolTxSize) $ ebTxSize <= mempoolTxSize where - ebTxSize = BS.length (ebTxBytes tx) + ebTxSize = BS.length (serializeTxForLeios tx) mempoolTxSize = fromIntegral (txByteSizeWithOverhead tx) + +-- | 'endorserBlockFromTxBytes' references a transaction by the hash of its +-- 'serializeTxForLeios' bytes. The Dijkstra ledger defines this reference as the +-- transaction's 'hashAnnotated', which is the hash of its 'originalBytes'. +-- Thus a node can check fetched bytes against the reference without decoding +-- them. Fails if 'serializeTxForLeios' returns other bytes, for example the +-- 'EncCBOR' encoding, which drops the @is_valid@ flag. +prop_dijkstraEbTxHashIsLedgerHash :: Tx TopTx DijkstraEra -> Property +prop_dijkstraEbTxHashIsLedgerHash tx = + map fst (endorserBlockReferences (endorserBlockFromTxBytes (pure (serializeTxForLeios tx)))) + === [extractHash (hashAnnotated tx)] diff --git a/ouroboros-consensus-cardano/test/shelley-test/Test/Consensus/Shelley/Forge.hs b/ouroboros-consensus-cardano/test/shelley-test/Test/Consensus/Shelley/Forge.hs index 762ce2f099..c568343c50 100644 --- a/ouroboros-consensus-cardano/test/shelley-test/Test/Consensus/Shelley/Forge.hs +++ b/ouroboros-consensus-cardano/test/shelley-test/Test/Consensus/Shelley/Forge.hs @@ -7,13 +7,11 @@ module Test.Consensus.Shelley.Forge (tests) where import qualified Cardano.Crypto.Hash as Hash import Cardano.Ledger.BaseTypes (StrictMaybe (..)) import qualified Cardano.Ledger.BaseTypes as SL (Globals (..), ProtVer (..)) -import Cardano.Ledger.Binary (serialize') import Cardano.Ledger.Block (EbReferencesAnnouncement (..)) import Cardano.Ledger.Coin (Coin (..)) -import Cardano.Ledger.Core (bodyTxL, eraProtVerLow, feeTxBodyL, ppProtocolVersionL) -import Cardano.Ledger.Dijkstra (DijkstraEra) +import Cardano.Ledger.Core (bodyTxL, feeTxBodyL, ppProtocolVersionL) import Cardano.Ledger.Dijkstra.PParams (ppMaxEndorserBlockReferencesSizeL) -import Cardano.Ledger.Hashes (HASH, unsafeMakeSafeHash) +import Cardano.Ledger.Hashes (HASH, originalBytes, unsafeMakeSafeHash) import Cardano.Ledger.MemoBytes (getMemoRawType) import qualified Cardano.Ledger.Shelley.API as SL ( mkMempoolEnv @@ -26,6 +24,7 @@ import qualified Cardano.Protocol.TPraos.OCert as SL (KESPeriod (..)) import Cardano.Slotting.EpochInfo (fixedEpochInfo) import Control.Tracer (nullTracer) import qualified Data.ByteString as BS +import Data.Foldable (toList) import Data.Maybe (isJust) import Lens.Micro ((&), (.~), (^.)) import Ouroboros.Consensus.Block @@ -41,6 +40,8 @@ import Ouroboros.Consensus.Leios.Types , encodeEndorserBlockSize , endorserBlockHash , endorserBlockReferences + , forgedEbBody + , forgedEbTxs ) import Ouroboros.Consensus.Protocol.Praos ( ConsensusConfig (..) @@ -92,27 +93,30 @@ prop_announcesEb = forAllBlind (genCoreNode (SL.KESPeriod 0)) $ \coreNode -> forAll genNumEbTxs $ \numEbTxs -> ioProperty $ do let ebTxs = take numEbTxs distinctTxs - ForgeBlockResult blk forgedEb <- forgeWith coreNode ebTxs + (_, ForgeBlockResult blk forgedEb) <- forgeWith coreNode ebTxs pure $ case forgedEb of Nothing -> counterexample "no endorser block" False - Just eb -> - conjoin - [ counterexample "the endorser block does not reference the transactions in order" $ - endorserBlockReferences eb === map reference ebTxs - , announcement blk - === SJust - EbReferencesAnnouncement - { ebReferencesAnnouncementHash = unsafeMakeSafeHash (endorserBlockHash eb) - , ebReferencesAnnouncementSize = encodeEndorserBlockSize eb - } - , counterexample "the block does not match its header" $ - blockMatchesHeader (getHeader blk) blk - ] + Just forged -> + let eb = forgedEbBody forged + in conjoin + [ counterexample "the endorser block does not reference the transactions in order" $ + endorserBlockReferences eb === map reference ebTxs + , counterexample "forgedEbTxs does not hold the referenced transaction bytes" $ + toList (forgedEbTxs forged) === map txBytes ebTxs + , announcement blk + === SJust + EbReferencesAnnouncement + { ebReferencesAnnouncementHash = unsafeMakeSafeHash (endorserBlockHash eb) + , ebReferencesAnnouncementSize = encodeEndorserBlockSize eb + } + , counterexample "the block does not match its header" $ + blockMatchesHeader (getHeader blk) blk + ] prop_noEb :: Property prop_noEb = forAllBlind (genCoreNode (SL.KESPeriod 0)) $ \coreNode -> ioProperty $ do - ForgeBlockResult blk forgedEb <- forgeWith coreNode [] + (_, ForgeBlockResult blk forgedEb) <- forgeWith coreNode [] pure $ (announcement blk, isJust forgedEb) === (SNothing, False) -- | A number of endorser-block transactions. @@ -126,12 +130,13 @@ genNumEbTxs = ] -- | Forge the block after the example ledger state, with the given endorser --- block transactions and no ranking block transactions. +-- block transactions and no ranking block transactions. Also return the +-- config of the forge. forgeWith :: CoreNode StandardCrypto -> [GenTx StandardDijkstraBlock] -> - IO (ForgeBlockResult StandardDijkstraBlock) -forgeWith coreNode ebTxs = do + IO (TopLevelConfig StandardDijkstraBlock, ForgeBlockResult StandardDijkstraBlock) +forgeWith coreNode ebTxs = fmap ((,) topLevelConfig) $ do hotKey <- instantiatePraosCredentials (SL.maxKESEvo globals) @@ -236,12 +241,15 @@ distinctTxs = -- | The reference that an endorser block holds for a transaction: the -- Blake2b-256 hash and the length of the transaction bytes. reference :: GenTx StandardDijkstraBlock -> (TxHash, BytesSize) -reference (ShelleyTx _ tx) = - ( Hash.castHash (Hash.hashWith @HASH id bytes) - , fromIntegral (BS.length bytes) +reference tx = + ( Hash.castHash (Hash.hashWith @HASH id (txBytes tx)) + , fromIntegral (BS.length (txBytes tx)) ) - where - bytes = serialize' (eraProtVerLow @DijkstraEra) tx + +-- | The bytes of a transaction that an endorser block references: the +-- block-body encoding, which keeps the @is_valid@ flag. +txBytes :: GenTx StandardDijkstraBlock -> BS.ByteString +txBytes (ShelleyTx _ tx) = originalBytes tx exampleDijkstraBlock :: StandardDijkstraBlock exampleDijkstraBlock = example Examples.exampleBlock diff --git a/ouroboros-consensus-diffusion/src/ouroboros-consensus-diffusion/Ouroboros/Consensus/NodeKernel.hs b/ouroboros-consensus-diffusion/src/ouroboros-consensus-diffusion/Ouroboros/Consensus/NodeKernel.hs index 62764db99d..bb802212ee 100644 --- a/ouroboros-consensus-diffusion/src/ouroboros-consensus-diffusion/Ouroboros/Consensus/NodeKernel.hs +++ b/ouroboros-consensus-diffusion/src/ouroboros-consensus-diffusion/Ouroboros/Consensus/NodeKernel.hs @@ -105,7 +105,12 @@ import Ouroboros.Consensus.Storage.ChainDB.API import qualified Ouroboros.Consensus.Storage.ChainDB.API as ChainDB import Ouroboros.Consensus.Storage.ChainDB.Init (InitChainDB) import qualified Ouroboros.Consensus.Storage.ChainDB.Init as InitChainDB -import Ouroboros.Consensus.Storage.LeiosDB.API (LeiosDbHandle) +import Ouroboros.Consensus.Storage.LeiosDB.API + ( LeiosDbHandle + , LeiosDbWriter (..) + , openWriter + , storeForgedEb + ) import Ouroboros.Consensus.Util.AnchoredFragment ( preferAnchoredCandidate ) @@ -502,6 +507,7 @@ data InternalState m addrNTN addrNTC blk = IS , registry :: ResourceRegistry m , btime :: BlockchainTime m , chainDB :: ChainDB m blk + , leiosDB :: LeiosDbHandle m , blockFetchInterface :: BlockFetchConsensusInterface (ConnectionId addrNTN) @@ -537,6 +543,7 @@ initInternalState NodeKernelArgs { tracers , chainDB + , leiosDB , registry , cfg , blockFetchSize @@ -613,9 +620,9 @@ forkBlockForging IS{..} (MkBlockForging blockForgingM) = forkLinkedWatcherAllocate registry label - blockForgingMLabel - finalize - ( \bf -> knownSlotWatcher btime $ + allocateForging + finalizeForging + ( \(bf, leiosDbWriter) -> knownSlotWatcher btime $ \currentSlot -> withEarlyExit_ $ forge @@ -625,17 +632,23 @@ forkBlockForging IS{..} (MkBlockForging blockForgingM) = chainDB mempool bf - (\_ _ -> pure ()) + (storeForgedEb leiosDbWriter . blockSlot) currentSlot ) where label :: String label = "NodeKernel.blockForging" - blockForgingMLabel = do + allocateForging = do bf <- blockForgingM labelThisThread $ Text.unpack $ forgeLabel bf - pure bf + leiosDbWriter <- openWriter leiosDB + pure (bf, leiosDbWriter) + + -- 'finalize' erases the KES key, so it runs first: 'closeWriter' waits for + -- the LeiosDB write queue. + finalizeForging (bf, leiosDbWriter) = + finalize bf `finally` closeWriter leiosDbWriter {------------------------------------------------------------------------------- TxSubmission integration diff --git a/ouroboros-consensus-diffusion/src/ouroboros-consensus-diffusion/Ouroboros/Consensus/NodeKernel/Forge.hs b/ouroboros-consensus-diffusion/src/ouroboros-consensus-diffusion/Ouroboros/Consensus/NodeKernel/Forge.hs index 6951940dc0..09137ccd55 100644 --- a/ouroboros-consensus-diffusion/src/ouroboros-consensus-diffusion/Ouroboros/Consensus/NodeKernel/Forge.hs +++ b/ouroboros-consensus-diffusion/src/ouroboros-consensus-diffusion/Ouroboros/Consensus/NodeKernel/Forge.hs @@ -35,7 +35,7 @@ import Ouroboros.Consensus.Ledger.SupportsMempool import Ouroboros.Consensus.Ledger.SupportsProtocol import Ouroboros.Consensus.Ledger.Tables.Utils (forgetLedgerTables) import Ouroboros.Consensus.Leios.Protocol (LeiosLedgerView, LeiosProtocol (getLeiosLedgerView)) -import Ouroboros.Consensus.Leios.Types (EndorserBlock) +import Ouroboros.Consensus.Leios.Types (ForgedEndorserBlock (..)) import Ouroboros.Consensus.Mempool import Ouroboros.Consensus.Mempool.API (MempoolMeasure (..)) import Ouroboros.Consensus.Node.Run @@ -73,7 +73,7 @@ forge :: -- | 'forge' calls this function after 'forgeBlock' and before -- 'addBlockToChainDB'. The arguments are the header of the forged ranking -- block and the endorser block that it announces. - (Header blk -> EndorserBlock -> m ()) -> + (Header blk -> ForgedEndorserBlock -> m ()) -> SlotNo -> WithEarlyExit m () forge forgeEventTracer forgeStateInfoTracer cfg chainDB mempool blockForging onForgedLeiosEb currentSlot = do @@ -169,7 +169,7 @@ forge forgeEventTracer forgeStateInfoTracer cfg chainDB mempool blockForging onF TraceForgedEndorserBlock currentSlot (blockHash (Block.rankingBlock forgeRes)) - eb + (forgedEbBody eb) (endorserBlockMeasure selectedTxs) addBlockToChainDB diff --git a/ouroboros-consensus-diffusion/test/consensus-test/Main.hs b/ouroboros-consensus-diffusion/test/consensus-test/Main.hs index b1941df8c0..4b3c86762f 100644 --- a/ouroboros-consensus-diffusion/test/consensus-test/Main.hs +++ b/ouroboros-consensus-diffusion/test/consensus-test/Main.hs @@ -8,6 +8,7 @@ import qualified Test.Consensus.Genesis.Tests (tests) import qualified Test.Consensus.HardFork.Combinator (tests) import qualified Test.Consensus.Leios.Cert (tests) import qualified Test.Consensus.Leios.ChainDB (tests) +import qualified Test.Consensus.Leios.Forge (tests) import qualified Test.Consensus.Leios.LeiosFetch (tests) import qualified Test.Consensus.Leios.LeiosNotify (tests) import qualified Test.Consensus.Leios.Relay (tests) @@ -47,6 +48,7 @@ tests = , Test.Consensus.Leios.Cert.tests , Test.Consensus.Leios.ValidClaims.tests , Test.Consensus.Leios.ChainDB.tests + , Test.Consensus.Leios.Forge.tests , Test.Consensus.Leios.LeiosFetch.tests , Test.Consensus.Leios.LeiosNotify.tests , Test.Consensus.Leios.Relay.tests diff --git a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/HardFork/Combinator.hs b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/HardFork/Combinator.hs index f3847d346a..e20132d3d2 100644 --- a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/HardFork/Combinator.hs +++ b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/HardFork/Combinator.hs @@ -16,11 +16,11 @@ module Test.Consensus.HardFork.Combinator (tests) where -import qualified Cardano.Crypto.Hash as Hash import Cardano.Ledger.BaseTypes (knownNonZeroBounded, nonZero, unNonZero) import Data.ByteString.Short (fromShort) import Data.Function (on) import Data.Functor.Identity (Identity (..)) +import Data.List.NonEmpty (NonEmpty (..), nonEmpty) import qualified Data.Map.Strict as Map import Data.MemPack import Data.SOP.BasicFunctors @@ -55,7 +55,7 @@ import Ouroboros.Consensus.Ledger.Abstract import Ouroboros.Consensus.Ledger.Extended import Ouroboros.Consensus.Ledger.Peras (initPerasState) import Ouroboros.Consensus.Ledger.SupportsMempool -import Ouroboros.Consensus.Leios.Types (EndorserBlock, mkEndorserBlock) +import Ouroboros.Consensus.Leios.Types (ForgedEndorserBlock, forgeLeiosEb) import Ouroboros.Consensus.Node.NetworkProtocolVersion import Ouroboros.Consensus.Node.ProtocolInfo import Ouroboros.Consensus.NodeId @@ -112,7 +112,7 @@ unit_hardForkForgeBlockPassesTxsAndReturnsEb = do case forged of Z (I blkA) -> blkA_body blkA @?= [txA 1, txA 2] S _ -> assertFailure "the block is not from era A" - forgedEb @?= Just (ebListing [txA 3]) + forgedEb @?= Just (mkForgedLeiosEb (txA 3 :| [])) where hardForkForging :: BlockForging Identity TestBlock hardForkForging = @@ -134,14 +134,12 @@ unit_hardForkForgeBlockPassesTxsAndReturnsEb = do pure ForgeBlockResult { rankingBlock = blkA{blkA_body = map forgetValidatedGenTxA (fbTxs argsA)} - , endorserBlock = Just $ ebListing $ map forgetValidatedGenTxA (fbEbTxs argsA) + , endorserBlock = mkForgedLeiosEb <$> nonEmpty (map forgetValidatedGenTxA (fbEbTxs argsA)) } } - ebListing :: [GenTx BlockA] -> EndorserBlock - ebListing txs = - mkEndorserBlock - [(Hash.castHash (Hash.hashWith fromShort (toRawTxIdHash (txId tx))), 0) | tx <- txs] + mkForgedLeiosEb :: NonEmpty (GenTx BlockA) -> ForgedEndorserBlock + mkForgedLeiosEb = forgeLeiosEb . fmap (fromShort . toRawTxIdHash . txId) txA :: Int -> GenTx BlockA txA n = TxA (TxIdA n) InitiateAtoB diff --git a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/ChainDB.hs b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/ChainDB.hs index c98f8f6c5f..17fe19e426 100644 --- a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/ChainDB.hs +++ b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/ChainDB.hs @@ -48,7 +48,7 @@ tests = "ChainDB and LeiosDB" [ testCase "ChainSel rejects a block whose certificate a minority made" $ runSimOrThrow - (whyInvalid (\nut -> addBlocks nut [announcingA, minorityCertifyingA]) minorityCertifyingA) + (whyInvalid (\node -> addBlocks node [announcingA, minorityCertifyingA]) minorityCertifyingA) @?= Just (ExtValidationErrorLeiosCert (LeiosCertInvalid (InsufficientWeight (1 / 4)))) , testCase "ChainSel does not verify a certificate again for a claim it established" $ runSimOrThrow (whyInvalid addSibling minoritySibling) @?= Nothing @@ -73,8 +73,8 @@ tests = -- announcements and the GC has run, these are the bodies of A and B that the -- LeiosDB still holds. bodiesAfterGc :: IOSim s ([(TxHash, BytesSize)], [(TxHash, BytesSize)]) -bodiesAfterGc = withNode $ \nut -> do - let leiosDb = getLeiosDB (nutKernel nut) +bodiesAfterGc = withNode $ \node -> do + let leiosDb = getLeiosDB (nodeKernel node) withWriter leiosDb $ \writer -> forM_ [(pointA, ebA), (pointB, ebB)] $ \(point, eb) -> do await =<< writeEbPoint writer point (encodeEndorserBlockSize eb) @@ -89,7 +89,7 @@ bodiesAfterGc = withNode $ \nut -> do ] -- Let the slots of the chain's blocks pass, so none is from the future. threadDelay 20 - addBlocks nut chain + addBlocks node chain -- Long enough for the copy to the ImmutableDB and for the delayed GC. threadDelay 100 withReader leiosDb $ \reader -> @@ -107,12 +107,12 @@ whyInvalid :: (NodeUnderTest (IOSim s) -> IOSim s ()) -> LeiosTestBlock -> IOSim s (Maybe (ExtValidationError LeiosTestBlock)) -whyInvalid add blk = withNode $ \nut -> do +whyInvalid add blk = withNode $ \node -> do -- Let the slots of the blocks pass, so none is from the future. threadDelay 20 - add nut + add node threadDelay 1 - isInvalid <- atomically $ forgetFingerprint <$> ChainDB.getIsInvalidBlock (nutChainDB nut) + isInvalid <- atomically $ forgetFingerprint <$> ChainDB.getIsInvalidBlock (nodeChainDB node) pure $ isInvalid (blockHash blk) -- | The node adds 'announcingA' and 'certifyingA', restarts, and then adds @@ -139,20 +139,20 @@ heldAcrossRestart whatOfA = do [ (offset, BS.replicate (fromIntegral size) 0) | (offset, (_txHash, size)) <- zip [0 ..] (endorserBlockReferences ebA) ] - beforeRestart <- withNodeOverDBs nodeDBs leiosDb $ \nut -> do + beforeRestart <- withNodeOverDBs nodeDBs leiosDb $ \node -> do -- Let the slots of the chain's blocks pass, so none is from the future. threadDelay 20 - addBlocks nut [announcingA, certifyingA] + addBlocks node [announcingA, certifyingA] threadDelay 1 - holds nut - withNodeOverDBs nodeDBs leiosDb $ \nut -> do - afterRestart <- holds nut - addBlocksAfter nut (NotOrigin (blockSlot announcingA)) [certifyingA] + holds node + withNodeOverDBs nodeDBs leiosDb $ \node -> do + afterRestart <- holds node + addBlocksAfter node (NotOrigin (blockSlot announcingA)) [certifyingA] threadDelay 1 - afterAddingAgain <- holds nut + afterAddingAgain <- holds node pure (beforeRestart, afterRestart, afterAddingAgain) where - holds nut = atomically $ ($ blockPoint certifyingA) <$> ChainDB.getIsFetched (nutChainDB nut) + holds node = atomically $ ($ blockPoint certifyingA) <$> ChainDB.getIsFetched (nodeChainDB node) -- | What the LeiosDB holds of A data WhatOfA @@ -174,9 +174,9 @@ minoritySibling = -- | Add 'certifyingA', which establishes its claim, and then 'minoritySibling'. addSibling :: NodeUnderTest (IOSim s) -> IOSim s () -addSibling nut = do - addBlocks nut [announcingA, certifyingA] - addBlocksAfter nut (NotOrigin (blockSlot announcingA)) [minoritySibling] +addSibling node = do + addBlocks node [announcingA, certifyingA] + addBlocksAfter node (NotOrigin (blockSlot announcingA)) [minoritySibling] rbHashOf :: LeiosTestBlock -> RankingBlockHash rbHashOf = toRankingBlockHash (Proxy @LeiosTestBlock) . blockHash @@ -202,15 +202,15 @@ ebB = mkEndorserBlock [(UnsafeHash (SBS.replicate 32 2), 10)] -- body but not its transactions. The tip of the selected chain before they -- arrive, and after. tipsAroundClosure :: IOSim s (Point LeiosTestBlock, Point LeiosTestBlock) -tipsAroundClosure = withNode $ \nut -> do - let leiosDb = getLeiosDB (nutKernel nut) - chainDB = nutChainDB nut +tipsAroundClosure = withNode $ \node -> do + let leiosDb = getLeiosDB (nodeKernel node) + chainDB = nodeChainDB node withWriter leiosDb $ \writer -> do await =<< writeEbPoint writer pointA (encodeEndorserBlockSize ebA) void $ await =<< writeEbBody writer pointA ebA -- Let the slots of the chain's blocks pass, so none is from the future. threadDelay 20 - addBlocks nut [announcingA, certifyingA] + addBlocks node [announcingA, certifyingA] threadDelay 1 tipBefore <- atomically $ ChainDB.getTipPoint chainDB withWriter leiosDb $ \writer -> @@ -242,9 +242,9 @@ tipWithClosureBeforeStart = do [ (offset, BS.replicate (fromIntegral size) 0) | (offset, (_txHash, size)) <- zip [0 ..] (endorserBlockReferences ebA) ] - withNodeOver leiosDb $ \nut -> do + withNodeOver leiosDb $ \node -> do -- Let the slots of the chain's blocks pass, so none is from the future. threadDelay 20 - addBlocks nut [announcingA, certifyingA] + addBlocks node [announcingA, certifyingA] threadDelay 1 - atomically $ ChainDB.getTipPoint (nutChainDB nut) + atomically $ ChainDB.getTipPoint (nodeChainDB node) diff --git a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/Environment.hs b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/Environment.hs index 72a3c04390..b8d1a16cb4 100644 --- a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/Environment.hs +++ b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/Environment.hs @@ -160,7 +160,7 @@ withNodeVsEnvConnection :: EnvPeer (IOSim s) -> (NodeVsEnvConnection (IOSim s) -> IOSim s a) -> IOSim s a -withNodeVsEnvConnection nut addr envPeer body = do +withNodeVsEnvConnection node addr envPeer body = do nodeToPeer <- Strict.newTBQueueIO bearerQueueCapacity peerToNode <- Strict.newTBQueueIO bearerQueueCapacity let nodeBearer = queueChannelAsBearer sduSize QueueChannel{readQueue = peerToNode, writeQueue = nodeToPeer} @@ -169,7 +169,7 @@ withNodeVsEnvConnection nut addr envPeer body = do peerConnId = ConnectionId{localAddress = addr, remoteAddress = nodeAddr} (bundle, version) <- concurrently - (runHandshakeClient nodeBearer nodeConnId handshakeArguments (nutVersions nut)) + (runHandshakeClient nodeBearer nodeConnId handshakeArguments (nodeVersions node)) (runHandshakeServer peerBearer peerConnId handshakeArguments peerVersions) >>= \case (Right (HandshakeNegotiationResult bundle version _), Right HandshakeNegotiationResult{}) -> @@ -180,12 +180,12 @@ withNodeVsEnvConnection nut addr envPeer body = do publicPeerSelectionStateVar <- makePublicPeerSelectionStateVar newPeerSharingAPI publicPeerSelectionStateVar - (seedEnvPeerSharing (nutSeeds nut)) + (seedEnvPeerSharing (nodeSeeds node)) ps_POLICY_PEER_SHARE_STICKY_TIME ps_POLICY_PEER_SHARE_MAX_PEERS let protocols = getOuroborosApplication (fromOuroborosBundle bundle) infos = mkMiniProtocolInfos (\_ _ -> Nothing) Outbound Duplex bundle - codecs = envCodecs nut version + codecs = envCodecs node version nodeMux <- Mx.new Mx.nullTracers infos peerMux <- Mx.new Mx.nullTracers infos withAsync (Mx.run nodeMux nodeBearer) $ \_ -> @@ -359,23 +359,23 @@ type NodeCodecs s = ByteString envCodecs :: NodeUnderTest (IOSim s) -> NodeToNodeVersion -> NodeCodecs s -envCodecs nut = +envCodecs node = NTN.defaultCodecs - (configCodec (nutTopLevelConfig nut)) + (configCodec (nodeTopLevelConfig node)) () (const encodePeerAddr) (const decodePeerAddr) envLeiosNotifyCodec :: NodeUnderTest (IOSim s) -> LeiosNotifyCodec s -envLeiosNotifyCodec nut = NTN.cLeiosNotifyCodec (envCodecs nut maxBound) +envLeiosNotifyCodec node = NTN.cLeiosNotifyCodec (envCodecs node maxBound) envLeiosFetchCodec :: NodeUnderTest (IOSim s) -> LeiosFetchCodec s -envLeiosFetchCodec nut = NTN.cLeiosFetchCodec (envCodecs nut maxBound) +envLeiosFetchCodec node = NTN.cLeiosFetchCodec (envCodecs node maxBound) -- | A LeiosFetch server that disconnects a client that sends any request. refusingLeiosFetchServer :: NodeUnderTest (IOSim s) -> Channel (IOSim s) ByteString -> IOSim s () -refusingLeiosFetchServer nut chan = - fst <$> runAnnotatedPeer nullTracer (envLeiosFetchCodec nut) chan (leiosFetchServerPeer refuse) +refusingLeiosFetchServer node chan = + fst <$> runAnnotatedPeer nullTracer (envLeiosFetchCodec node) chan (leiosFetchServerPeer refuse) where refuse = MkLeiosFetchRequestHandler $ \_request -> throwIO $ userError "the environment's LeiosFetch server serves nothing" diff --git a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/Forge.hs b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/Forge.hs new file mode 100644 index 0000000000..4d844f4082 --- /dev/null +++ b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/Forge.hs @@ -0,0 +1,129 @@ +{-# LANGUAGE LambdaCase #-} +{-# LANGUAGE NamedFieldPuns #-} +{-# LANGUAGE OverloadedStrings #-} + +-- | Tests that 'forkBlockForging' writes each endorser block that it forges to +-- the LeiosDB. +module Test.Consensus.Leios.Forge (tests) where + +import Control.Monad.Class.MonadTimer.SI (threadDelay) +import Control.Monad.IOSim (IOSim, runSimOrThrow) +import Data.ByteString (ByteString) +import qualified Data.ByteString.Char8 as BS8 +import Data.List (sort) +import Data.List.NonEmpty (NonEmpty (..)) +import Ouroboros.Consensus.Block +import Ouroboros.Consensus.Ledger.Abstract (GetTip (..)) +import Ouroboros.Consensus.Leios.Types +import Ouroboros.Consensus.NodeKernel (NodeKernel (..)) +import Ouroboros.Consensus.Storage.ChainDB.API (getCurrentChain) +import Ouroboros.Consensus.Storage.LeiosDB.API (LeiosDbReader (..), withReader) +import Ouroboros.Consensus.Util.IOLike (atomically) +import qualified Ouroboros.Network.AnchoredFragment as AF +import Test.Consensus.Leios.NodeUnderTest +import Test.Consensus.Leios.TestBlock +import Test.Tasty +import Test.Tasty.HUnit +import Test.Util.TestBlock (TestHash (..)) + +tests :: TestTree +tests = + testGroup + "Forging" + [ testCase + "the node stores each endorser block that it forges, at the point that the header announces" + unit_storesForgedEbs + ] + +-- | The node runs 'forkBlockForging' with 'ebForging' for slots 0 to 3. Then +-- the test reads the chain and the LeiosDB. +-- +-- * If the node forges no block, the other checks pass with no data. The +-- first check catches this. +-- * If 'forkBlockForging' does not give 'storeForgedEb' to 'forge', or if +-- 'storeForgedEb' writes at a slot or a hash that differs from the +-- announcement in the header, the second check fails. +-- * If 'storeForgedEb' does not write the transaction bytes, the third check +-- fails. +unit_storesForgedEbs :: Assertion +unit_storesForgedEbs = do + let ForgeOutcome{announcedEbs, storedPoints} = runSimOrThrow forgedEbs + assertBool "the node forged no block" (not (null announcedEbs)) + sort storedPoints @?= sort [(announcedSlot eb, announcedHash eb) | eb <- announcedEbs] + map storedTxs announcedEbs @?= map (Just . forgedTxs) announcedEbs + +-- | What the chain and the LeiosDB hold after the node forges. +data ForgeOutcome = ForgeOutcome + { announcedEbs :: [AnnouncedEb] + -- ^ The endorser blocks that the blocks on the chain announce, oldest first. + , storedPoints :: [(SlotNo, EndorserBlockHash)] + -- ^ The points in the LeiosDB. + } + +-- | An endorser block that a block on the chain announces. +data AnnouncedEb = AnnouncedEb + { announcedSlot :: SlotNo + , announcedHash :: EndorserBlockHash + , forgedTxs :: [ByteString] + -- ^ The bytes of the transactions that 'ebForging' puts in the endorser block. + , storedTxs :: Maybe [ByteString] + -- ^ The bytes of the transactions that the LeiosDB holds for 'announcedHash', + -- or 'Nothing' if it does not hold all of them. + } + +-- | The node forges in each of the first slots. +forgedEbs :: IOSim s ForgeOutcome +forgedEbs = withNode $ \node -> do + setBlockForging (nodeKernel node) [MkBlockForging (pure ebForging)] + -- A slot is one second, so the node forges in slots 0 to 3. The half slot + -- lets the forge of slot 3 add its block to the ChainDB. + threadDelay 3.5 + chain <- atomically $ getCurrentChain (nodeChainDB node) + withReader (getLeiosDB (nodeKernel node)) $ \reader -> do + announcedEbs <- + sequence + [ AnnouncedEb (blockSlot hdr) ebHash [txInSlot (blockSlot hdr)] + . fmap (map snd) + <$> lookupEbClosure reader ebHash + | hdr <- AF.toOldestFirst chain + , Just (ebHash, _size) <- [lthAnnouncement hdr] + ] + storedPoints <- testScanEbPoints reader + pure ForgeOutcome{announcedEbs, storedPoints} + +-- | A forging that leads in every slot. Each block extends the ledger tip and +-- announces an endorser block with one transaction, whose bytes depend on the +-- slot, so that each endorser block has its own hash. +ebForging :: BlockForging (IOSim s) LeiosTestBlock +ebForging = + BlockForging + { forgeLabel = "ebForging" + , canBeLeader = () + , updateForgeState = \_ _ _ -> pure $ ForgeStateUpdated () + , checkCanForge = \_ _ _ _ _ -> Right () + , forgeBlock = \ForgeBlockArgs{fbCurrentSlotNo, fbCurrentTickedLedgerState} -> do + let forgedEb = forgeLeiosEb (txInSlot fbCurrentSlotNo :| []) + eb = forgedEbBody forgedEb + ebHash = endorserBlockHash eb + tip = castPoint (getTip fbCurrentTickedLedgerState) + pure + ForgeBlockResult + { rankingBlock = + setAnnouncement ebHash (encodeEndorserBlockSize eb) $ + setSlot fbCurrentSlotNo $ + blockAfter tip + , endorserBlock = Just forgedEb + } + , finalize = pure () + } + +-- | A block whose predecessor is the given point. 'forgeBlock' sets its slot. +blockAfter :: Point LeiosTestBlock -> LeiosTestBlock +blockAfter = \case + GenesisPoint -> firstLeiosBlock 0 + BlockPoint _ (TestHash path) -> + successorLeiosBlock $ + LeiosTestBlock (ltbHeader (firstLeiosBlock 0)){lthHash = TestHash path} + +txInSlot :: SlotNo -> ByteString +txInSlot slot = BS8.pack ("tx in slot " <> show (unSlotNo slot)) diff --git a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/LeiosFetch/Node.hs b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/LeiosFetch/Node.hs index 2c73646b77..ff5918dde7 100644 --- a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/LeiosFetch/Node.hs +++ b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/LeiosFetch/Node.hs @@ -101,23 +101,23 @@ type EnvClient s = -- | Run the node's server, with 'heldEb' in its LeiosDB, against a peer that -- runs this client. nodeServerAgainst :: EnvClient s -> IOSim s ServerOutcome -nodeServerAgainst client = withNode $ \nut -> do - holdHeldEb nut +nodeServerAgainst client = withNode $ \node -> do + holdHeldEb node let envPeer = EnvPeer { epLeiosNotifyServer = \_chan -> atomically retry , epLeiosNotifyClient = Nothing - , epLeiosFetchServer = refusingLeiosFetchServer nut - , epLeiosFetchClient = Just $ runEnvClient nut client + , epLeiosFetchServer = refusingLeiosFetchServer node + , epLeiosFetchClient = Just $ runEnvClient node client } - withNodeVsEnvConnection nut (PeerAddr 1) envPeer $ \conn -> do + withNodeVsEnvConnection node (PeerAddr 1) envPeer $ \conn -> do server <- nodeLeiosFetchServer conn atomically server >>= \case Right () -> pure ServerEnded Left e -> pure $ ServerFailed (show e) runEnvClient :: NodeUnderTest (IOSim s) -> EnvClient s -> Channel (IOSim s) ByteString -> IOSim s () -runEnvClient nut client chan = fst <$> runAnnotatedPeer nullTracer (envLeiosFetchCodec nut) chan client +runEnvClient node client chan = fst <$> runAnnotatedPeer nullTracer (envLeiosFetchCodec node) chan client doneClient :: EnvClient s doneClient = Yield ReflClientAgency MsgDone $ Done ReflNobodyAgency () @@ -187,7 +187,7 @@ heldTx = MkLeiosTx . heldTxBytes -- | Write 'heldEb' and all its transactions but the last. holdHeldEb :: NodeUnderTest (IOSim s) -> IOSim s () -holdHeldEb nut = withWriter (getLeiosDB (nutKernel nut)) $ \writer -> do +holdHeldEb node = withWriter (getLeiosDB (nodeKernel node)) $ \writer -> do await =<< writeEbPoint writer heldPoint (encodeEndorserBlockSize heldEb) _ <- await =<< writeEbBody writer heldPoint heldEb _ <- await =<< writeTxs writer heldPoint [(i, heldTxBytes i) | i <- [0 .. 68]] @@ -200,17 +200,17 @@ holdHeldEb nut = withWriter (getLeiosDB (nutKernel nut)) $ \writer -> do -- | Whether the node's initiator bundle, and its initiator-and-responder bundle, -- include LeiosFetch among their hot mini-protocols for this version. nodeRunsLeiosFetch :: NodeToNodeVersion -> IOSim s (Bool, Bool) -nodeRunsLeiosFetch ntnVersion = withNode $ \nut -> +nodeRunsLeiosFetch ntnVersion = withNode $ \node -> pure ( runsLeiosFetch $ - NTN.initiator mempty defaultMiniProtocolParameters ntnVersion versionData (nutApps nut ()) + NTN.initiator mempty defaultMiniProtocolParameters ntnVersion versionData (nodeApps node ()) , runsLeiosFetch $ NTN.initiatorAndResponder mempty defaultMiniProtocolParameters ntnVersion versionData - (nutApps nut ()) + (nodeApps node ()) ) where versionData = nodeVersionData ntnVersion diff --git a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/LeiosNotify/Node.hs b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/LeiosNotify/Node.hs index d27ec5e6a5..22b9904b01 100644 --- a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/LeiosNotify/Node.hs +++ b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/LeiosNotify/Node.hs @@ -376,16 +376,16 @@ runNodeClientAgainst :: STM (IOSim s) (Message (LeiosNotify LeiosPoint (Header Blk) LeiosVote) StBusy StIdle) -> Int -> IOSim s NodeClientOutcome -runNodeClientAgainst nut next target = do +runNodeClientAgainst node next target = do requests <- newTVarIO 0 let envPeer = EnvPeer - { epLeiosNotifyServer = runReplyingServer (envLeiosNotifyCodec nut) requests next + { epLeiosNotifyServer = runReplyingServer (envLeiosNotifyCodec node) requests next , epLeiosNotifyClient = Nothing - , epLeiosFetchServer = refusingLeiosFetchServer nut + , epLeiosFetchServer = refusingLeiosFetchServer node , epLeiosFetchClient = Nothing } - withNodeVsEnvConnection nut (PeerAddr 1) envPeer $ \conn -> do + withNodeVsEnvConnection node (PeerAddr 1) envPeer $ \conn -> do client <- nodeLeiosNotifyClient conn early <- atomically $ @@ -404,24 +404,24 @@ runNodeClientAgainst nut next target = do -- | A peer that never has a notification: the node keeps exactly its -- pipelining bound of requests outstanding. nodeClientKeepsPipelineFull :: IOSim s NodeClientOutcome -nodeClientKeepsPipelineFull = withNode $ \nut -> - runNodeClientAgainst nut retry leiosNotifyPipelineDepth +nodeClientKeepsPipelineFull = withNode $ \node -> + runNodeClientAgainst node retry leiosNotifyPipelineDepth -- | Six replies, announcements from six elections, free six slots in the -- pipeline, which the node fills again. nodeClientRefillsPipeline :: IOSim s NodeClientOutcome -nodeClientRefillsPipeline = withNode $ \nut -> do +nodeClientRefillsPipeline = withNode $ \node -> do queue <- newTQueueIO atomically $ mapM_ (writeTQueue queue) $ [ MsgLeiosBlockAnnouncement $ ltbHeader $ setIssuer issuer $ ann 1 $ firstLeiosBlock 0 | issuer <- [0 .. 5] ] - runNodeClientAgainst nut (readTQueue queue) (leiosNotifyPipelineDepth + 6) + runNodeClientAgainst node (readTQueue queue) (leiosNotifyPipelineDepth + 6) nodeClientUnpromptedCancel :: IOSim s NodeClientOutcome -nodeClientUnpromptedCancel = withNode $ \nut -> - runNodeClientAgainst nut (pure MsgCanceled) maxBound +nodeClientUnpromptedCancel = withNode $ \node -> + runNodeClientAgainst node (pure MsgCanceled) maxBound -- | Announce an endorser block whose hash repeats the given byte 32 times. ann :: Word8 -> LeiosTestBlock -> LeiosTestBlock @@ -429,33 +429,33 @@ ann n = setAnnouncement (UnsafeHash (SBS.replicate 32 n)) 1000 -- | A peer that relays these headers, one per request. nodeClientAgainstAnnouncements :: [LeiosTestBlock] -> IOSim s NodeClientOutcome -nodeClientAgainstAnnouncements blks = withNode $ \nut -> do +nodeClientAgainstAnnouncements blks = withNode $ \node -> do queue <- newTQueueIO atomically $ mapM_ (writeTQueue queue . MsgLeiosBlockAnnouncement . ltbHeader) blks - runNodeClientAgainst nut (readTQueue queue) (leiosNotifyPipelineDepth + length blks) + runNodeClientAgainst node (readTQueue queue) (leiosNotifyPipelineDepth + length blks) -- | The announcement arrives long after its slot: the peer connects only then. nodeClientTooOld :: IOSim s NodeClientOutcome -nodeClientTooOld = withNode $ \nut -> do +nodeClientTooOld = withNode $ \node -> do threadDelay (realToFrac maxAnnouncementAgeRecv + 100) queue <- newTQueueIO atomically $ writeTQueue queue $ MsgLeiosBlockAnnouncement $ ltbHeader $ ann 1 (firstLeiosBlock 0) - runNodeClientAgainst nut (readTQueue queue) (leiosNotifyPipelineDepth + 1) + runNodeClientAgainst node (readTQueue queue) (leiosNotifyPipelineDepth + 1) -- | By the time the node connects, the current slot is past what its -- immutable tip, still at genesis, can forecast to. nodeClientGated :: IOSim s Int -nodeClientGated = withNodeForecastingOnly 5 $ \nut -> do +nodeClientGated = withNodeForecastingOnly 5 $ \node -> do threadDelay 10 requests <- newTVarIO 0 let envPeer = EnvPeer - { epLeiosNotifyServer = runReplyingServer (envLeiosNotifyCodec nut) requests retry + { epLeiosNotifyServer = runReplyingServer (envLeiosNotifyCodec node) requests retry , epLeiosNotifyClient = Nothing - , epLeiosFetchServer = refusingLeiosFetchServer nut + , epLeiosFetchServer = refusingLeiosFetchServer node , epLeiosFetchClient = Nothing } - withNodeVsEnvConnection nut (PeerAddr 1) envPeer $ \conn -> do + withNodeVsEnvConnection node (PeerAddr 1) envPeer $ \conn -> do threadDelay 10 n <- readTVarIO requests connTerminate conn @@ -465,14 +465,14 @@ nodeClientGated = withNodeForecastingOnly 5 $ \nut -> do -- | How many elections, body offers, closure offers and votes the client's state -- holds once one of each is in, and once the immutable tip has passed them. nodePrunesClientState :: IOSim s ((Int, Int, Int, Int), (Int, Int, Int, Int)) -nodePrunesClientState = withNodeCommittee committee 100000 $ \nut -> do +nodePrunesClientState = withNodeCommittee committee 100000 $ \node -> do requests <- newTVarIO 0 queue <- newTQueueIO let envPeer = EnvPeer - { epLeiosNotifyServer = runReplyingServer (envLeiosNotifyCodec nut) requests (readTQueue queue) + { epLeiosNotifyServer = runReplyingServer (envLeiosNotifyCodec node) requests (readTQueue queue) , epLeiosNotifyClient = Nothing - , epLeiosFetchServer = refusingLeiosFetchServer nut + , epLeiosFetchServer = refusingLeiosFetchServer node , epLeiosFetchClient = Nothing } withAsync @@ -486,18 +486,18 @@ nodePrunesClientState = withNodeCommittee committee 100000 $ \nut -> do ] ) $ \_feeder -> - withNodeVsEnvConnection nut (PeerAddr 1) envPeer $ \_conn -> do + withNodeVsEnvConnection node (PeerAddr 1) envPeer $ \_conn -> do atomically $ readTVar requests >>= check . (> leiosNotifyPipelineDepth + 3) - beforePruning <- clientStateSize nut - advanceImmutableTip nut - afterPruning <- clientStateSize nut + beforePruning <- clientStateSize node + advanceImmutableTip node + afterPruning <- clientStateSize node pure (beforePruning, afterPruning) -- | How many elections, body offers, closure offers and votes the node's -- clients remember. clientStateSize :: NodeUnderTest (IOSim s) -> IOSim s (Int, Int, Int, Int) -clientStateSize nut = atomically $ do - states <- IOLike.readTVar (getLeiosNotifyClientStates (nutKernel nut)) +clientStateSize node = atomically $ do + states <- IOLike.readTVar (getLeiosNotifyClientStates (nodeKernel node)) sizes <- mapM (fmap size . IOLike.readTVar) (Map.elems states) pure ( sum [e | (e, _, _, _) <- sizes] @@ -531,11 +531,11 @@ feedAt queue = go 0 nodeClientAgainstSchedule :: [(DiffTime, Message (LeiosNotify LeiosPoint (Header Blk) LeiosVote) StBusy StIdle)] -> IOSim s NodeClientOutcome -nodeClientAgainstSchedule schedule = withNodeCommittee committee 100000 $ \nut -> do +nodeClientAgainstSchedule schedule = withNodeCommittee committee 100000 $ \node -> do queue <- newTQueueIO let msgs = (0, someAnnouncement) : schedule withAsync (feedAt queue msgs) $ \_feeder -> - runNodeClientAgainst nut (readTQueue queue) (leiosNotifyPipelineDepth + length msgs) + runNodeClientAgainst node (readTQueue queue) (leiosNotifyPipelineDepth + length msgs) -- | A vote from this seat for the announcement this block carries, signed with -- the seat's key. @@ -562,9 +562,9 @@ equivocation = ann 2 (forkLeiosBlock (firstLeiosBlock 0)) -- | Make the first two slots immutable: k is 10, so twelve blocks do it. advanceImmutableTip :: NodeUnderTest (IOSim s) -> IOSim s () -advanceImmutableTip nut = do +advanceImmutableTip node = do threadDelay 20 - addBlocks nut $ + addBlocks node $ take 12 $ iterate successorLeiosBlock (firstLeiosBlock 1) threadDelay 1 @@ -573,27 +573,27 @@ advanceImmutableTip nut = do -- the immutable tip has passed it. nodeClientOfferTooOld :: Message (LeiosNotify LeiosPoint (Header Blk) LeiosVote) StBusy StIdle -> IOSim s NodeClientOutcome -nodeClientOfferTooOld offer = withNode $ \nut -> do +nodeClientOfferTooOld offer = withNode $ \node -> do queue <- newTQueueIO atomically $ writeTQueue queue someAnnouncement announced <- newEmptyTMVarIO let feed = do atomically $ readTMVar announced - advanceImmutableTip nut + advanceImmutableTip node atomically $ writeTQueue queue offer next = do msg <- readTQueue queue _ <- tryPutTMVar announced () pure msg - withAsync feed $ \_feeder -> runNodeClientAgainst nut next maxBound + withAsync feed $ \_feeder -> runNodeClientAgainst node next maxBound -- | A peer that announces 'someOfferedPoint', then sends these. nodeClientAgainstOffers :: [Message (LeiosNotify LeiosPoint (Header Blk) LeiosVote) StBusy StIdle] -> IOSim s NodeClientOutcome -nodeClientAgainstOffers msgs = withNode $ \nut -> do +nodeClientAgainstOffers msgs = withNode $ \node -> do queue <- newTQueueIO atomically $ mapM_ (writeTQueue queue) (someAnnouncement : msgs) - runNodeClientAgainst nut (readTQueue queue) (leiosNotifyPipelineDepth + 1 + length msgs) + runNodeClientAgainst node (readTQueue queue) (leiosNotifyPipelineDepth + 1 + length msgs) someAnnouncement :: Message (LeiosNotify LeiosPoint (Header Blk) LeiosVote) StBusy StIdle someAnnouncement = MsgLeiosBlockAnnouncement $ ltbHeader $ ann 1 (firstLeiosBlock 0) @@ -608,19 +608,19 @@ newtype NodeServerOutcome = NodeServerOutcome Int -- | The node never has a notification, so the peer's requests are all -- outstanding when it quits. nodeServerCancels :: IOSim s NodeServerOutcome -nodeServerCancels = withNode $ \nut -> do +nodeServerCancels = withNode $ \node -> do requests <- newTVarIO 0 replies <- newTVarIO 0 stop <- newEmptyTMVarIO - let codec' = envLeiosNotifyCodec nut + let codec' = envLeiosNotifyCodec node envPeer = EnvPeer { epLeiosNotifyServer = runReplyingServer codec' requests retry , epLeiosNotifyClient = Just $ runPipelinedClient codec' replies 3 stop - , epLeiosFetchServer = refusingLeiosFetchServer nut + , epLeiosFetchServer = refusingLeiosFetchServer node , epLeiosFetchClient = Nothing } - withNodeVsEnvConnection nut (PeerAddr 1) envPeer $ \conn -> do + withNodeVsEnvConnection node (PeerAddr 1) envPeer $ \conn -> do threadDelay 10 atomically $ putTMVar stop () for_ (connPeerLeiosNotifyClient conn) $ \client -> atomically client >>= either throwIO pure @@ -631,20 +631,20 @@ nodeServerCancels = withNode $ \nut -> do -- | A peer pipelining one request more than the node's server allows, so the -- node disconnects it. nodeServerExcessive :: IOSim s ExcessiveOutcome -nodeServerExcessive = withNode $ \nut -> do +nodeServerExcessive = withNode $ \node -> do requests <- newTVarIO 0 replies <- newTVarIO 0 stop <- newEmptyTMVarIO - let codec' = envLeiosNotifyCodec nut + let codec' = envLeiosNotifyCodec node envPeer = EnvPeer { epLeiosNotifyServer = runReplyingServer codec' requests retry , epLeiosNotifyClient = Just $ runPipelinedClient codec' replies (leiosNotifyPipelineDepth + 1) stop - , epLeiosFetchServer = refusingLeiosFetchServer nut + , epLeiosFetchServer = refusingLeiosFetchServer node , epLeiosFetchClient = Nothing } - withNodeVsEnvConnection nut (PeerAddr 1) envPeer $ \conn -> do + withNodeVsEnvConnection node (PeerAddr 1) envPeer $ \conn -> do server <- nodeLeiosNotifyServer conn atomically server >>= \case Left e | Just ExnLeiosNotifyExcessiveRequests <- fromException e -> pure $ ExcessiveOutcome Disconnected @@ -653,17 +653,17 @@ nodeServerExcessive = withNode $ \nut -> do -- | Whether the node's initiator bundle, and its initiator-and-responder bundle, -- include LeiosNotify among their hot mini-protocols for this version. nodeRunsLeiosNotify :: NodeToNodeVersion -> IOSim s (Bool, Bool) -nodeRunsLeiosNotify ntnVersion = withNode $ \nut -> +nodeRunsLeiosNotify ntnVersion = withNode $ \node -> pure ( runsLeiosNotify $ - NTN.initiator mempty defaultMiniProtocolParameters ntnVersion versionData (nutApps nut ()) + NTN.initiator mempty defaultMiniProtocolParameters ntnVersion versionData (nodeApps node ()) , runsLeiosNotify $ NTN.initiatorAndResponder mempty defaultMiniProtocolParameters ntnVersion versionData - (nutApps nut ()) + (nodeApps node ()) ) where versionData = nodeVersionData ntnVersion diff --git a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/NodeUnderTest.hs b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/NodeUnderTest.hs index 8fbde141d9..210d3dea79 100644 --- a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/NodeUnderTest.hs +++ b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/NodeUnderTest.hs @@ -5,13 +5,12 @@ {-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE TypeApplications #-} --- | One real node, the node under test (NUT), and a mocked environment around --- it. The @nut@ prefix of names here abbreviates it. +-- | One real node, the node under test, and a mocked environment around it. -- -- The node is built by the node's own 'initNodeKernel' over a ChainDB on a -- mocked filesystem, so the code under test is the node's, not a --- reimplementation of it. Its block is 'LeiosTestBlock'. It does not forge, --- since the test gives it no block forging. +-- reimplementation of it. Its block is 'LeiosTestBlock'. It forges only if the +-- test calls 'setBlockForging'. -- -- The node's node-to-node applications and the versions it offers them at are -- built by the node's own 'mkNodeToNodeApps' and @@ -149,18 +148,18 @@ type Blk = LeiosTestBlock -- | What a test can vary about the node. data NodeUnderTestConfig = NodeUnderTestConfig - { nutcSecurityParam :: SecurityParam - , nutcLedgerConfig :: LeiosTestLedgerConfig - , nutcRandomness :: StdGen + { nodeConfigSecurityParam :: SecurityParam + , nodeConfigLedger :: LeiosTestLedgerConfig + , nodeConfigRandomness :: StdGen -- ^ The root of all the test's randomness; see 'Seeds'. } defaultNodeUnderTestConfig :: SecurityParam -> LeiosTestLedgerConfig -> NodeUnderTestConfig -defaultNodeUnderTestConfig nutcSecurityParam nutcLedgerConfig = +defaultNodeUnderTestConfig nodeConfigSecurityParam nodeConfigLedger = NodeUnderTestConfig - { nutcSecurityParam - , nutcLedgerConfig - , nutcRandomness = mkStdGen 0 + { nodeConfigSecurityParam + , nodeConfigLedger + , nodeConfigRandomness = mkStdGen 0 } -- | All the randomness of a Leios node-vs-environment test. @@ -182,8 +181,8 @@ data Seeds a = Seeds splitSeeds :: StdGen -> Seeds StdGen splitSeeds = evalState (mapM (\() -> state splitGen) (Seeds () () () () () ())) -nodeSeeds :: NodeUnderTestConfig -> Seeds StdGen -nodeSeeds = splitSeeds . nutcRandomness +configSeeds :: NodeUnderTestConfig -> Seeds StdGen +configSeeds = splitSeeds . nodeConfigRandomness -- | The peer address these tests use: a peer is just a number. newtype PeerAddr = PeerAddr Int @@ -193,33 +192,33 @@ newtype PeerAddr = PeerAddr Int -- | A running node, and the handles a test needs on it for its gray-box -- predicates. data NodeUnderTest m = NodeUnderTest - { nutChainDB :: ChainDB m Blk - , nutTopLevelConfig :: TopLevelConfig Blk - , nutKernel :: NodeKernel m PeerAddr (ConnectionId ()) Blk - , nutApps :: BlockNodeToNodeVersion Blk -> NodeApps m - , nutVersions :: NodeVersions m - , nutSeeds :: Seeds StdGen + { nodeChainDB :: ChainDB m Blk + , nodeTopLevelConfig :: TopLevelConfig Blk + , nodeKernel :: NodeKernel m PeerAddr (ConnectionId ()) Blk + , nodeApps :: BlockNodeToNodeVersion Blk -> NodeApps m + , nodeVersions :: NodeVersions m + , nodeSeeds :: Seeds StdGen } -- | Add a chain to the node's ChainDB, oldest first, starting at genesis addBlocks :: IOLike m => NodeUnderTest m -> [Blk] -> m () -addBlocks nut = addBlocksAfter nut Origin +addBlocks node = addBlocksAfter node Origin -- | Add a chain to the node's ChainDB, oldest first, given the slot of its -- first block's predecessor -- -- A 'LeiosTestBlock''s ledger view is the same at every slot. addBlocksAfter :: IOLike m => NodeUnderTest m -> WithOrigin SlotNo -> [Blk] -> m () -addBlocksAfter nut predecessorSlot blks = +addBlocksAfter node predecessorSlot blks = sequence_ $ zipWith ( \predSlot -> - ChainDB.addBlock_ (nutChainDB nut) noPunishment predSlot leiosView + ChainDB.addBlock_ (nodeChainDB node) noPunishment predSlot leiosView ) (predecessorSlot : map (NotOrigin . blockSlot) blks) blks where - leiosView = SJust (ltlcLeios (configLedger (nutTopLevelConfig nut))) + leiosView = SJust (ltlcLeios (configLedger (nodeTopLevelConfig node))) -- | The node's node-to-node applications, at every mini-protocol's bytes. type NodeApps m = @@ -269,7 +268,7 @@ withNodeUnderTest :: withNodeUnderTest cfg tracers nodeDBs leiosDB body = do -- What a node with Genesis disabled, the default, runs with. (genesisArgs, updateChainDbArgs) <- mkGenesisNodeKernelArgs disableGenesisConfig - let seeds = nodeSeeds cfg + let seeds = configSeeds cfg -- The ChainDB and the node each get their own resource registry. The node's -- registry is nested inside the ChainDB's, and the body runs inside both. So -- on the way out, the body finishes first, then the node's threads stop, and @@ -280,7 +279,7 @@ withNodeUnderTest cfg tracers nodeDBs leiosDB body = do <$> openChainDB dbRegistry (topLevelConfigOf cfg) - (leiosTestInitExtLedger (nutcLedgerConfig cfg)) + (leiosTestInitExtLedger (nodeConfigLedger cfg)) (mountedIn (nodeDBsImm nodeDBs)) (mountedIn (nodeDBsVol nodeDBs)) (seedSnapshotDelay seeds) @@ -345,18 +344,18 @@ withNodeUnderTest cfg tracers nodeDBs leiosDB body = do (const decodePeerAddr) body NodeUnderTest - { nutChainDB = chainDB - , nutTopLevelConfig = topLevelConfigOf cfg - , nutKernel = kernel - , nutApps = apps - , nutVersions = + { nodeChainDB = chainDB + , nodeTopLevelConfig = topLevelConfigOf cfg + , nodeKernel = kernel + , nodeApps = apps + , nodeVersions = mkNodeToNodeInitiatorResponderVersions (featureFlags kernelArgs) (miniProtocolParameters kernelArgs) nodeVersionData (supportedNodeToNodeVersions (Proxy @Blk)) apps - , nutSeeds = seeds + , nodeSeeds = seeds } -- | A node whose ledger view forecasts further ahead than any test runs. @@ -428,7 +427,7 @@ forecastingOnly committee quorumStakeThreshold range = k = SecurityParam $ knownNonZeroBounded @10 -- | The largest endorser block, and total size of the transactions it --- references, that the NUT's ledger admits. +-- references, that the node's ledger admits. maxEbBodySize :: BytesSize maxEbBodySize = 5000 @@ -467,8 +466,8 @@ mkBlockchainTime cfg registry chainDB = } topLevelConfigOf :: NodeUnderTestConfig -> TopLevelConfig Blk -topLevelConfigOf NodeUnderTestConfig{nutcSecurityParam, nutcLedgerConfig} = - singleNodeLeiosTestConfig nutcLedgerConfig nutcSecurityParam +topLevelConfigOf NodeUnderTestConfig{nodeConfigSecurityParam, nodeConfigLedger} = + singleNodeLeiosTestConfig nodeConfigLedger nodeConfigSecurityParam -- | A node's database directory, mounting each database in its own -- subdirectory as 'stdMkChainDbHasFS' does, but on a mock file system. diff --git a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/Relay.hs b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/Relay.hs index 33fc1ed9d5..bbb618f57f 100644 --- a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/Relay.hs +++ b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/Relay.hs @@ -592,8 +592,8 @@ relayPeers end ups downs = do -- | 'relayPeers', with a node that takes this long to verify each vote, and -- without failing for what the node relays. runRelay :: VerifyingTime -> DiffTime -> [PeerSpec] -> [DiffTime] -> ([PeerOutcome], [PeerOutcome]) -runRelay verifying end ups downs = splitAt (length ups) $ runSimOrThrow $ withNode verifying $ \nut -> - forConcurrently (zip [1 ..] specs) $ \(i, spec) -> runPeer nut end (PeerAddr i) spec +runRelay verifying end ups downs = splitAt (length ups) $ runSimOrThrow $ withNode verifying $ \node -> + forConcurrently (zip [1 ..] specs) $ \(i, spec) -> runPeer node end (PeerAddr i) spec where specs = ups <> [PeerSpec at [] | at <- downs] @@ -606,20 +606,20 @@ relayToOne at ups = _ -> error "relayToOne: one downstream peer" runPeer :: NodeUnderTest (IOSim s) -> DiffTime -> PeerAddr -> PeerSpec -> IOSim s PeerOutcome -runPeer nut end addr PeerSpec{psConnectAt, psSends} = do +runPeer node end addr PeerSpec{psConnectAt, psSends} = do threadDelay psConnectAt queue <- newTQueueIO received <- newTVarIO [] - let c = envLeiosNotifyCodec nut + let c = envLeiosNotifyCodec node envPeer = EnvPeer { epLeiosNotifyServer = runFeedingServer c (readTQueue queue) , epLeiosNotifyClient = Just $ runRecordingClient c received - , epLeiosFetchServer = refusingLeiosFetchServer nut + , epLeiosFetchServer = refusingLeiosFetchServer node , epLeiosFetchClient = Nothing } withAsync (feedAt queue psSends) $ \_feeder -> - withNodeVsEnvConnection nut addr envPeer $ \conn -> do + withNodeVsEnvConnection node addr envPeer $ \conn -> do Time now <- getMonotonicTime threadDelay (end - now) ended <- case Map.lookup leiosNotifyMiniProtocolNum (connNodeInitiators conn) of diff --git a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/TestBlock.hs b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/TestBlock.hs index 4fd7fe2535..16328dca61 100644 --- a/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/TestBlock.hs +++ b/ouroboros-consensus-diffusion/test/consensus-test/Test/Consensus/Leios/TestBlock.hs @@ -788,8 +788,8 @@ deriving via {------------------------------------------------------------------------------- Mempool - 'RunNode' demands a mempool even of a node that never forges. This one - rejects every transaction. + 'RunNode' demands a mempool. This one rejects every transaction, so a forged + block holds none. -------------------------------------------------------------------------------} data instance GenTx LeiosTestBlock = LeiosTestGenTx @@ -942,8 +942,8 @@ instance SerialiseNodeToClientConstraints LeiosTestBlock instance RunNode LeiosTestBlock --- | The node under test has no credentials, so it never forges; these exist --- only because 'RunNode' asks for them. +-- | A test's block forging has no credentials, so nothing can stop it from +-- forging. These exist only because 'RunNode' asks for them. type instance CannotForge LeiosTestBlock = Void type instance ForgeStateInfo LeiosTestBlock = () diff --git a/ouroboros-consensus.cabal b/ouroboros-consensus.cabal index 4c11c8616d..ee4d9416f5 100644 --- a/ouroboros-consensus.cabal +++ b/ouroboros-consensus.cabal @@ -1522,6 +1522,7 @@ test-suite consensus-diffusion-test Test.Consensus.Leios.ChainDB Test.Consensus.Leios.Committee Test.Consensus.Leios.Environment + Test.Consensus.Leios.Forge Test.Consensus.Leios.LeiosFetch Test.Consensus.Leios.LeiosFetch.Codec Test.Consensus.Leios.LeiosFetch.Node @@ -1913,7 +1914,6 @@ test-suite shelley-test cardano-ledger-alonzo:{cardano-ledger-alonzo, testlib}, cardano-ledger-api, cardano-ledger-babbage:testlib, - cardano-ledger-binary, cardano-ledger-conway:testlib, cardano-ledger-core, cardano-ledger-dijkstra:{cardano-ledger-dijkstra, testlib}, diff --git a/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Block/Forging.hs b/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Block/Forging.hs index 3608e7ce18..4b73ded147 100644 --- a/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Block/Forging.hs +++ b/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Block/Forging.hs @@ -40,7 +40,7 @@ import Ouroboros.Consensus.Block.SupportsPeras (PerasCert) import Ouroboros.Consensus.Config import Ouroboros.Consensus.Ledger.Abstract import Ouroboros.Consensus.Ledger.SupportsMempool -import Ouroboros.Consensus.Leios.Types (EndorserBlock) +import Ouroboros.Consensus.Leios.Types (ForgedEndorserBlock) import Ouroboros.Consensus.Protocol.Abstract import Ouroboros.Consensus.Ticked @@ -289,7 +289,7 @@ data ForgeBlockArgs blk = ForgeBlockArgs data ForgeBlockResult blk = ForgeBlockResult { rankingBlock :: !blk -- ^ The forged block. - , endorserBlock :: !(Maybe EndorserBlock) + , endorserBlock :: !(Maybe ForgedEndorserBlock) -- ^ The endorser block that 'rankingBlock' announces. 'forgeBlock' builds -- it from 'fbEbTxs'. A @blk@ that does not support Leios returns 'Nothing'. } diff --git a/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Leios/NotifyLog.hs b/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Leios/NotifyLog.hs index 3f84a1a6fe..b8b6d3bf8c 100644 --- a/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Leios/NotifyLog.hs +++ b/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Leios/NotifyLog.hs @@ -72,7 +72,9 @@ import Ouroboros.Consensus.Util.Time (nominalDelay) data Notification blk = NotifyAnnouncement !(AnnouncingHeader blk) | NotifyVote !LeiosVote - -- TODO: add offers + -- TODO: add offers. 'storeForgedEb' does not wait for the writes, + -- so offer a self-forged EB only after the LeiosDB sends 'AcquiredEb', and + -- its txs only after 'AcquiredEbTxs'. deriving stock Generic deriving anyclass instance NoThunks (Header blk) => NoThunks (Notification blk) diff --git a/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Leios/Types.hs b/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Leios/Types.hs index 9563670053..d46005bddf 100644 --- a/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Leios/Types.hs +++ b/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Leios/Types.hs @@ -43,6 +43,10 @@ module Ouroboros.Consensus.Leios.Types , TxOffset , mkEndorserBlock , endorserBlockFromTxBytes + , ForgedEndorserBlock + , forgedEbBody + , forgedEbTxs + , forgeLeiosEb , endorserBlockHash , endorserBlockReferences , encodeEndorserBlock @@ -295,6 +299,27 @@ endorserBlockFromTxBytes = -- | The position of a transaction in its endorser block. type TxOffset = Int +-- | An endorser block that this node forged, with the bytes of its +-- transactions. Only 'forgeLeiosEb' builds it, so that the bytes always match +-- the references. +data ForgedEndorserBlock = ForgedEndorserBlock + { forgedEbBody :: !EndorserBlock + , forgedEbTxs :: !(NonEmpty ByteString) + -- ^ The bytes of the transactions that 'forgedEbBody' references, in the + -- same order. + } + deriving stock (Show, Eq) + +-- | The endorser block that references the given transactions, in the given +-- order. The reference to a transaction holds the Blake2b-256 hash and the +-- length of its bytes. +forgeLeiosEb :: NonEmpty ByteString -> ForgedEndorserBlock +forgeLeiosEb txBytes = + ForgedEndorserBlock + { forgedEbBody = endorserBlockFromTxBytes txBytes + , forgedEbTxs = txBytes + } + -- | The hash ranking blocks announce for the endorser block: that of its -- encoding. endorserBlockHash :: EndorserBlock -> EndorserBlockHash diff --git a/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/LeiosDB/API.hs b/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/LeiosDB/API.hs index 0b2dec4735..5a9ee89807 100644 --- a/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/LeiosDB/API.hs +++ b/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/LeiosDB/API.hs @@ -20,24 +20,31 @@ module Ouroboros.Consensus.Storage.LeiosDB.API , awaitAll , withWriter , allocateWriter + , storeForgedEb , CompletedEbs , TxOffset ) where import Cardano.Slotting.Slot (SlotNo) import Control.Concurrent.Class.MonadSTM.Strict (StrictTChan) +import Control.Monad (void) import Control.ResourceRegistry (ResourceRegistry, allocate) import Data.ByteString (ByteString) -import Data.Foldable (traverse_) +import Data.Foldable (toList, traverse_) import GHC.Stack (HasCallStack) import NoThunks.Class (OnlyCheckWhnfNamed (..)) import Ouroboros.Consensus.Leios.Types ( BytesSize , EndorserBlock , EndorserBlockHash - , LeiosPoint + , ForgedEndorserBlock + , LeiosPoint (..) , TxHash , TxOffset + , encodeEndorserBlockSize + , endorserBlockHash + , forgedEbBody + , forgedEbTxs ) import Ouroboros.Consensus.Storage.LeiosDB.Trace (LeiosDbStats (..)) import Ouroboros.Consensus.Util.IOLike (IOLike, MonadThrow, NoThunks (..), bracket) @@ -194,3 +201,21 @@ allocateReader registry db = snd <$> allocate registry (\_ -> openReader db) clo allocateWriter :: IOLike m => ResourceRegistry m -> LeiosDbHandle m -> m (LeiosDbWriter m) allocateWriter registry db = snd <$> allocate registry (\_ -> openWriter db) closeWriter + +-- | Write an endorser block that this node forged to the LeiosDB. +-- +-- The slot is the slot of the ranking block that announces the endorser block. +-- That ranking block must announce the 'endorserBlockHash' and the +-- 'encodeEndorserBlockSize' of the endorser block body. +-- +-- This function does not await the writes. On SQLite, it blocks while the +-- write queue is full, and if a write fails, the node stops. Peers can get the +-- header before the endorser block is stored. +storeForgedEb :: Monad m => LeiosDbWriter m -> SlotNo -> ForgedEndorserBlock -> m () +storeForgedEb writer slot forgedEb = do + _ <- writeEbPoint writer point (encodeEndorserBlockSize body) + _ <- writeEbBody writer point body + void $ writeTxs writer point (zip [0 ..] (toList (forgedEbTxs forgedEb))) + where + body = forgedEbBody forgedEb + point = MkLeiosPoint slot (endorserBlockHash body) diff --git a/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/LeiosDB/Impl/SQLite/WriteQueue.hs b/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/LeiosDB/Impl/SQLite/WriteQueue.hs index c3a90ebf79..a4a0951e65 100644 --- a/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/LeiosDB/Impl/SQLite/WriteQueue.hs +++ b/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/LeiosDB/Impl/SQLite/WriteQueue.hs @@ -207,6 +207,7 @@ writerQueueDepth :: Natural writerQueueDepth = numUpstreamPeers + forge + maintenance + slack where numUpstreamPeers = 20 - forge = 1 + -- the point, the body and the txs of a forged endorser block + forge = 3 maintenance = 4 slack = 2 diff --git a/ouroboros-consensus/test/leios-storage-test/Test/Ouroboros/Storage/LeiosDB.hs b/ouroboros-consensus/test/leios-storage-test/Test/Ouroboros/Storage/LeiosDB.hs index 86167d68ed..b91fa5cbf8 100644 --- a/ouroboros-consensus/test/leios-storage-test/Test/Ouroboros/Storage/LeiosDB.hs +++ b/ouroboros-consensus/test/leios-storage-test/Test/Ouroboros/Storage/LeiosDB.hs @@ -26,8 +26,11 @@ import Control.Monad.Class.MonadTime.SI (diffTime, getMonotonicTime) import Control.Tracer (Tracer (..), emit, nullTracer) import qualified Data.ByteString as BS import qualified Data.ByteString.Short as SBS +import Data.Foldable (toList) import Data.Function ((&)) import Data.List (nub, sortOn) +import Data.List.NonEmpty (NonEmpty (..)) +import qualified Data.List.NonEmpty as NE import qualified Data.Map.Strict as Map import Data.Maybe (isJust) import qualified Data.Text as T @@ -41,7 +44,10 @@ import Ouroboros.Consensus.Leios.Types , TxHash , TxOffset , encodeEndorserBlockSize + , endorserBlockHash , endorserBlockReferences + , forgeLeiosEb + , forgedEbBody , mkEndorserBlock ) import Ouroboros.Consensus.Storage.LeiosDB @@ -53,6 +59,7 @@ import Ouroboros.Consensus.Storage.LeiosDB , Promise (..) , TraceLeiosDb (..) , newLeiosDBInMemory + , storeForgedEb , truncateLeiosDbAfterSlot , withLeiosDBSQLite , withReader @@ -247,6 +254,11 @@ mkTestGroups impl = , testProperty "partial txs returns Nothing" $ prop_completedEbPartialTxs impl , testProperty "no body returns Nothing" $ prop_completedEbNoBody impl ] + , testGroup + "storeForgedEb" + [ testProperty "stores the point, the body and the txs of the endorser block" $ + prop_storeForgedEbThenLookup impl + ] ] -- * QuickCheck generators @@ -281,6 +293,12 @@ genEb numTxs = do genPointAndEb :: Int -> Gen (LeiosPoint, EndorserBlock) genPointAndEb numTxs = (,) <$> genPoint <*> genEb numTxs +-- | Transaction bytes, distinct from each other. +genDistinctTxBytes :: Gen (NonEmpty BS.ByteString) +genDistinctTxBytes = NE.nub <$> ((:|) <$> genTxBytes <*> listOf genTxBytes) + where + genTxBytes = chooseInt (1, 32) >>= fmap BS.pack . vector + -- | Bytes for the tx at the given offset of the EB: of its declared size, as -- a write must be to be accepted, and distinct per offset. txBytesFor :: EndorserBlock -> TxOffset -> BS.ByteString @@ -1080,6 +1098,29 @@ data Op -- body, as it would on the wire. type Bodies = Map.Map EndorserBlockHash (EndorserBlock, [Int]) +-- | 'storeForgedEb' writes the endorser block at the point of the given slot +-- and the hash of the body. After the writer closes, a reader sees that point +-- and the bytes of all its txs. +prop_storeForgedEbThenLookup :: DbImpl -> Property +prop_storeForgedEbThenLookup impl = + forAll genSlotNo $ \slot -> + forAll genDistinctTxBytes $ \txs -> + ioProperty $ withFreshDb impl $ \db -> do + let forged = forgeLeiosEb txs + body = forgedEbBody forged + ebHash = endorserBlockHash body + withWriter db $ \writer -> storeForgedEb writer slot forged + (points, closure) <- + withReader db $ \reader -> + (,) <$> testScanEbPoints reader <*> lookupEbClosure reader ebHash + pure $ + conjoin + [ counterexample "the point is not the slot and the hash of the body" $ + points === [(slot, ebHash)] + , counterexample "the stored txs are not the forged ones" $ + closure === Just (zip (map fst (endorserBlockReferences body)) (toList txs)) + ] + -- | Property: running the same operations against both implementations, a -- reader sees the same in each, and each write completes the same endorser -- blocks.