feat: serialize blob info snapshot in-process at the epoch boundary#3480
feat: serialize blob info snapshot in-process at the epoch boundary#3480shuowang12 wants to merge 1 commit into
Conversation
|
Warning: This PR modifies one of the example config files. Please consider the
|
ab81bd4 to
66fa63b
Compare
66fa63b to
fbeb651
Compare
|
I have some review suggestions. cc: @halfprice @sadhansood The main goal if the PR is to start writing snapshot files on a few mainnet nodes to gather metrics (serialize time, size, cross-node digest). I expect the review focus to be on safety and durability. Open but not blocking: the encoding format (blob_info_snapshot.rs). I polished the encoding format and tried several variants back and forth and now it is in good shape and welcome to review. It should be ready for future milestones but still flexible to change as we have not started it using it. |
Description
Adds opt-in serialization of the blob info snapshot at the epoch boundary, behind
blob_info_snapshot.enabled(off by default).When enabled, the storage node serializes the three blob-info column families (
per_object_blob_info,storage_pool_info,per_object_pooled_blob_info) to a local file once per epoch and reports the serialization duration, size, and content digest. The node does not read snapshots back yet or use it in anyway.The serialization runs after GC phase 1 settles the tables and before
execute_epoch_change, while event processing is blocked, so the serialized state is the deterministic post-GC point. A crash before completion replays the handler and retries to create the snapshot and will short-circuits if the file already exists.The encoding format is versioned and self-delimiting: magic, version, a BCS header { epoch, event_cursor }, then three count-prefixed sections of length-framed (ObjectID, value) entries in ascending key order.
Test plan
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that
a user might notice and any actions they must take to implement updates. (Add release notes after the colon for each item)