Skip to content

Implementation of Archive Node on TRON #6289

Open
0 of 2 issues completed
Open

Description

@halibobo1205

Background

Currently, Java-tron does not support Archive Node. An archive node can query historical state data from the blockchain. Unlike a fullnode, an archive node not only saves the most recent blockchain state data but also completely records all historical state data from the genesis block to the latest block. This allows users to directly query state data such as account balances and contract storage at any block height without needing to recalculate from the genesis block (replay transactions), greatly improving the efficiency of historical state data queries.

Rationale

Why should this feature exist?

Fullnodes only store the current latest block state (used to verify new blocks), while archive nodes additionally store state snapshots of each historical block (e.g., balances for each address, contract storage data, etc.). Archive nodes can support historical state queries. For example, querying the balance of an address at block height 10,000 cannot be directly provided by a fullnode, but an archive node can do this.

What are the use-cases?

  • DApp developers: testing and verifying smart contract behavior under different historical states
  • Data analysis services: providing advanced analytical functions based on historical states
  • Researchers: analyzing historical transaction patterns, state changes, and network behavior

Specification

Support Ethereum-compatible interfaces, including but not limited to:

  • eth_getBalance: query an account's balance at a specific block height
  • eth_getCode: query contract bytecode at a specific block height
  • eth_getStorageAt: query the value at a specific position in contract storage at a specific block height
  • eth_call: execute read-only contract calls on the state at a specific block height

Implementation

Archive nodes need a world state trie, which TRON needs to implement.

Ethereum state trie implementation

Ethereum supported a state trie in its initial version, implementing core state trie logic:

  1. The block header contains three roots: transactionRoot, receiptRoot, and stateRoot
  2. Every block state data change is reflected in the stateRoot
  3. Account data, including contracts (nonce, balance, storageRoot, codeHash), is all uniformly encapsulated in the account
  4. Uses MPT (Merkle Patricia Trie) for state data storage (including CRUD operations)

Image

TRON's current situation

  1. The block header contains 2 roots: txTrieRoot and accountStateRoot. TxTrieRoot is consistent with Ethereum's transactionRoot, but accountState only includes balance (the proposal is not enabled) and does not include contract data (storageRoot, codeHash) like ETH does.

  2. Supports historical account balance queries: After enabling balance.history.lookup, account balances at specific heights can be queried via wallet/getaccountbalance.

  3. State data is quite dispersed and not fully encapsulated in accounts (e.g., contract data, TRC10 are stored separately in different DBs). After organization, 25 databases need state statistics, including account data, TRC10 data, contract data, voting data, delegation data, etc. All state data needs to be organized and collected to generate the stateRoot

Implementation considerations for Archive nodes:

  1. stateRoot generation
    TRON's state data is quite dispersed, involving many state databases. Unlike ETH, which generates root directly based on accounts, TRON needs to collect dispersed state data to generate a stateRoot.

  2. Should stateRoot participate in consensus?
    Consensus issues:

    • A proposal is needed to generate stateRoot at a specified height, requiring consideration of how to calculate the initial effective height of stateRoot, whether to generate a genesis stateRoot based on the current state, and if not, how to ensure network-wide data consistency.
    • Historical data before this cannot generate stateRoot, and cannot support historical state queries.
    • Generating and verifying state roots impacts performance, potentially reducing TPS and transaction execution efficiency, affecting SRs and the ordinary fullNode.
  3. Compatibility with all ETH state query interfaces

    • Support for eth_getBalance, eth_getCode, eth_getStorageAt, eth_call
    • Future support for eth_getBlockReceipts(Implement eth_getBlockReceipts method #5910 ) and debug_traceCall(Support the debug_traceCall API #5778 )
    • For discussion: whether to support eth_getTransactionCount (requires adding nonce field to Account, requiring a hard fork proposal) and eth_getProof (requires storageRoot, lite fullnodes, L2 support, etc.)
  4. Performance impact
    Archive nodes significantly impact block execution and transaction execution efficiency, especially when historical state data is large. The implementation must ensure the normal operation of block synchronization and transaction execution.

  5. Storage impact
    The transaction volume of TRON is far greater than that of ETH, with the archive node data expected to be 3-4 times that of ETH, estimated at 80 T+. Current single disk storage will be limited, requiring consideration of using multiple segmented Archive Nodes combined into a complete historical query collection.

Based on the above analysis, stateRoot will not participate in consensus to reduce the impact on SRs and the ordinary fullnodes during archive node implementation. Segmented multi-disk storage of state data will be used to solve single-disk storage limitations. Additionally, since archive-related functionality implementation involves significant changes affecting many functions, integration into the main branch is not currently considered, and development will first proceed on an independent branch.

Activity

  1. waynercheung commented on Apr 18, 2025

    @waynercheung
    Collaborator

    Hi @@halibobo1205 , it's a good news that TRON will implement an archive node to offer the historical information.

    And about the stateRoot generation, you have mentioned that it's different with ETH, can you provide more details about it?
    such as is there also a MPT for account?

  2. lily309 commented on Apr 18, 2025

    @lily309

    As I understand it, since java-tron uses protobuf as a prototype for database storage and API interaction. In the definition of the TRON protocol, the structure of accounts is more complex, so what kind of form will the archive node take when storing these historical account states?

    A simple example may make it easier to describe the problem, for example, the structure of account A has balance/stake information/vote information, etc. If in a certain block, account A initiates a transfer, then only one balance field in the information of account A has changed. In the archive node implementation, does the account A corresponding to the height of the block where the transaction occurred still store all the complete account information? Or is there some special storage structure that reduces the storage overhead?

  3. halibobo1205 commented on Apr 18, 2025

    @halibobo1205
    CollaboratorAuthor

    @lily309 still stores all the complete account information. The storage may become larger in this case, but the query operation will become easier. Still, at the same time, due to the database storage becoming larger, the query may also increase the time-consuming.
    Storing only changing fields is good for storage, but it can get complicated for queries. Are there any known optimizations?
    BTW, if Tron only implements historical queries, replace the MPT with flat KV(such as erigon's Flat KV) instead of MPTs?

  4. halibobo1205 commented on Apr 18, 2025

    @halibobo1205
    CollaboratorAuthor

    stateRoot generation
    ETH is about the MPT of the account; Tron's state data is split and not centralized in the account. The MPT tree is roughly designed as follows:

    1. Each type of data has its own MPT
    Image
    1. All data forms a single MPT
    Image

    @waynercheung, I favor the second one, which I'll explain in more detail later.

  5. lily309 commented on Apr 18, 2025

    @lily309

    Understood.

    Of course, I'm also quite interested in erigon, who have implemented an archive node for Ethereum with only a tiny disk space.

    I may not know much about the exact technical details yet, but it's nice to see that the TRON development community is considering a more efficient storage structure to solve the huge disk footprint problem posed by archive nodes, so if you guys have a follow-up update, please let me know.

  6. waynercheung commented on Apr 21, 2025

    @waynercheung
    Collaborator

    Hi @halibobo1205 , where is the storage of contract data?
    Does the "Storage Keys" in the above image include the contract data?

    For ETH, each account has a storage root, as "Storage trie is where all contract data lives. There is a separate storage trie for each account." mentioned by storage-trie.

  7. waynercheung commented on Apr 21, 2025

    @waynercheung
    Collaborator

    @waynercheung, I favor the second one, which I'll explain in more detail later.

    Hi @halibobo1205 , I have another question about this, to my knowledge, java-tron has an important concept called 'proposal'. Is this data included in the MPT tree?

  8. Federico2014 commented on Apr 21, 2025

    @Federico2014
    Collaborator

    Ethereum will use the Verkle tree to replace Merkle-Patricia Trie. Will you consider adopting the Verkle tree directly to improve performance?

  9. 317787106 commented on Apr 21, 2025

    @317787106
    Collaborator

    @halibobo1205 The stateRoot can compare with other nodes to ensure consistency, but how to query the balance by specifying the block height in eth_getBalance? Accumulate changes?

  10. 317787106 commented on Apr 21, 2025

    @317787106
    Collaborator

    There are 25 database modifications included, so what is the selection criteria? receiptRoot of ETH is not generated, it's transactioninfo of TRON, why not used instateRoot ?

  11. halibobo1205 commented on Apr 21, 2025

    @halibobo1205
    CollaboratorAuthor

    Does the "Storage Keys" in the above image include the contract data?

    Yes, contract data, Tron is currently associated with code, contract, storage-row, contract-state, that is, if using a separate root, there are 4.
    cc @yanghang8612

  12. waynercheung commented on Apr 21, 2025

    @waynercheung
    Collaborator

    Yes, contract data, Tron is currently associated with code, contract, storage-row, contract-state, that is, if using a separate root, there are 4. cc @yanghang8612

    What does the "Storage Row Data" mean?
    It just includes the contract data or it also includes other data?

    Besides that, for Tron, there is only one MPT and just one stateRoot?
    Because for ETH, there are four MPT which is State Trie, Storage Trie, Transactions Trie and Receipts Trie, and there are three roots: stateRoot, transactionsRoot and receiptsRoot.

  13. bladehan1 commented on Apr 21, 2025

    @bladehan1
    Collaborator

    Ethereum has implemented the path-based-storage-model to solve the problem of dynamic pruning of the world state tree for ordinary full nodes. In this change (or considering the future), ordinary full nodes also generate StateRoot. Should we consider using the path-based-storage-model?

  14. fortuna502 commented on Apr 21, 2025

    @fortuna502

    There are 25 database modifications included, so what is the selection criteria? receiptRoot of ETH is not generated, it's transactioninfo of TRON, why not used instateRoot ?

    @halibobo1205 Can you show more details about the 25 database? what are they?

  15. 45 remaining items

  16. halibobo1205 commented on May 15, 2026

    @halibobo1205
    CollaboratorAuthor

    Quick update on where we are.

    4.8.2 is still in its testing cycle; in parallel, we've started an AI-assisted research pass on the archive-node design. Sharing the preliminary direction here — details will keep shifting as we prototype.

    Landscape recalibration (May 2026):

    • Erigon v3 (GA 2025-03) replaced the v2 ChangeSet model we referenced earlier in this thread with a three-layer immutable-file design at txNum granularity: Domain (.kv) for the latest state, History (.v) for prior values, InvertedIndex (.ef) for key → txNums (Elias-Fano compressed). Mainnet ETH archive on E3 is ~1.6–1.8 TB, vs ~3 TB on v2 and >20 TB on legacy Geth HBSS.
    • Reth 2.0 uses static files + MDBX, archive ~2.1 TB, but keeps a per-block ChangeSet model — coarser than E3.
    • EIP-4444 is live on Ethereum mainnet (2025); base full nodes will increasingly stop serving history, which actually strengthens the case for a TRON archive node.
    • Geth / BSC PBSS are MPT-internal optimizations — not our target architecture.
    • Verkle is deferred to Hegota (H2 2026 candidate), and does not apply to TRON, given that stateRoot is out of consensus.

    What this means for Java-tron:

    1. The 75 TB MPT prototype confirms MPT is the wrong substrate for our query workload, not that the storage cost is intrinsic.
    2. The ">500 GB / >700 tx/block" cliff is most likely trie write amplification; a flat KV + segment-file model removes that hot path.
    3. Since stateRoot is not in TRON consensus, there is no reason to keep a trie at all. Design target: adapt E3's three-layer model to TRON's ~25 state DBs — a small live KV at the tip, per-DB Domain / History / InvertedIndex segments, history organized by txNum. This subsumes the ChangeSet-only sketch I posted upthread.

    Open questions we're now researching (input welcome):

    • Mapping TRON's StateType keyspace onto E3-style domains: one combined domain vs per-DB.
    • Snapshot cutover boundary: rolling window vs cold-archive split.
    • Whether to expose a txNum-granular history API (would enable transaction-level debug_traceCall).
    • Indexer model for fields not naturally addressable by (address, slot) — proposals, delegated resources, market orders, etc.

    We aim to share the next round of progress updates around Q3 2026.

  17. warku123 commented on Aug 13, 2026

    @warku123
    Contributor

    Design question for the txNum-based history model.

    The Domain/History/InvertedIndex design assumes every state change belongs to a txNum. TRON has state writes that no transaction carries: maintenance-period processing (SR re-election, every 6h), block/vote rewards (payReward, Manager.java:1917), adaptive energy updates (EnergyProcessor, Manager.java:1911), resource decay, proposal activation. In a txNum-indexed history these become source-less changes — "value at txNum T" can't distinguish a system write from a missing record, and replay up to a txNum boundary can't reproduce them.

    Erigon already solves this: every block reserves 2 virtual system-tx slots (systemBegin / systemEnd); consensus-driven writes like miner rewards are attributed to the trailing virtual tx (execution/types/block.go, BaseTxnID, TxCountToTxAmount = txsLen + 2). The same pattern fits TRON directly — e.g. reserve per-block system slots in the synthesized txNum space and attribute maintenance/reward/DPS writes to them.

    Questions:

    1. Should the txNum layout reserve per-block system-write slots (Erigon-style), and how many does TRON need (maintenance-period batches may touch many stores at once)?
    2. Affected stores appear to include at least AccountStore (rewards), DynamicPropertiesStore (energy/proposals), WitnessStore/WitnessScheduleStore/VotesStore (maintenance) — worth an explicit inventory before fixing the layout.
    3. How should system slots interact with segment unwinding on reorgs?
  18. halibobo1205 commented on Aug 26, 2026

    @halibobo1205
    CollaboratorAuthor

    @warku123 Good catch, and your read of the Erigon precedent matches where our design is heading — BaseTxnID with TxCountToTxAmount = txsLen + 2 is the right reference point. Source-less state writes have to become explicit positions in the txNum timeline; otherwise block-end state cannot be reconstructed unambiguously.

    1. Reserve two unconditional system positions per block

    I would use:

    BLOCK_PREPARE | USER_TX ... | BLOCK_FINALIZE

    BLOCK_PREPARE contains state transitions that must be visible before the first user transaction. BLOCK_FINALIZE contains trailing consensus-driven transitions such as rewards, adaptive-energy updates, maintenance processing and proposal activation.

    The number of slots should follow externally observable execution phases, not the number of stores touched. A maintenance round may update many stores, but if those updates form one atomic block-finalization transition, they should share one txNum.

    Multiple writes to the same (domain, key) within one system phase must be coalesced into the first observed before-value and the final after-value. Intermediate values are not externally observable at that phase boundary.

    Both positions must be reserved unconditionally, including empty blocks and phases that produce no changes. This gives every block a stable temporal shape.

    The semantics must remain explicit:

    • A missing changeset at a reserved position means the key was unchanged during that phase.
    • A tombstone means the key was deleted or absent after the transition.
    • A missing system position or block range means corruption or unavailable history.

    Two system slots are sufficient as long as all block-driven writes occur either before all user transactions or after all user transactions. If a future consensus rule introduces an observable system transition between user transactions, that requires a new phase and schema version rather than overloading the existing slots.

    2. Freeze an explicit write inventory before freezing the layout

    The inventory should cover every write-capable store and every path that can bypass the generic store write hook. Each store should have one explicit classification:

    • directly captured history domain;
    • semantic backing captured through a higher-level hook;
    • derived or intentionally excluded, with a documented reconstruction argument.

    Likely state families include AccountStore, DynamicPropertiesStore, WitnessStore, VotesStore, ProposalStore, DelegationStore, delegated-resource stores, ContractStore, ContractStateStore, code and contract storage.

    AccountAssetStore requires special treatment because optimized TRC10 balances may be physically separated from the account row. The archive should capture the effective logical asset transition rather than blindly archive physical flush writes.

    WitnessScheduleStore may be classified as derived from votes and witness state rather than stored as an independent history domain, but that exclusion must be explicit.

    Classification should fail closed. If a block is active and a write reaches an unknown store or an unclassified bypass path, the block's archive capture must fail rather than silently omit the transition. A source-inventory or build-time test should also compare known store declarations against the registry so the inventory cannot silently become stale.

    For an initial TVM-state milestone, reads outside the declared coverage boundary should return a specific unsupported-history error. They must never fall back to latest state.

    Writes should be classified by their actual execution position, not merely by service name. For example, resource decay performed while executing a user transaction belongs to that user txNum; only genuinely block-driven resource updates belong to a system phase.

    3. Unwind system positions as part of the block range

    The prepare and finalize positions should be ordinary members of the block's txNum range. A head unwind therefore removes, atomically:

    • all temporal rows belonging to the block range;
    • both system positions;
    • user-transaction positions;
    • transaction and block lookup rows;
    • the canonical range and journal metadata.

    We recommend defining txNum as a canonical-chain coordinate. Under that model, unwinding the head rewinds the allocator to the orphaned range's first txNum. The replacement branch is captured again and may numerically reuse the released coordinate space.

    This keeps canonical history dense and gap-free. Safety against stale or ABA results must come from validating block hash, canonical generation/epoch and query snapshot lifetime, not from permanently burning txNums.

    A never-reuse allocator is also possible, but it is a different model: it requires a separately persisted high-water mark, permits permanent gaps, and must survive crashes independently of canonical range cleanup. That complexity should only be accepted if txNum is intentionally defined as an execution-attempt identifier rather than a canonical-chain coordinate.

    Published or otherwise irreversible ranges should not participate in normal shallow reorg unwind. A conflict with an already published range should fail stop and require repair.

    Queries for an orphaned block hash must fail closed. A height formerly occupied by an orphan is not permanently invalid: once the winning canonical block at that height is captured and published, height-based queries should resolve to the new canonical range.

    Independent validation

    Internal archive consistency checks are insufficient for system writes because a consistently omitted reward can still produce plausible history.

    Reward validation must model all applicable paths, not only signer rewards:

    • block reward for the signer;
    • standby-witness reward weighted by votes;
    • brokerage split;
    • delegation-cycle reward entries;
    • transaction-fee-pool reward;
    • proposal-controlled parameter changes;
    • reward withdrawal and resulting account deltas.

    A controlled private chain can simplify the oracle by fixing or disabling optional reward components. For each block it should verify that state immediately before finalization excludes system effects, while block-end state contains the exact independently calculated account, delegation and dynamic-property changes.

    Empty blocks, maintenance boundaries, reorg replacement, crash between canonical commit and archive publication, and restart reconciliation should all preserve the same two-position block shape.

    Happy to go deeper on any of these — the maintenance-batch ordering assumption (one logical txNum per round) is the one most likely to need revisiting first, so a counterexample there would be especially useful.

  19. warku123 commented on Aug 27, 2026

    @warku123
    Contributor

    This comment focuses only on checkpoint terminology and crash consistency.

    1. Terminology collision

    In an earlier design update, halibobo1205 called java-tron’s existing WAL a “checkpoint database” to demonstrate existing ChangeSet capability. In this issue, checkpoint therefore already means the crash-recovery WAL.

    The archive design needs a separate term for the anchor established after 19 solidified blocks. I suggest solidified anchor or epoch anchor, reserving checkpoint for the existing WAL semantics.

    2. Semantics of the existing checkpoint mechanism

    • SnapshotManager.createCheckpoint() runs during flush, traverses all revoking stores, flattens recent cross-database change sets into (dbName + key) → (value + operator), and writes them to a separate checkpoint store.
      chainbase/src/main/java/org/tron/core/db2/core/SnapshotManager.java:357-405
    • pruneCheckpoint() retains only about three recent timestamp directories; this is short-lived crash recovery, not archival storage.
      chainbase/src/main/java/org/tron/core/db2/core/SnapshotManager.java:442-467
    • recover() replays only checkpoints that have already been flushed.
      chainbase/src/main/java/org/tron/core/db2/core/SnapshotManager.java:522-550
    • Flush timing varies relative to solidification: maxFlushCount is 1 during catch-up and larger during normal operation.
      framework/src/main/java/org/tron/core/db/Manager.java:1075-1089

    Thus, if the change-log is derived at flush time, a crash window between solidification and materialization is expected, not a corner case.

    3. Reuse boundary

    createCheckpoint()’s change-set traversal is approximately 70% reusable as a per-block cross-database change collector, but it lacks block/phase markers, before values, append-only retention, and a digest.

    The precise proposal is: reuse the traversal code, not its lifecycle or semantics. The archive change-log must be append-only, sealed at the solidification boundary, and idempotent by (blockNum, blockHash).

    4. Crash consistency requires the same fate

    This directly addresses the proposed test: crash between canonical commit and archive publication, and restart reconciliation.

    The change-log must commit with the state DB:

    • committing the log first creates a “log exists, state does not” window;
    • committing it later can cause missing entries.

    Generate the change-log from the same collection point as the checkpoint by reusing createCheckpoint()’s capture point. During recovery, reconcile against checkpoint recover() as the source of truth.
    chainbase/src/main/java/org/tron/core/db2/core/SnapshotManager.java:357-405,522-550

    Use (blockNum, blockHash) as the archive idempotency key so restart reconciliation neither republishes nor omits a canonical block.

    Q1 / Q2

    Q1: Do you agree that checkpoint should be reserved for the crash-recovery WAL, with the solidification boundary named solidified anchor or similar?

    Q2: Can same-source change-log generation at the checkpoint capture point—with state-DB-equivalent commit fate and (blockNum, blockHash) idempotency—serve as the baseline crash-consistency design?

  20. halibobo1205 commented on Aug 27, 2026

    @halibobo1205
    CollaboratorAuthor

    @warku123
    Both questions are well posed, and the terminology one is overdue. Answering in order, with one factual correction that changes the conclusion of Q2.

    Q1 — Agreed, and the split needs to go one step further

    checkpoint is already taken by the crash-recovery WAL in this thread, and overloading it would be actively harmful because the things involved have different retention semantics.

    There are four distinct concepts here, and I would name them separately:

    • checkpoint — the existing crash-recovery WAL. Pruned to a few recent directories. Unchanged.
    • in-flight journal — the per-block archive record, written before materialization. Durable but transient: orphan cleanup is allowed, and it is retired once published.
    • solidified anchor — the finality boundary at which a block becomes eligible for materialization. This is a boundary. Your term; I would adopt it.
    • publish — the act of materializing journal records into archive segments. The resulting published segment is the canonical, permanent, append-only artifact.

    The permanence claim belongs to the published segment, not to the journal. If a design does want the journal itself to be append-only, then an orphan must be retired by writing a cancellation/tombstone record rather than by physical deletion — but then the journal is no longer cheap to prune, so I would keep the two lifetimes distinct.

    Most of the ambiguity in earlier discussions came from using one word for the record and the act, not from the boundary. Worth fixing all four at once.

    Q2 — The invariants are right; the sourcing point is where I would diverge

    Where I would not follow is sourcing the in-flight journal from createCheckpoint()'s capture point, for three reasons.

    1. The traversal flattens, by construction. createCheckpoint() walks for (int i = 0; i < flushCount; ++i) over the snapshot chain and merges everything into a single map keyed by (dbName + key). When flushCount > 1 that is, last-write-wins across multiple blocks: per-block boundaries and intermediate values are gone before the batch is written. A txNum-indexed history needs per-block and per-phase attribution, which is exactly what the merge discards. The ~70% reuse estimate holds for the shape of the traversal, but the missing 30% is the part that carries the information.

    2. The flush-count direction is the opposite of what you describe, and it makes (1) worse. In Manager.processBlock, the configured maxFlushCount is applied when the head block is ≥60s old — i.e., during catch-up — and SnapshotManager.DEFAULT_MIN_FLUSH_COUNT (which is 1) applies near head during normal operation. So batching is heaviest precisely during a from-0 archive sync, which is where the overwhelming majority of archive writing happens. A flush-sourced log would lose the most attribution in the phase that matters most, and would look deceptively well-behaved when tested near head.

    3. Before-values are the History layer, and they are cheapest at the mutation point. At flush time, the pre-image has already been superseded in the snapshot chain, so recovering it means reading back through the revoking layers at exactly the moment you are trying to write. At the mutation point the pre-image is still unambiguous — though capturing it may cost one additional read on the consensus hot path, which is a real cost that belongs in the design budget rather than being waved away. Repeated writes to the same key within one phase should retain the first before-value and the final after-value; intermediate values are not observable at the phase boundary.

    There is also a fourth point about "same fate" itself: even when capturing at the same code point, archive segments live in a different durable unit than the canonical state DB. Committing them in one write batch requires them to be the same database instance. Otherwise "same fate" degrades to "same capture point, different commit" — which reintroduces the window it was meant to close. Single-database atomicity does not solve canonical/archive cross-DB 2PC, and a design should not claim that it does.

    The replacement protocol: one-directional ordering, not atomicity

    Rejecting cross-DB atomicity only helps if something concrete replaces it. The ordering I would specify:

    1. capture in memory during block apply
    2. persist the journal as PREPARED, before the canonical commit
    3. if the journal write fails -> abort the canonical commit
    4. commit canonical state
    5. mark the journal ACKNOWLEDGED
    6. once publishableFrontier passes it, publish and retire the journal atomically
    

    Step 6's atomic scope has to be stated precisely, or it is not a guarantee: publish segment rows, block marker, publication cursor, and journal retirement in one atomic transaction of the same archive backend. Publication advances strictly by block height and never skips a covered block.

    Step 3 is the load-bearing one. Because the journal is durable before the canonical commit and a journal failure aborts that commit, the ordering is one-directional: the only reachable inconsistency is an orphan journal (journal exists, canonical does not), never the silent inverse (canonical committed, nothing captured). That is a deliberate trade — an archive-enabled node stops rather than advancing uncaptured — and it is what makes the window decidable instead of merely narrow.

    A crash between 4 and 5 leaves a journal that is durable and canonically valid but unacknowledged; restart reconcile re-acknowledges it by comparing the canonical hash at that height. No information is lost, because acknowledgement is derivable state.

    Idempotency key: identity only

    (blockNum, blockHash) is the right identity, and it should stay exactly that. I would not fold a schema or content digest into the key — I initially considered it, and it is wrong: if the digest is part of the key, the same block under two schema versions yields two distinct legal keys, which bypasses conflict detection instead of triggering it.

    Instead, schemaVersion / schemaDigest / contentDigest belong in the journal envelope, participating in content authentication. The rule is then unambiguous: the same (blockNum, blockHash) observed with a different schema or a different payload is corruption, not idempotent re-delivery.

    Publish must gate on durability, not only on finality

    A block being solidified is not sufficient to make it publishable. The correct frontier is:

    publishableFrontier = min(solidifiedAnchor, durableCanonicalFrontier)
    

    where durableCanonicalFrontier is the highest block that crash recovery — root plus checkpoint replay — can guarantee to restore. Recent canonical blocks still living only in revoking snapshots, and blocks inside a pending batched flush, are not durable yet. A block that is solidified but not yet durable may keep its journal record, but must not be materialized into an irreversible archive segment: doing so creates precisely the "archive ahead of recoverable canonical" state that cannot be reconciled afterwards.

    Reconcile: compare hashes at the same height, respect the frontier, and run after recovery

    "Journal above canonical head" is only one of the cases. The full rule:

    journal height > canonical head
        -> orphan; roll back
    
    journal height <= head, hash != canonical hash at that height
        -> orphan; roll back. If already published -> fail stop
    
    hash == canonical hash, height > publishableFrontier
        -> retain and acknowledge as in-flight; do NOT publish
    
    hash == canonical hash, height <= publishableFrontier
        -> publish idempotently, then retire the journal
    
    any canonical block in [lastPublished + 1 .. canonicalHead], within the
    declared archive coverage, having neither a journal record nor a published
    segment
        -> fatal capture gap
    

    Two things worth calling out. The second case is what "above head" alone misses: a reorg that happened while the node was down leaves a journal record at or below head whose hash no longer matches.

    And the gap check must span all the way to canonicalHead, not stop at the frontier. A canonical block above the frontier with no journal is just as unrecoverable — the in-memory capture opportunity is already gone — so deferring the check until that block solidifies would only discover the gap long after the last moment it could have been prevented. Scoping the check to the declared archive coverage keeps it from firing on intentionally uncaptured state.

    On your framing of the checkpoint as truth source, I would put it slightly differently: reconcile must run after SnapshotManager.recover() completes. The checkpoint is not the final source of truth — the canonical chain is, and it has unbounded retention where pruneCheckpoint() keeps only a few recent directories — but checkpoint replay is the step that produces the post-recovery canonical state you reconcile against. Sequencing matters more than which one is called authoritative.

    Under this model, the two failure modes you name become explicit, decidable states rather than silent divergence: "log exists, state does not" is a deterministic rollback, and "missing entries" is a fatal gap.

    Your reuse boundary — "the traversal code, not its lifecycle or semantics" — is the right instinct. I would draw it one step earlier: reuse the idea of a cross-database change collector, but site it where before-values and block boundaries still exist, rather than where they have already been merged away.

  21. halibobo1205 commented on Sep 2, 2026

    @halibobo1205
    CollaboratorAuthor

    Phase 1: Research Update

    1. Erigon V3 uses a global txNum timeline for transaction and system-driven state changes. Domains, History, and Inverted Indexes provide historical reads without storing a complete state snapshot at every block.

    2. java-tron’s existing RevokingDB and checkpoint mechanisms primarily support short-term rollback and crash recovery. They are not sufficient as long-term archive storage.

    Next, I will map java-tron’s state-write paths and system-generated state changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

  • Status
    In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions