Skip to content

docs(archive): review state archive performance work - #53

Draft
bladehan1 wants to merge 176 commits into
developfrom
feature/archive_block2
Draft

bladehan1 wants to merge 176 commits into
developfrom
feature/archive_block2

Conversation

@bladehan1

@bladehan1 bladehan1 commented Sep 11, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

Opens the current State Archive implementation and performance evidence for code-difference review only.

Why are these changes required?

The branch contains the current State Archive persistence, recovery, serving, ARM/AMD validation, and performance-tracking changes that need review before any merge decision.

This PR has been tested by:

  • Existing branch-level validation and recorded State Archive harness evidence
  • No new product change is introduced by opening this Draft PR

Follow up

Review only. Keep this PR in Draft; do not merge or enable auto-merge.

Extra details

This PR is intentionally created in the origin fork bladehan1/java-tron, not tronprotocol/java-tron. No upstream repository or runtime deployment is changed.


Summary by cubic

Review-only PR for the State Archive branch. It adds an opt-in archive for durable reverse history, recovery, serving, historical reads, and path-state roots; with archive features disabled, the existing current-state path keeps its current behavior.

State Archive

  • Captures canonical block snapshots and persists bounded history body, index, and commit records with backpressure, reorg handling, and restart-safe recovery.
  • Adds exact-27 startup admission, native LevelDB/RocksDB recovery, immutable serving generations, engine identity checks, and fail-closed drift detection.
  • Supports request-owned historical reads with query limits and cancellation; historical VM execution rejects SELFDESTRUCT, CREATE, and CREATE2.
  • Adds path-state rebuild, catch-up, transitions, reorg rewind, P66 migration, compaction, runtime status, and non-consensus block metadata.
  • Adds offline block replay, file tools, and resumable HTTP export.

Performance Evidence

  • Records ARM/AMD HLT-002 observations and JDK17/rocksdb 9.7.4 comparisons while retaining JDK8 controls.
  • Includes layered trie deltas, owned snapshot heads, parallel bounded rebuilds with progress, and reduced cross-store reads through lazy migration.

Keep this PR in Draft; do not merge or auto-merge.

Written for commit 9c483be. Summary will update on new commits.

Review in cubic

bladehan1 added 30 commits July 17, 2026 14:54
Capture canonical block metadata and materialize reverse diffs from completed block snapshots. Classify archive state stores explicitly and project account-asset companion mutations before root merge.
Add deterministic history body and index records, ordered durable commit markers, and bounded asynchronous persistence.

Gate checkpoint advancement on committed archive history and cover crash, corruption, backpressure, and reorg behavior.
Persist reverse state history with bounded append-only body, index, and commit structures. Wire an opt-in archive writer into snapshot flush and add restart-safe account change indexing. Expose an experimental historical account balance endpoint for trace comparison.
Persist bounded restart checkpoints, truncation recovery, H/C/D/R progress authorities, and immutable serving generations. Pin committed history resources and fail closed on identity or durability drift.
Add durable engine identities and native LevelDB/RocksDB snapshot leases. Assemble exact Store generations under the SnapshotManager barrier and release partial acquisitions on authority drift.
Apply business mutations with participant progress in native sync batches. Freshly validate H/C/D authorities under the archive barrier and advance reader visibility only after exact convergence.
Unify LevelDB and RocksDB participants behind one recovery contract. Persist canonical mutation plans and bind checkpoint, participant, and reader progress to the same digest.
Collect exact ordinary and AccountAsset post-state without Store scans. Enforce target identity, bounded ownership, one-shot sealing, abort, and terminal payload release.
Drive an exact-27 capture batch through mixed native participants, inject a partial apply failure, and verify durable-plan replay, digest propagation, ownership, and restart convergence.
Apply two capture-produced targets through exact-27 participants. Verify per-target plan digests, overwrite and delete semantics, partial replay boundaries, reader publication, and restart convergence.
Prepare one target-bound Account and AccountAsset projection before snapshot merge, retain its forward payload through the revoking lifecycle, and seal it only against durable history markers.

Keep the integration default-off and preserve the existing proposal 66 physical layout.
Classify ABI as mutable auxiliary metadata outside the versioned state participant set while retaining an explicit tombstone classification.
Exercise proposal-bound projection, layer-to-flush ownership, durable marker sealing, retry and reorg boundaries, exact participant coverage, and ABI exclusion.
Expose durable marker receipt creation through direct and asynchronous history sinks, then gate forward payload sealing on the exact committed flush range.

Retain submitted frozen ranges across durability or receipt failures so retries do not resubmit history.
Verify direct and asynchronous receipt authority, disabled behavior, and retry ordering across durable wait and receipt failures without duplicate history submission.
Expose explicit block-final stages with lightweight session commit. Sample account node reads, hash verification, decoding and native reads behind an opt-in flag, and export prepared-attempt metrics with bounded labels.
@github-actions

Copy link
Copy Markdown

❌ Math Usage Detection Results

Found forbidden usage of java.lang.Math in the following files:

./chainbase/src/main/java/org/tron/core/db2/archive/ArchiveFormatAdmissionValidator.java
./chainbase/src/main/java/org/tron/core/db2/archive/ArchiveHistoryWriter.java
./chainbase/src/main/java/org/tron/core/db2/archive/ArchiveTargetMutationPlanCodec.java
./chainbase/src/main/java/org/tron/core/db2/archive/ArchiveWalStartupValidator.java
./chainbase/src/main/java/org/tron/core/db2/archive/BlockReverseDiff.java
./chainbase/src/main/java/org/tron/core/db2/archive/HistoricalAccountAssetPrefixResolver.java
./chainbase/src/main/java/org/tron/core/db2/archive/PersistentServingKeyIndexGeneration.java
./chainbase/src/main/java/org/tron/core/db2/archive/ServingKeyIndexGeneration.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveFiveLaneDurabilityProofV3.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveFiveLaneSegmentWriterV3.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveHistoryCatalogV3.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveHotStore.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveRuntimeOwner.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveServingWorkerV3.java
./chainbase/src/main/java/org/tron/core/db2/core/CommonCheckpointPayload.java
./chainbase/src/main/java/org/tron/core/db2/core/CommonCheckpointRedoCoordinator.java
./chainbase/src/main/java/org/tron/core/db2/core/CommonCheckpointRuntime.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathMerkleTrie.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateBlockTransition.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateCatchUpQueue.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateCommitmentCodec.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateFlushTarget.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateLayerLimits.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateNativeNodeStore.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateNativeSnapshotSource.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateNodeStoreSet.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStatePhysicalOverlayHead.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStatePhysicalStoreSet.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateRebuildCoordinator.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateRoot.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateSnapshotDelta.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateStackTrie.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateStoreTrieBuilder.java
./framework/src/main/java/org/tron/core/db2/stateroot/PathStatePhysicalOracle.java
./framework/src/main/java/org/tron/program/BlockReplay.java
./framework/src/main/java/org/tron/program/HttpBlockExport.java
./framework/src/test/java/org/tron/core/db2/archive/PersistentServingKeyIndexGenerationTest.java
./framework/src/test/java/org/tron/core/db2/stateroot/PathStateNativeNodeStoreTest.java
./framework/src/test/java/org/tron/core/db2/stateroot/PathStateNativeSnapshotSourceTest.java

Please review if this usage is intended.

Caution

Note: You should use org.tron.common.math.StrictMathWrapper.
If you need to use java.lang.Math, please provide a justification.

@github-actions

Copy link
Copy Markdown

❌ Math Usage Detection Results

Found forbidden usage of java.lang.Math in the following files:

./chainbase/src/main/java/org/tron/core/db2/archive/ArchiveFormatAdmissionValidator.java
./chainbase/src/main/java/org/tron/core/db2/archive/ArchiveHistoryWriter.java
./chainbase/src/main/java/org/tron/core/db2/archive/ArchiveTargetMutationPlanCodec.java
./chainbase/src/main/java/org/tron/core/db2/archive/ArchiveWalStartupValidator.java
./chainbase/src/main/java/org/tron/core/db2/archive/BlockReverseDiff.java
./chainbase/src/main/java/org/tron/core/db2/archive/HistoricalAccountAssetPrefixResolver.java
./chainbase/src/main/java/org/tron/core/db2/archive/PersistentServingKeyIndexGeneration.java
./chainbase/src/main/java/org/tron/core/db2/archive/ServingKeyIndexGeneration.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveFiveLaneDurabilityProofV3.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveFiveLaneSegmentWriterV3.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveHistoryCatalogV3.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveHotStore.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveRuntimeOwner.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveServingWorkerV3.java
./chainbase/src/main/java/org/tron/core/db2/core/CommonCheckpointPayload.java
./chainbase/src/main/java/org/tron/core/db2/core/CommonCheckpointRedoCoordinator.java
./chainbase/src/main/java/org/tron/core/db2/core/CommonCheckpointRuntime.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathMerkleTrie.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateBlockTransition.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateCatchUpQueue.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateCommitmentCodec.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateFlushTarget.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateLayerLimits.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateNativeNodeStore.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateNativeSnapshotSource.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateNodeStoreSet.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStatePhysicalOverlayHead.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStatePhysicalStoreSet.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateRebuildCoordinator.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateRoot.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateSnapshotDelta.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateStackTrie.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateStoreTrieBuilder.java
./framework/src/main/java/org/tron/core/db2/stateroot/PathStatePhysicalOracle.java
./framework/src/main/java/org/tron/program/BlockReplay.java
./framework/src/main/java/org/tron/program/HttpBlockExport.java
./framework/src/test/java/org/tron/core/db2/archive/PersistentServingKeyIndexGenerationTest.java
./framework/src/test/java/org/tron/core/db2/stateroot/PathStateNativeNodeStoreTest.java
./framework/src/test/java/org/tron/core/db2/stateroot/PathStateNativeSnapshotSourceTest.java

Please review if this usage is intended.

Caution

Note: You should use org.tron.common.math.StrictMathWrapper.
If you need to use java.lang.Math, please provide a justification.

@github-actions

Copy link
Copy Markdown

❌ Math Usage Detection Results

Found forbidden usage of java.lang.Math in the following files:

./chainbase/src/main/java/org/tron/core/db2/archive/ArchiveFormatAdmissionValidator.java
./chainbase/src/main/java/org/tron/core/db2/archive/ArchiveHistoryWriter.java
./chainbase/src/main/java/org/tron/core/db2/archive/ArchiveTargetMutationPlanCodec.java
./chainbase/src/main/java/org/tron/core/db2/archive/ArchiveWalStartupValidator.java
./chainbase/src/main/java/org/tron/core/db2/archive/BlockReverseDiff.java
./chainbase/src/main/java/org/tron/core/db2/archive/HistoricalAccountAssetPrefixResolver.java
./chainbase/src/main/java/org/tron/core/db2/archive/PersistentServingKeyIndexGeneration.java
./chainbase/src/main/java/org/tron/core/db2/archive/ServingKeyIndexGeneration.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveFiveLaneDurabilityProofV3.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveFiveLaneSegmentWriterV3.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveHistoryCatalogV3.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveHotStore.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveRuntimeOwner.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveServingWorkerV3.java
./chainbase/src/main/java/org/tron/core/db2/core/CommonCheckpointPayload.java
./chainbase/src/main/java/org/tron/core/db2/core/CommonCheckpointRedoCoordinator.java
./chainbase/src/main/java/org/tron/core/db2/core/CommonCheckpointRuntime.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathMerkleTrie.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateBlockTransition.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateCatchUpQueue.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateCommitmentCodec.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateFlushTarget.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateLayerLimits.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateNativeNodeStore.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateNativeSnapshotSource.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateNodeStoreSet.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStatePhysicalOverlayHead.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStatePhysicalStoreSet.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateRebuildCoordinator.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateRoot.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateSnapshotDelta.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateStackTrie.java
./chainbase/src/main/java/org/tron/core/db2/stateroot/PathStateStoreTrieBuilder.java
./framework/src/main/java/org/tron/core/db2/stateroot/PathStatePhysicalOracle.java
./framework/src/main/java/org/tron/program/BlockReplay.java
./framework/src/main/java/org/tron/program/HttpBlockExport.java
./framework/src/test/java/org/tron/core/db2/archive/PersistentServingKeyIndexGenerationTest.java
./framework/src/test/java/org/tron/core/db2/stateroot/PathStateNativeNodeStoreTest.java
./framework/src/test/java/org/tron/core/db2/stateroot/PathStateNativeSnapshotSourceTest.java

Please review if this usage is intended.

Caution

Note: You should use org.tron.common.math.StrictMathWrapper.
If you need to use java.lang.Math, please provide a justification.

- halt after five-lane rotation seal and post-rotation marker, verify
  identity, rebuilt durability proof and zero-action second reopen
- halt across common checkpoint wal publish, redo, lost-ack and retire
  boundaries, verify recovery convergence and no temporary authority
- halt with committed common and unpublished serving index, verify
  replay of (I,W] from durable progress and zero-action restart
- normal reopen trusts catalog-sealed segments after O(1) length, seal
  frame and manifest checks instead of scanning all history
- current segments resume at the durability proof marker chain and only
  the uncheckpointed tail is scanned; identity rebuilds field-identical
  to the retained recovery scan
- missing, stale or drifting proof, catalog or sealed records fail
  closed; torn uncheckpointed tails still repair through the explicit
  recovery scan; no configuration switch is added
restore published-proof conservative scanning and trim recovery; add short-checkpoint replay coverage and a valid Catalog proof fixture
Bind recovery intent to the source and target catalog identities, commit terminal-aware catalog state before reopen, and roll catalog plus five-lane tails back to the exact published Common checkpoint with proof-bounded I/O.

Retain legacy catalog and intent compatibility and add crash-boundary, mixed-terminal, short-prefix, and partial-successor coverage.
Retain the remote conservative-recovery commit in branch history while resolving its overlapping implementation and tests to the verified checkpoint-bound recovery path.
Publish an exact source-bound catalog transition when the five archive lanes already match the durable checkpoint but the selected catalog still lags behind. Keep repeated checkpoint reopen zero-action and require catalog binding for mutation-free recovery intents.
Prune obsolete catalog generations after every successful writer recovery or fast reopen. This closes the durable window where CURRENT is committed but process termination interrupts retention cleanup, while preserving fail-closed identity validation before deletion.
Replace the catalog-bound archive writer with five catalogless lanes, segment metadata, bounded recovery, and single-lane point reads.\n\nUse request-height historical reads without a checkpoint-wide lease, keep serving-index construction fail closed, and align the runtime configuration with format version 4.
Add the V5 five-lane block index, compact frame codec, append and reopen lifecycle, single-lane point reads, serving-index integration, and fail-closed file-descriptor admission.\n\nKeep format V4 as the default and require explicit V5 selection with an isolated root and canonical Block Store metadata.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant