docs(archive): review state archive performance work

.helper/ai-archive/execution/records/ITER-20260910-397.md

@@ -0,0 +1,18 @@
+# ITER-20260910-397
+
+## Slice
+
+HLT-002 AMD-002 JDK17/RocksDB 9.7.4 candidate gate and baseline restoration.
+
+## Observed
+
+- AMD-002 installed OpenJDK `17.0.20` at `/usr/lib/jvm/java-17-openjdk-amd64/bin/java`; installation changed the system `java` alternative, so the live control was relaunched with the explicit JDK8 binary `/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java`.
+- Initial JDK17 smoke failed closed before database open with `Java 1.8 is required for amd64 architecture. Detected version 17`. This was an application architecture gate, not a RocksDB JNI failure.
+- Commit `9ea27f2797` adds the opt-in system property `tron.java17.x86.candidate`; default x86 behavior remains Java8-only. The commit is signed and pushed to `origin/feature/archive_block2`.
+- Rebuilt JDK17 + RocksDB 9.7.4 candidate on AMD-002; build succeeded and produced SHA-256 `91a0560fc9a4581d9f5643e0233c60772be00cfb8945154760942f83af7ef76e`.
+- With `-Dtron.java17.x86.candidate=true` and P2P disabled, candidate startup succeeded. Existing 27 PathState stores, Chainbase LevelDB stores, Common checkpoint, and State Archive stores opened; no JNI/linkage exception observed. Candidate unit was stopped after the startup gate, so no performance claim is made.
+- AMD-002 control was restored using explicit JDK8, unit `amd002-state-archive-current-8ac7a6afb1-live.service`; follow-up verification showed active PID and ports 8090/18888/9527 listening.
+
+## Conclusion
+
+JDK17 application startup compatibility is now proven for the existing AMD snapshot when explicitly opted in, while the production/default x86 gate remains unchanged. HLT-002 still lacks an attributable P2P performance window and long-term compaction evidence; keep the 9.7.4/JDK17 runtime as a candidate only.

.helper/ai-archive/execution/records/ITER-20260910-398.md

@@ -0,0 +1,17 @@
+# ITER-20260910-398
+
+## Slice
+
+HLT-002 AMD-002 JDK17 + RocksDB 9.7.4 P2P short window.
+
+## Observed
+
+- Candidate unit `amd002-hlt002-jdk17-rocks97-9ea27f2797` started with `/usr/lib/jvm/java-17-openjdk-amd64/bin/java`, `-Dtron.java17.x86.candidate=true`, existing snapshot runtime, and P2P enabled.
+- Window start via `/wallet/getnowblock`: block `85150075`.
+- After approximately 120 seconds, head was `85150103`: +28 blocks, approximately `0.23 block/s`.
+- Candidate logs showed peer block reception and no `ERROR`, `Exception`, or `Caused by` entries in the sampled tail.
+- Candidate was stopped; AMD-002 control was restored with explicit JDK8 binary `/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java`. Unit active with `NRestarts=0`; ports 8090/18888/9527 listening.
+
+## Conclusion
+
+The JDK17/RocksDB9.7.4 candidate passes a short P2P liveness window on the existing snapshot. Because the candidate started from an older snapshot than the live control, the +28-block rate is mechanism evidence only, not a strict A/B performance promotion. A synchronized control/candidate pair and longer window remain required before native-options tuning or 30m/6h gates.

.helper/ai-archive/execution/records/ITER-20260910-399.md

@@ -0,0 +1,16 @@
+# ITER-20260910-399
+
+## Slice
+
+HLT-002 AMD-002 same-snapshot sequential A/B short window.
+
+## Observed
+
+- While control was stopped, two reflink runtimes were created from the same `runtime-current-8ac7a6afb1` snapshot; existing runtimes and failed candidates were retained.
+- JDK8 control (`amd002-hlt002-ab-control-9ea27f2797`): measured 120 seconds from block `85150420` to `85150443`, +23 blocks, approximately `0.192 block/s`.
+- JDK17 + RocksDB 9.7.4 candidate (`amd002-hlt002-ab-jdk17-9ea27f2797`): measured 120 seconds from block `85150409` to `85150435`, +26 blocks, approximately `0.217 block/s`.
+- Candidate produced no sampled `ERROR`, `Exception`, or `Caused by` lines. Both units were stopped cleanly. AMD-002 original live was restored with explicit JDK8 and remains the active control.
+
+## Conclusion
+
+The synchronized snapshot experiment provides directional evidence of approximately 13% higher short-window throughput for the JDK17/RocksDB9.7.4 candidate. It is not a promotion result: runs were sequential, network conditions were not randomized, and each window is only 120 seconds. Repeat interleaved windows and collect checkpoint/compaction/resource metrics before native-options tuning or long-term gates.

.helper/ai-archive/execution/records/ITER-20260910-400.md

@@ -0,0 +1,17 @@
+# ITER-20260910-400
+
+## Slice
+
+HLT-002 second same-snapshot A/B preparation and candidate startup failure.
+
+## Observed
+
+- AMD-002 live was stopped and two fresh reflink runtimes were created from `runtime-current-8ac7a6afb1`; existing runtimes were retained.
+- The JDK17 candidate unit reached `ActiveState=active` at the systemd level but did not expose API port 8090. Startup log reported `BeanCreationException` caused by `RuntimeException: Asset num is wrong!` during manager initialization.
+- No performance window was counted. The candidate was stopped and AMD-002 control was restored with explicit JDK8; PID and ports 8090/18888/9527 are healthy.
+- Post-failure inspection showed the failed clone's `database/block` had `CURRENT=MANIFEST-000002` and only about 16 KB, while the source live database had `CURRENT=MANIFEST-001533` and about 4.7 GB. The copy was incomplete, explaining `latestBlockHeaderNumber=0` and `Asset num is wrong!`.
+- AssetUpdateHelper logged `Total block:1`, confirming the cloned block store exposed latest height zero rather than the source live height.
+
+## Conclusion
+
+Do not treat this as a JDK17/RocksDB incompatibility: the earlier clean snapshot candidate passed startup and P2P liveness. Before another A/B attempt, complete the copy under a long-running job and gate startup on regular-file totals, database CURRENT/MANIFEST presence and size, decoded archive asset identity, and PathState/Common checkpoint alignment; preserve this failed runtime as evidence.

.helper/ai-archive/execution/records/ITER-20260910-401.md

@@ -0,0 +1,16 @@
+# ITER-20260910-401
+
+## Slice
+
+HLT-002 verified long-running reflink copy and startup-gate check.
+
+## Observed
+
+- A dedicated systemd copy unit completed successfully after the live node was stopped. Source `runtime-current-8ac7a6afb1` and destination `runtime-hlt002-safe-jdk17` were compared before any candidate use.
+- `database/block`: 769 regular files and 5,050,533,467 apparent bytes on both sides; `CURRENT` and `MANIFEST-001533` SHA-256 matched.
+- `path-state-root`: 3,272 regular files on both sides; `CURRENT` SHA-256 matched. `state-archive/hot`: 149 regular files and 2,476,596,233 apparent bytes on both sides; `CURRENT` SHA-256 matched.
+- A subsequent candidate smoke was accidentally started while the control had already been restored, so Prometheus port binding conflicted. Its API output came from the live control and is discarded; the candidate was stopped immediately. AMD-002 control remains active with JDK8 and ports 8090/18888/9527.
+
+## Conclusion
+
+The copy-integrity gate now distinguishes the prior 16-KB incomplete clone from a complete snapshot. The next candidate run must stop control for the entire smoke/performance window and verify ownership of 8090/18888/9527 before counting any API or throughput result.

.helper/ai-archive/execution/records/ITER-20260910-402.md

@@ -0,0 +1,15 @@
+# ITER-20260910-402
+
+## Slice
+
+HLT-002 complete-copy candidate startup gate.
+
+## Observed
+
+- A fresh snapshot was copied by the long-running `amd002-hlt002-safe-copy2` unit while control was stopped. `database/block` matched source at 771 files and 5,062,113,945 bytes; `CURRENT` SHA-256 matched.
+- With control kept stopped for the entire smoke, JDK17 + RocksDB9.7.4 candidate unit `amd002-hlt002-safe2-jdk17` started successfully using the explicit opt-in property. API port 8090 and PBFT port 9527 were owned by the candidate; `/wallet/getnowblock` returned head `85150943`. Latest logs showed all API services started and JVM `17.0.20`.
+- Candidate was stopped and AMD-002 control was restored with explicit JDK8. Existing incomplete/failed runtimes remain retained.
+
+## Conclusion
+
+The complete-copy and exclusive-port gates now produce a valid JDK17 startup result from the latest snapshot. This closes the prior snapshot/port false negatives; a P2P performance window can now be run from this validated destination, with control kept stopped until candidate teardown.

.helper/ai-archive/execution/records/ITER-20260910-403.md

@@ -0,0 +1,16 @@
+# ITER-20260910-403
+
+## Slice
+
+HLT-002 verified-snapshot JDK17/RocksDB9.7.4 P2P window.
+
+## Observed
+
+- A fresh candidate runtime was copied by long-running unit `amd002-hlt002-p2p-copy`; block DB validation matched source at 772 files and 5,069,146,804 bytes, with identical `CURRENT`.
+- Control remained stopped while candidate unit `amd002-hlt002-p2p-jdk17` owned ports 8090/18888/9527.
+- 120-second P2P window: head `85151013→85151045`, +32 blocks, approximately `0.267 block/s`. At sampling, process CPU was about 471% and RSS about 9.8 GB.
+- Candidate was stopped and AMD-002 control restored with explicit JDK8. Historical error lines in the reused candidate log were from prior failed experiments; no new failure occurred in this window.
+
+## Conclusion
+
+The verified-copy and exclusive-port gates now support a valid JDK17 P2P sample. The observed rate is directionally above the earlier JDK8 same-snapshot sample (`0.192 block/s`), but remains a short, non-interleaved measurement; repeat control/candidate windows with synchronized resource and checkpoint telemetry before promotion.

.helper/ai-archive/execution/records/ITER-20260910-404.md

@@ -0,0 +1,16 @@
+# ITER-20260910-404
+
+## Slice
+
+HLT-002 AMD-002 host exclusivity and slow-sync attribution.
+
+## Observed
+
+- AMD-002 reports 16 vCPUs on an AMD EPYC 7R13 host. At observation time load average was `4.81/5.09/4.86`; the FullNode was the only material user process at about 467% CPU. Alloy and system agents were each near 1% or below.
+- The live unit has no CPU quota (`CPUQuotaPerSecUSec=infinity`), MemoryHigh 29 GiB, MemoryMax 30 GiB, swap disabled, and MemoryCurrent about 17.0 GB. `vmstat` showed 85–94% idle CPU and no swap activity.
+- I/O PSI was non-zero (`some/full avg10 about 5.3%`), while the node had 525 listed peers. This points away from host CPU contention and toward storage wait plus per-block database/write/commit work; peer count alone does not establish useful sync bandwidth.
+- The environment is not proven bare-metal exclusive: it is a 16-vCPU host/VM with background system services. No competing heavy workload was observed during the sample.
+
+## Conclusion
+
+The observed ~0.2–0.3 block/s is not explained by another process monopolizing AMD-002 CPU or by the cgroup CPU limit. Next HLT-002 windows must collect interval disk latency/utilization, write amplification, checkpoint/compaction timing, and per-block stage timings; only then tune RocksDB options or claim a storage fix.

.helper/ai-archive/execution/records/ITER-20260910-405.md

@@ -0,0 +1,16 @@
+# ITER-20260910-405
+
+## Slice
+
+HLT-002 ARM-001 versus AMD-002 live configuration attribution.
+
+## Observed
+
+- The actual ARM-001 host is `10.255.10.101` (the previously checked `10.255.10.84` currently runs a Geth archive process). ARM-001 FullNode PID `1665805` runs JDK8 with `/data/blade/state-archive-4f143bd6b8/config-p66-live.conf`.
+- ARM-001 live config reports `db.engine=ROCKSDB`, State Archive enabled, and `pathStateRoot.mode=shadow`; its service has no cgroup MemoryHigh/MemoryMax and no CPU quota. Host load was `7.06` on 16 vCPUs, FullNode about 454% CPU, IO PSI avg10 about 2.55%.
+- AMD-002 live runs JDK8 with Chainbase LevelDB, synchronous PathState/Archive semantics, MemoryHigh 29 GiB and MemoryMax 30 GiB, no CPU quota. Host load was about `4.56`, FullNode about 410% CPU, IO PSI avg10 about 5.38%, cgroup MemoryCurrent about 30.1 GB.
+- ARM-001 did not expose the expected 8090/18888 API ports in this observation; only 9527 was listening. Therefore no current same-window block-rate comparison was performed.
+
+## Conclusion
+
+AMD's lower observed rate is plausibly explained by the heavier synchronous persistence path, Chainbase LevelDB, and tighter memory/cgroup/IO conditions rather than lack of host CPU. ARM historical rates are not an apples-to-apples control until ARM is actually running the documented synchronous configuration with matching JDK, engines, cgroup, snapshot head, and transaction density.

.helper/ai-archive/execution/records/ITER-20260911-406.md

@@ -0,0 +1,16 @@
+# ITER-20260911-406
+
+## Slice
+
+HLT-002 ARM/AMD runtime alignment requested by operator.
+
+## Observed
+
+- ARM-001 (`10.255.10.101`) live JVM is OpenJDK `17.0.20` and its current unit already has `MemoryHigh=MemoryMax=31138512896` (30 GiB), with swap disabled. The effective config has `pathStateRoot.mode=shadow`, `volatileSnapshotBenchmark=false`, and `asyncPrepareBenchmark=false`.
+- `StorageConfig.PathStateRootConfig.postProcess()` rejects every mode other than `shadow`; writing literal `mode=sync` would fail closed at startup. The effective synchronous behavior is represented by the two benchmark flags remaining false.
+- AMD-002 was switched from explicit JDK8 to the signed JDK17+RocksDB9.7.4 candidate JAR, with `-Dtron.java17.x86.candidate=true`, MemoryHigh 29 GiB, MemoryMax 30 GiB, and swap disabled. Unit `amd002-state-archive-current-9ea27f2797-live` is active with zero restarts, ports 8090/18888/9527 listening.
+- After startup, AMD head advanced `85151840→85151866` in about 60 seconds; this is a live liveness sample, not a long-term performance claim.
+
+## Conclusion
+
+The requested JDK17 AMD live switch is complete. ARM already matches JDK17 and 30-GiB cgroup limits, but cannot be changed to a literal `sync` mode without a separate format/config contract change; current flags already disable volatile and async benchmark paths. Fair performance comparison remains blocked on aligning ARM's actual runtime mode/engine semantics with AMD and collecting matched windows.

.helper/ai-archive/execution/records/ITER-20260911-407.md

@@ -0,0 +1,16 @@
+# ITER-20260911-407
+
+## Slice
+
+HLT-002 ARM/AMD aligned live model deployment.
+
+## Observed
+
+- ARM-001 was restarted with explicit `/usr/lib/jvm/java-17-openjdk-arm64/bin/java`, Xmx18G/direct1G, MemoryHigh=29 GiB, MemoryMax=30 GiB, MemorySwapMax=0. Its effective `pathStateRoot.mode=shadow` remains required by `StorageConfig`; both volatile and async benchmark flags are false, which is the supported synchronous path.
+- The first restart attempt left the transient unit in a stopped state; after reset-failed and a clean re-create, `state-archive-8ac7-live.service` became active with PID `1811527`, zero restarts, and the 30-GiB cgroup settings.
+- AMD-002 was switched live to explicit JDK17 + RocksDB9.7.4 candidate JAR with `tron.java17.x86.candidate=true`, MemoryHigh=29 GiB, MemoryMax=30 GiB, swap disabled. Unit `amd002-state-archive-current-9ea27f2797-live` is active with PID `2726724`, zero restarts, ports 8090/18888/9527 listening, and head advanced `85151840→85151866` in approximately 60 seconds.
+- ARM's literal `mode=sync` was not written because the current code rejects it at config post-processing; changing that contract is a separate implementation task.
+
+## Conclusion
+
+Both live nodes now use JDK17 and the same 29-GiB high/30-GiB max cgroup envelope. Their storage engines and supported PathState mode still differ (ARM Chainbase RocksDB; AMD Chainbase LevelDB), so future speed comparisons must retain that distinction.

.helper/ai-archive/execution/records/ITER-20260911-408.md

@@ -0,0 +1,16 @@
+# ITER-20260911-408
+
+## Slice
+
+HLT-002 aligned live baseline rate observation.
+
+## Observed
+
+- ARM-001 and AMD-002 are both live on JDK17 with Xmx18G/direct1G, MemoryHigh 29 GiB, MemoryMax 30 GiB, swap disabled, Common checkpoint enabled, PathState/Archive RocksDB, `mode=shadow`, and volatile/async benchmark flags false.
+- The remaining intentional engine difference is Chainbase: ARM RocksDB versus AMD LevelDB.
+- Concurrent metrics sampling used `tron:block_process_latency_seconds_count{sync="true"}`. ARM increased `530→565` over about 73 seconds (`~0.48 block/s`); AMD increased `777→804` over about 76 seconds (`~0.36 block/s`). Header gauges at the later sample were ARM `83,851,674` and AMD `85,152,623`.
+- This was online natural traffic, not locked tx/block, peers, network, or interleaved A/B. No restart or error gate was observed during sampling.
+
+## Conclusion
+
+The aligned baseline currently shows ARM processing faster than AMD by roughly 35% in this short observation, consistent with Chainbase engine as a candidate factor. It is not causal proof; use matched snapshot/traffic windows and collect per-block stage, I/O, and checkpoint/compaction metrics before attributing the gap to LevelDB versus RocksDB.

.helper/ai-archive/execution/records/ITER-20260911-409.md

@@ -0,0 +1,16 @@
+# ITER-20260911-409
+
+## Slice
+
+HLT-002 aligned live 1k-block observation setup.
+
+## Observed
+
+- ARM-001 current header gauge at setup: `83,851,674`; 1k target: `83,852,674`.
+- AMD-002 current header gauge at setup: `85,152,623`; 1k target: `85,153,623`.
+- Both nodes remain JDK17 with Xmx18G/direct1G, MemoryHigh 29 GiB, MemoryMax 30 GiB, swap disabled, Common checkpoint enabled, PathState/Archive RocksDB, `mode=shadow`, and volatile/async benchmark flags false. Chainbase remains the sole intentional engine difference (ARM RocksDB, AMD LevelDB).
+- The observation contract is to retain complete 1k ranges and collect header/block-process counters, PushBlock and checkpoint stage timings, tx count/tx per block, CPU, I/O PSI, RSS/cgroup memory, GC and error/restart counters. No cross-height rate merging is allowed.
+
+## Conclusion
+
+The 1k-block quantitative bottleneck observation is armed from explicit per-node anchors. Final attribution is deferred until each target is reached and the complete windows are exported; expected candidates are Chainbase engine write/compaction cost, checkpoint amortization, and I/O wait.

.helper/ai-archive/execution/records/ITER-20260911-410.md

@@ -0,0 +1,16 @@
+# ITER-20260911-410
+
+## Slice
+
+HLT-002 1k-block night-watch deployment.
+
+## Observed
+
+- Watcher script `hlt002-1k-watch.sh` was committed as `1ab34783c9` and cgroup-path fix as `eb0a2baeed`, then installed on ARM-001 and AMD-002.
+- ARM watcher unit `arm001-hlt002-1k-watch` is active; first corrected sample head `83,851,985`, process count `882`, cgroup current `16,640,135,168` bytes, CPU `464%`, RSS `12,186,192 KiB`.
+- AMD watcher unit `amd002-hlt002-1k-watch` is active; first corrected sample head `85,152,911`, process count `1,096`, cgroup current `30,057,975,808` bytes, CPU `367%`, RSS `21,535,316 KiB`, IO PSI avg60 about `19.16%`.
+- Targets remain ARM `83,852,959` and AMD `85,153,885`; watchers stop themselves at target and preserve TSV evidence.
+
+## Conclusion
+
+Night watch is active on both live nodes. The first corrected sample already shows materially higher AMD memory pressure and IO PSI; final bottleneck attribution waits for complete 1k ranges and stage counters.