Skip to content

Commit 4e98a4a

Browse files
ensi321dappliong11tech
committed
feat: implement EIP-6110 (#6042)
* Add immutable in the dependencies * Initial change to pubkeyCache * Added todos * Moved unfinalized cache to epochCache * Move populating finalized cache to afterProcessEpoch * Specify unfinalized cache during state cloning * Move from unfinalized to finalized cache in afterProcessEpoch * Confused myself * Clean up * Change logic * Fix cloning issue * Clean up redundant code * Add CarryoverData in epochCtx.createFromState * Fix typo * Update usage of pubkeyCache * Update pubkeyCache usage * Fix lint * Fix lint * Add 6110 to ChainConfig * Add 6110 to BeaconPreset * Define 6110 fork and container * Add V6110 api to execution engine * Update test * Add depositReceiptsRoot to process_execution_payload * State transitioning to EIP6110 * State transitioning to EIP6110 * Light client change in EIP-6110 * Update tests * produceBlock * Refactor processDeposit to match the spec * Implement processDepositReceipt * Implement 6110 fork guard for pubkeyCache * Handle changes in eth1 deposit * Update eth1 deposit test * Fix typo * Lint * Remove embarassing comments * Address comments * Modify applyDeposit signature * Update packages/state-transition/src/cache/epochCache.ts Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com> * Update packages/state-transition/src/cache/epochCache.ts Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com> * Update packages/state-transition/src/cache/pubkeyCache.ts Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com> * Remove old code * Rename fields in epochCache and immutableData * Remove CarryoverData * Move isAfter6110 from var to method * Fix cyclic import * Fix operations spec runner * Fix for spec test * Fix spec test * state.depositReceiptsStartIndex to BigInt * getDeposit requires cached state * default depositReceiptsStartIndex value in genesis * Fix pubkeyCache bug * newUnfinalizedPubkeyIndexMap in createCachedBeaconState * Lint * Pass epochCache instead of pubkey2IndexFn in apis * Address comments * Add unit test on pubkey cache cloning * Add unfinalizedPubkeyCacheSize to metrics * Add unfinalizedPubkeyCacheSize to metrics * Clean up code * Add besu to el-interop * Add 6110 genesis file * Template for sim test * Add unit test for getEth1DepositCount * Update sim test * Update besudocker * Finish beacon api calls in sim test * Update epochCache.createFromState() * Fix bug unfinalized validators are not finalized * Add sim test to run a few blocks * Lint * Merge branch 'unstable' into 611 * Add more check to sim test * Update besu docker image instruction * Update sim test with correct tx * Address comment + cleanup * Clean up code * Properly handle promise rejection * Lint * Update packages/beacon-node/src/execution/engine/types.ts Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com> * Update comments * Accept type undefined in ExecutionPayloadBodyRpc * Update comment and semantic * Remove if statement when adding finalized validator * Comment on repeated insert on finalized cache * rename createFromState * Add comment on getPubkey() * Stash change to reduce diffs * Stash change to reduce diffs * Lint * addFinalizedPubkey on finalized checkpoint * Update comment * Use OrderedMap for unfinalized cache * Pull out logic of deleting pubkeys for batch op * Add updateUnfinalizedPubkeys in regen * Update updateUnfinalizedPubkeys logic * Add comment * Add metrics for state context caches * Address comment * Address comment * Deprecate eth1Data polling when condition is reached * Fix conflicts * Fix sim test * Lint * Fix type * Fix test * Fix test * Lint * Update packages/light-client/src/spec/utils.ts Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com> * Fix spec test * Address comments * Improve cache logic on checkpoint finalized * Update sim test according to new cache logic * Update comment * Lint * Finalized pubkey cache only update once per checkpoint * Add perf test for updateUnfinalizedPubkeys * Add perf test for updateUnfinalizedPubkeys * Tweak params for perf test * Freeze besu docker image version for 6110 * Add benchmark result * Use Map instead of OrderedMap. Update benchmark * Minor optimization * Minor optimization * Add memory test for immutable.js * Update test * Reduce code duplication * Lint * Remove try/catch in updateUnfinalizedPubkeys * Introduce EpochCache metric * Add historicalValidatorLengths * Polish code * Migrate state-transition unit tests to vitest * Fix calculation of pivot index * `historicalValidatorLengths` only activate post 6110 * Update sim test * Lint * Update packages/state-transition/src/cache/epochCache.ts Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com> * Improve readability on historicalValidatorLengths * Update types * Fix calculation * Add eth1data poll todo * Add epochCache.getValidatorCountAtEpoch * Add todo * Add getStateIterator for state cache * Partial commit * Update perf test * updateUnfinalizedPubkeys directly modify states from regen * Update sim test. Lint * Add todo * some improvements and a fix for effectiveBalanceIncrements fork safeness * rename eip6110 to elctra * fix electra-interop.test.ts --------- Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com> Co-authored-by: gajinder <develop@g11tech.io>
1 parent 1e4afd9 commit 4e98a4a

81 files changed

Lines changed: 2096 additions & 213 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎packages/beacon-node/src/chain/chain.ts‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -871,6 +871,9 @@ export class BeaconChain implements IBeaconChain {
871871
metrics.forkChoice.balancesLength.set(forkChoiceMetrics.balancesLength);
872872
metrics.forkChoice.nodes.set(forkChoiceMetrics.nodes);
873873
metrics.forkChoice.indices.set(forkChoiceMetrics.indices);
874+
875+
const headState = this.getHeadState();
876+
metrics.headState.unfinalizedPubkeyCacheSize.set(headState.epochCtx.unfinalizedPubkey2index.size);
874877
}
875878

876879
private onClockSlot(slot: Slot): void {
@@ -959,6 +962,40 @@ export class BeaconChain implements IBeaconChain {
959962
if (headState) {
960963
this.opPool.pruneAll(headBlock, headState);
961964
}
965+
966+
const cpEpoch = cp.epoch;
967+
const electraEpoch = headState?.config.ELECTRA_FORK_EPOCH ?? Infinity;
968+
969+
if (headState === null) {
970+
this.logger.verbose("Head state is null");
971+
} else if (cpEpoch >= electraEpoch) {
972+
// Get the validator.length from the state at cpEpoch
973+
// We are confident the last element in the list is from headEpoch
974+
// Thus we query from the end of the list. (cpEpoch - headEpoch - 1) is negative number
975+
const pivotValidatorIndex = headState.epochCtx.getValidatorCountAtEpoch(cpEpoch);
976+
977+
if (pivotValidatorIndex !== undefined) {
978+
// Note EIP-6914 will break this logic
979+
const newFinalizedValidators = headState.epochCtx.unfinalizedPubkey2index.filter(
980+
(index, _pubkey) => index < pivotValidatorIndex
981+
);
982+
983+
// Populate finalized pubkey cache and remove unfinalized pubkey cache
984+
if (!newFinalizedValidators.isEmpty()) {
985+
this.regen.updateUnfinalizedPubkeys(newFinalizedValidators);
986+
}
987+
}
988+
}
989+
990+
// TODO-Electra: Deprecating eth1Data poll requires a check on a finalized checkpoint state.
991+
// Will resolve this later
992+
// if (cpEpoch >= (this.config.ELECTRA_FORK_EPOCH ?? Infinity)) {
993+
// // finalizedState can be safely casted to Electra state since cp is already post-Electra
994+
// if (finalizedState.eth1DepositIndex >= (finalizedState as CachedBeaconStateElectra).depositReceiptsStartIndex) {
995+
// // Signal eth1 to stop polling eth1Data
996+
// this.eth1.stopPollingEth1Data();
997+
// }
998+
// }
962999
}
9631000

9641001
async updateBeaconProposerData(epoch: Epoch, proposers: ProposerPreparationData[]): Promise<void> {

‎packages/beacon-node/src/chain/regen/queued.ts‎

Lines changed: 49 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
import {toHexString} from "@chainsafe/ssz";
22
import {phase0, Slot, allForks, RootHex, Epoch} from "@lodestar/types";
33
import {IForkChoice, ProtoBlock} from "@lodestar/fork-choice";
4-
import {CachedBeaconStateAllForks, computeEpochAtSlot} from "@lodestar/state-transition";
4+
import {CachedBeaconStateAllForks, UnfinalizedPubkeyIndexMap, computeEpochAtSlot} from "@lodestar/state-transition";
55
import {Logger} from "@lodestar/utils";
66
import {routes} from "@lodestar/api";
77
import {CheckpointHex, toCheckpointHex} from "../stateCache/index.js";
@@ -125,6 +125,54 @@ export class QueuedStateRegenerator implements IStateRegenerator {
125125
return this.checkpointStateCache.updatePreComputedCheckpoint(rootHex, epoch);
126126
}
127127

128+
/**
129+
* Remove `validators` from all unfinalized cache's epochCtx.UnfinalizedPubkey2Index,
130+
* and add them to epochCtx.pubkey2index and epochCtx.index2pubkey
131+
*/
132+
updateUnfinalizedPubkeys(validators: UnfinalizedPubkeyIndexMap): void {
133+
let numStatesUpdated = 0;
134+
const states = this.stateCache.getStates();
135+
const cpStates = this.checkpointStateCache.getStates();
136+
137+
// Add finalized pubkeys to all states.
138+
const addTimer = this.metrics?.regenFnAddPubkeyTime.startTimer();
139+
140+
// We only need to add pubkeys to any one of the states since the finalized caches is shared globally across all states
141+
const firstState = (states.next().value ?? cpStates.next().value) as CachedBeaconStateAllForks | undefined;
142+
143+
if (firstState !== undefined) {
144+
firstState.epochCtx.addFinalizedPubkeys(validators, this.metrics?.epochCache ?? undefined);
145+
} else {
146+
this.logger.warn("Attempt to delete finalized pubkey from unfinalized pubkey cache. But no state is available");
147+
}
148+
149+
addTimer?.();
150+
151+
// Delete finalized pubkeys from unfinalized pubkey cache for all states
152+
const deleteTimer = this.metrics?.regenFnDeletePubkeyTime.startTimer();
153+
const pubkeysToDelete = Array.from(validators.keys());
154+
155+
for (const s of states) {
156+
s.epochCtx.deleteUnfinalizedPubkeys(pubkeysToDelete);
157+
numStatesUpdated++;
158+
}
159+
160+
for (const s of cpStates) {
161+
s.epochCtx.deleteUnfinalizedPubkeys(pubkeysToDelete);
162+
numStatesUpdated++;
163+
}
164+
165+
// Since first state is consumed from the iterator. Will need to perform delete explicitly
166+
if (firstState !== undefined) {
167+
firstState?.epochCtx.deleteUnfinalizedPubkeys(pubkeysToDelete);
168+
numStatesUpdated++;
169+
}
170+
171+
deleteTimer?.();
172+
173+
this.metrics?.regenFnNumStatesUpdated.observe(numStatesUpdated);
174+
}
175+
128176
/**
129177
* Get the state to run with `block`.
130178
* - State after `block.parentRoot` dialed forward to block.slot

‎packages/beacon-node/src/chain/stateCache/fifoBlockStateCache.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,10 @@ export class FIFOBlockStateCache implements BlockStateCache {
172172
}));
173173
}
174174

175+
getStates(): IterableIterator<CachedBeaconStateAllForks> {
176+
throw new Error("Method not implemented.");
177+
}
178+
175179
/**
176180
* For unit test only.
177181
*/

‎packages/beacon-node/src/chain/stateCache/persistentCheckpointsCache.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -587,6 +587,10 @@ export class PersistentCheckpointStateCache implements CheckpointStateCache {
587587
});
588588
}
589589

590+
getStates(): IterableIterator<CachedBeaconStateAllForks> {
591+
throw new Error("Method not implemented.");
592+
}
593+
590594
/** ONLY FOR DEBUGGING PURPOSES. For spec tests on error */
591595
dumpCheckpointKeys(): string[] {
592596
return Array.from(this.cache.keys());

‎packages/beacon-node/src/chain/stateCache/stateContextCache.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ export class StateContextCache implements BlockStateCache {
3333
this.maxStates = maxStates;
3434
this.cache = new MapTracker(metrics?.stateCache);
3535
if (metrics) {
36-
this.metrics = metrics.stateCache;
36+
this.metrics = {...metrics.stateCache, ...metrics.epochCache};
3737
metrics.stateCache.size.addCollect(() => metrics.stateCache.size.set(this.cache.size));
3838
}
3939
}
@@ -127,6 +127,10 @@ export class StateContextCache implements BlockStateCache {
127127
}));
128128
}
129129

130+
getStates(): IterableIterator<CachedBeaconStateAllForks> {
131+
return this.cache.values();
132+
}
133+
130134
private deleteAllEpochItems(epoch: Epoch): void {
131135
for (const rootHex of this.epochIndex.get(epoch) || []) {
132136
this.cache.delete(rootHex);

‎packages/beacon-node/src/chain/stateCache/stateContextCheckpointsCache.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,10 @@ export class CheckpointStateCache implements CheckpointStateCacheInterface {
167167
}));
168168
}
169169

170+
getStates(): IterableIterator<CachedBeaconStateAllForks> {
171+
return this.cache.values();
172+
}
173+
170174
/** ONLY FOR DEBUGGING PURPOSES. For spec tests on error */
171175
dumpCheckpointKeys(): string[] {
172176
return Array.from(this.cache.keys());

‎packages/beacon-node/src/chain/stateCache/types.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ export interface BlockStateCache {
2828
prune(headStateRootHex: RootHex): void;
2929
deleteAllBeforeEpoch(finalizedEpoch: Epoch): void;
3030
dumpSummary(): routes.lodestar.StateCacheItem[];
31+
getStates(): IterableIterator<CachedBeaconStateAllForks>; // Expose beacon states stored in cache. Use with caution
3132
}
3233

3334
/**
@@ -65,6 +66,7 @@ export interface CheckpointStateCache {
6566
processState(blockRootHex: RootHex, state: CachedBeaconStateAllForks): Promise<number>;
6667
clear(): void;
6768
dumpSummary(): routes.lodestar.StateCacheItem[];
69+
getStates(): IterableIterator<CachedBeaconStateAllForks>; // Expose beacon states stored in cache. Use with caution
6870
}
6971

7072
export enum CacheItemType {

‎packages/beacon-node/src/eth1/eth1DepositDataTracker.ts‎

Lines changed: 28 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,11 @@
11
import {phase0, ssz} from "@lodestar/types";
22
import {ChainForkConfig} from "@lodestar/config";
3-
import {BeaconStateAllForks, becomesNewEth1Data} from "@lodestar/state-transition";
3+
import {
4+
BeaconStateAllForks,
5+
CachedBeaconStateAllForks,
6+
CachedBeaconStateElectra,
7+
becomesNewEth1Data,
8+
} from "@lodestar/state-transition";
49
import {ErrorAborted, TimeoutError, fromHex, Logger, isErrorAborted, sleep} from "@lodestar/utils";
510

611
import {IBeaconDb} from "../db/index.js";
@@ -67,6 +72,8 @@ export class Eth1DepositDataTracker {
6772
/** Dynamically adjusted batch size to fetch deposit logs */
6873
private eth1GetLogsBatchSizeDynamic = MAX_BLOCKS_PER_LOG_QUERY;
6974
private readonly forcedEth1DataVote: phase0.Eth1Data | null;
75+
/** To stop `runAutoUpdate()` in addition to AbortSignal */
76+
private stopPolling: boolean;
7077

7178
constructor(
7279
opts: Eth1Options,
@@ -81,6 +88,8 @@ export class Eth1DepositDataTracker {
8188
this.depositsCache = new Eth1DepositsCache(opts, config, db);
8289
this.eth1DataCache = new Eth1DataCache(config, db);
8390
this.eth1FollowDistance = config.ETH1_FOLLOW_DISTANCE;
91+
// TODO Electra: fix scenario where node starts post-Electra and `stopPolling` will always be false
92+
this.stopPolling = false;
8493

8594
this.forcedEth1DataVote = opts.forcedEth1DataVote
8695
? ssz.phase0.Eth1Data.deserialize(fromHex(opts.forcedEth1DataVote))
@@ -109,10 +118,22 @@ export class Eth1DepositDataTracker {
109118
}
110119
}
111120

121+
// TODO Electra: Figure out how an elegant way to stop eth1data polling
122+
stopPollingEth1Data(): void {
123+
this.stopPolling = true;
124+
}
125+
112126
/**
113127
* Return eth1Data and deposits ready for block production for a given state
114128
*/
115-
async getEth1DataAndDeposits(state: BeaconStateAllForks): Promise<Eth1DataAndDeposits> {
129+
async getEth1DataAndDeposits(state: CachedBeaconStateAllForks): Promise<Eth1DataAndDeposits> {
130+
if (
131+
state.epochCtx.isAfterElectra() &&
132+
state.eth1DepositIndex >= (state as CachedBeaconStateElectra).depositReceiptsStartIndex
133+
) {
134+
// No need to poll eth1Data since Electra deprecates the mechanism after depositReceiptsStartIndex is reached
135+
return {eth1Data: state.eth1Data, deposits: []};
136+
}
116137
const eth1Data = this.forcedEth1DataVote ?? (await this.getEth1Data(state));
117138
const deposits = await this.getDeposits(state, eth1Data);
118139
return {eth1Data, deposits};
@@ -141,7 +162,10 @@ export class Eth1DepositDataTracker {
141162
* Returns deposits to be included for a given state and eth1Data vote.
142163
* Requires internal caches to be updated regularly to return good results
143164
*/
144-
private async getDeposits(state: BeaconStateAllForks, eth1DataVote: phase0.Eth1Data): Promise<phase0.Deposit[]> {
165+
private async getDeposits(
166+
state: CachedBeaconStateAllForks,
167+
eth1DataVote: phase0.Eth1Data
168+
): Promise<phase0.Deposit[]> {
145169
// No new deposits have to be included, continue
146170
if (eth1DataVote.depositCount === state.eth1DepositIndex) {
147171
return [];
@@ -162,7 +186,7 @@ export class Eth1DepositDataTracker {
162186
private async runAutoUpdate(): Promise<void> {
163187
let lastRunMs = 0;
164188

165-
while (!this.signal.aborted) {
189+
while (!this.signal.aborted && !this.stopPolling) {
166190
lastRunMs = Date.now();
167191

168192
try {

‎packages/beacon-node/src/eth1/index.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,10 @@ export class Eth1ForBlockProduction implements IEth1ForBlockProduction {
106106
startPollingMergeBlock(): void {
107107
return this.eth1MergeBlockTracker.startPollingMergeBlock();
108108
}
109+
110+
stopPollingEth1Data(): void {
111+
return this.eth1DepositDataTracker?.stopPollingEth1Data();
112+
}
109113
}
110114

111115
/**
@@ -140,4 +144,8 @@ export class Eth1ForBlockProductionDisabled implements IEth1ForBlockProduction {
140144
startPollingMergeBlock(): void {
141145
// Ignore
142146
}
147+
148+
stopPollingEth1Data(): void {
149+
// Ignore
150+
}
143151
}

‎packages/beacon-node/src/eth1/interface.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,11 @@ export interface IEth1ForBlockProduction {
6262
* - head state not isMergeTransitionComplete
6363
*/
6464
startPollingMergeBlock(): void;
65+
66+
/**
67+
* Should stop polling eth1Data after a Electra block is finalized AND deposit_receipts_start_index is reached
68+
*/
69+
stopPollingEth1Data(): void;
6570
}
6671

6772
/** Different Eth1Block from phase0.Eth1Block with blockHash */

0 commit comments

Comments
 (0)