feat: implement BalancesTreeCache

[twoeths]

Motivation

• At every epoch transition, we always create balances tree from scratch and it causes memory spikes at processRewardsAndPenalties step (10m) chart:

• while after a state is finalized, we remove checkpoint state and don't use it anymore. We can recreate the balances tree using that unused balances tree from that state leveraging feat: ListUintNum64Type create ViewDU from existing tree ssz#402

• see also Improve processRewardsAndPenalties #6229

Description

• create BalancesTreeCache which is populated after we remove a checkpoint state, and consume by epoch transition using same design of feat: async shuffling refactor #6938
  • state-transition defines IBalancesTreeCache interface and consume it if exists
  • spec test does not have it
  • prod node provide an implementation of BalancesTreeCache and put that to EpochCache
• use this api feat: ListUintNum64Type create ViewDU from existing tree ssz#402 ListUintNum64Type.toViewDU(value, viewDU)

Resolves #6229

Testing

• processRewardsAndPenalties 10m is improved a lot, there are 2 spikes in 12h time frame, most likely due to batchHashTreeRoot() we used inside ListUintNum64Type.toViewDU(value, viewDU)

• 6h chart shows ~250ms save in average

• but prepareNextEpoch is saved ~350ms because we hashed state.balances in ListUintNum64Type.toViewDU(value, viewDU), and that's majority of hash work for prepareNextEpoch

• state.hashTreeRoot() is not a little bit confusing, but majority of work is done by this work and state.validators.commit()

[twoeths]

a note about BalancesTreeCache, if network is not finalized in an epoch then we'll miss the cache

it's not likely to miss the cache if we go with n-historical flag

with an average time of ~830s afterProcessEpoch

we can expect the prepareNextEpoch on holesky to be around 1.35s - 0.8s ~= 550ms

[twoeths]

works well for a while (more than 2w of testing)