In-memory tree states #5533

michaelsproul · 2024-04-08T06:21:50Z

Issue Addressed

This is the memory-only portion of tree-states, i.e. without any database schema changes.

Upgrade in-memory and on-disk state representation with tree states #3206

Proposed Changes

Make BeaconState cheap to clone thanks to persistent data structures from milhouse.
Replace the snapshot cache with a new state cache inside the store which holds 32x more states for around the same memory cost 🎉
Delete the now-unnecessary block_production_state.
Delete StateProcessingStrategy. This could cause us to do more tree hashing in the case of an attester shuffling cache miss, but this is mitigated by 1) tree-states has faster tree hashing due to structural sharing, 2) the state cache is likely to hit even when the shuffling cache misses. It is less risky to remove the concept of inconsistent states altogether so that they can't end up in the new state_cache and poison it. We can re-optimise this codepath in future by adopting the HotStateSummary changes from full tree-states.

michaelsproul · 2024-04-23T05:50:04Z

I pushed a commit 970f3df which seems to have made memory usage less spikey on a restart (see 12:30 in chart). The problem previously was that the head state would not get cached in the state_cache on startup, creating the possibility for a cache-miss and reload from disk, which would increase memory usage due to the lack of full structural sharing.

The memory usage is still a bit spikier than I'd like, and this is despite a complete absence of cache misses. This may be memory usage unrelated to the state cache.

Even so, tree-states is using consistently less memory with lower spikes than stable:

The yellow line at the bottom is this branch, the rest are running variants of stable/unstable.

realbigsean

🚀

realbigsean · 2024-04-23T23:47:22Z

@mergify queue

mergify · 2024-04-23T23:47:28Z

queue

🛑 The pull request has been removed from the queue `default`

Pull request #5533 has been dequeued by a dequeue command.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

jimmygchen · 2024-04-24T00:02:48Z

beacon_node/beacon_chain/src/beacon_chain.rs

@@ -4529,10 +4404,11 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
                let block = self
                    .get_blinded_block(&parent_block_root)?
                    .ok_or(Error::MissingBeaconBlock(parent_block_root))?;
-                let state = self
-                    .get_state(&block.state_root(), Some(block.slot()))?
+                let (state_root, state) = self


I think the log above needs to be updated to remove snapshot cache:

lighthouse/beacon_node/beacon_chain/src/beacon_chain.rs

Lines 4398 to 4403 in 970f3df

info!(

self.log,

"Missed snapshot cache during withdrawals calculation";

"slot" => proposal_slot,

"parent_block_root" => ?parent_block_root

);

jimmygchen · 2024-04-24T00:10:33Z

Some minor comments:

The description for --state-cache-size needs an update i think - replace the "snapshot cache" and also update the default to 128.
Remove snapshot cache related metrics here

realbigsean · 2024-04-24T00:13:45Z

@mergify dequeue

mergify · 2024-04-24T00:13:52Z

dequeue

✅ The pull request has been removed from the queue `default`

realbigsean · 2024-04-24T00:14:30Z

sorry @jimmygchen didn't realize you were mid-review!

jimmygchen · 2024-04-24T00:22:30Z

All good, happy to merge this as it's gone through some testing and you've already done a round of review!
I'm just going through the changes to help me understand them, it'd probably take me a while. I haven't found anything major that's worth stopping the mege, I think we're good to merge and address further comments in a separate PR?

I'd be keen to get this in so

we avoid further code conflicts
others working on unstable can help discover potential issues not yet identified

realbigsean · 2024-04-24T00:22:51Z

@jimmygchen has expressed offline he's good with merging and allowing feedback resolution in a followup PR so as to avoid conflicts

realbigsean · 2024-04-24T00:22:58Z

@mergify queue

mergify · 2024-04-24T00:23:02Z

queue

🛑 The pull request has been removed from the queue `default`

Pull request #5533 has been dequeued by a dequeue command.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

realbigsean · 2024-04-24T00:23:17Z

@mergify requeue

mergify · 2024-04-24T00:23:48Z

requeue

✅ This pull request will be re-embarked automatically

The followup queue command will be automatically executed to re-embark the pull request

mergify · 2024-04-24T00:23:49Z

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 6196289

jimmygchen · 2024-04-24T04:18:59Z

beacon_node/beacon_chain/src/metrics.rs

-    }
+    set_gauge_by_usize(
+        &BLOCK_PROCESSING_SNAPSHOT_CACHE_SIZE,
+        beacon_chain.store.state_cache_len(),


Can we rename the metrics to BLOCK_PROCESSING_STATE_CACHE_SIZE instead?

michaelsproul added 18 commits April 8, 2024 11:45

Consensus changes

1b4620f

EF tests

bba555d

lcli

9d3ff45

common and watch

1f5516a

account manager

6b5a37c

cargo

67a881d

fork choice

19f3020

promise cache

083cf64

beacon chain

429db72

interop genesis

eff12d7

http api

7e66ce1

lighthouse

c34ccab

op pool

0adb24f

beacon chain misc

3d09bf5

parallel state cache

a197ffb

store

4139cbe

fix issues in store

2d551c4

IT COMPILES

8fca797

michaelsproul added work-in-progress PR is a work-in-progress optimization Something to make Lighthouse run more efficiently. tree-states Upcoming state and database overhaul labels Apr 8, 2024

michaelsproul mentioned this pull request Apr 8, 2024

Unify and lower state caches #5313

Open

Remove some unnecessary module qualification

2b3b71a

dapplion mentioned this pull request Apr 9, 2024

Revert Arced pubkey optimization #5536

Merged

dapplion and others added 6 commits April 11, 2024 14:13

Revert Arced pubkey optimization (#5536)

8fda723

Merge remote-tracking branch 'origin/unstable' into tree-states-memory

de7ae83

Fix caching, rebasing and some tests

cf341b6

Remove unused deps

38b0765

Merge remote-tracking branch 'origin/unstable' into tree-states-memory

7592355

Small cleanups

f4285e5

Cache state after get_advanced_hot_state

970f3df

michaelsproul mentioned this pull request Apr 23, 2024

lighthouse takes a long time to process the getHeader block request. As a result, a block miss #5626

Closed

michaelsproul added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Apr 23, 2024

realbigsean approved these changes Apr 23, 2024

View reviewed changes

mergify bot added a commit that referenced this pull request Apr 23, 2024

Merge of #5533

e0bd72b

mergify bot mentioned this pull request Apr 23, 2024

merge queue: embarking unstable (4cad1fc) and #5533 together #5637

Closed

5 tasks

jimmygchen reviewed Apr 24, 2024

View reviewed changes

mergify bot added a commit that referenced this pull request Apr 24, 2024

Merge of #5533

0923806

mergify bot mentioned this pull request Apr 24, 2024

merge queue: embarking unstable (4cad1fc) and #5533 together #5639

Closed

5 tasks

mergify bot merged commit 6196289 into unstable Apr 24, 2024
27 checks passed

mergify bot deleted the tree-states-memory branch April 24, 2024 01:22

jimmygchen reviewed Apr 24, 2024

View reviewed changes

jimmygchen mentioned this pull request Apr 30, 2024

Remove snapshot cache related code #5661

Merged

michaelsproul mentioned this pull request May 13, 2024

Fix hot state disk leak #5768

Merged

michaelsproul mentioned this pull request Jul 8, 2024

Delete cached_tree_hash #6060

Merged

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

In-memory tree states #5533

In-memory tree states #5533

michaelsproul commented Apr 8, 2024 •

edited

Loading

michaelsproul commented Apr 23, 2024

realbigsean left a comment

realbigsean commented Apr 23, 2024

mergify bot commented Apr 23, 2024 •

edited

Loading

jimmygchen Apr 24, 2024

jimmygchen commented Apr 24, 2024

realbigsean commented Apr 24, 2024

mergify bot commented Apr 24, 2024

realbigsean commented Apr 24, 2024

jimmygchen commented Apr 24, 2024

realbigsean commented Apr 24, 2024

realbigsean commented Apr 24, 2024

mergify bot commented Apr 24, 2024

realbigsean commented Apr 24, 2024

mergify bot commented Apr 24, 2024

mergify bot commented Apr 24, 2024 •

edited

Loading

jimmygchen Apr 24, 2024

	info!(
	self.log,
	"Missed snapshot cache during withdrawals calculation";
	"slot" => proposal_slot,
	"parent_block_root" => ?parent_block_root
	);

In-memory tree states #5533

In-memory tree states #5533

Conversation

michaelsproul commented Apr 8, 2024 • edited Loading

Issue Addressed

Proposed Changes

michaelsproul commented Apr 23, 2024

realbigsean left a comment

Choose a reason for hiding this comment

realbigsean commented Apr 23, 2024

mergify bot commented Apr 23, 2024 • edited Loading

🛑 The pull request has been removed from the queue default

jimmygchen Apr 24, 2024

Choose a reason for hiding this comment

jimmygchen commented Apr 24, 2024

realbigsean commented Apr 24, 2024

mergify bot commented Apr 24, 2024

✅ The pull request has been removed from the queue default

realbigsean commented Apr 24, 2024

jimmygchen commented Apr 24, 2024

realbigsean commented Apr 24, 2024

realbigsean commented Apr 24, 2024

mergify bot commented Apr 24, 2024

🛑 The pull request has been removed from the queue default

realbigsean commented Apr 24, 2024

mergify bot commented Apr 24, 2024

✅ This pull request will be re-embarked automatically

mergify bot commented Apr 24, 2024 • edited Loading

✅ The pull request has been merged automatically

jimmygchen Apr 24, 2024

Choose a reason for hiding this comment

michaelsproul commented Apr 8, 2024 •

edited

Loading

mergify bot commented Apr 23, 2024 •

edited

Loading

🛑 The pull request has been removed from the queue `default`

✅ The pull request has been removed from the queue `default`

🛑 The pull request has been removed from the queue `default`

mergify bot commented Apr 24, 2024 •

edited

Loading