Use skiplists to save on the bitmap size when compressing leaves #454

gballet · 2024-09-11T10:58:39Z

Instead of using a bitmap, use two bytes:

the first byte will indicate the length of the range of empty leaves
the second byte will indicate the length of the next range of non-empty leaves

There are ~200M accounts in the state, for which we use at most 6 bytes instead of 32 for a bitmap. This is expected to save (32 - 6) * 200 ~ 5GB

There are further optimizations that will not be handled by this PR, but that I'm writing down here so that I don't forget:

left-trim leaves
only store the C1/C2 if necessary (I need to find a bit to flag which ones are not present)

This PR also contains some fixes for the EoA encoding, that got broken after the Nyota costs.

gballet · 2024-09-11T10:59:56Z

tree.go

@@ -170,6 +170,7 @@ const (
 	leafType       byte = 2
 	eoAccountType  byte = 3
 	singleSlotType byte = 4
+	skipListType   byte = 8


this is set in bit order, because I'm hoping to mix encodings in the future.

eoAccountType has value 3 which isn't respecting that idea. Is it worth changing now? If not, we're ina a half-baked idea.

we could change it to 4, no long-term db has been using this.

gballet · 2024-09-11T11:00:32Z

tree.go

 				// Code hash should be the empty code hash
 				isEoA = v != nil && bytes.Equal(v, EmptyCodeHash[:])
-			case 4:
-				// Code size must be 0
-				isEoA = v != nil && bytes.Equal(v, zero32[:])
 			default:


This is the EoA fix for Nyota.

gballet · 2024-09-11T11:02:12Z

proof_test.go

I believe that the changes in this PR are due to me building on top of two of your commits, which I need for my replay experiments.

gballet · 2024-09-11T16:23:32Z

proof_ipa.go

+			k[StemSize] = ins.Suffix
+			keys = append(keys, k[:])
+			prevalues = append(prevalues, nil)
+			postvalues = append(postvalues, slices.Clone(ins.New[:]))


This, and line 410, are the fixes for the test that was broken.

Weird, so this means that stemdiff is somewhat mutated after affecting the returned Proof?

the area that is used to store the location of the iterator is reused over and over, and if we pass a reference to it, it will change as the iterator progress. This is not the first time we fall for this, and probably not the last time either, it's quite subtle and easy to forget.

jsign

LGTM, left some comments for your consideration.

jsign · 2024-09-11T16:42:32Z

proof_ipa.go

+			k[StemSize] = ins.Suffix
+			keys = append(keys, k[:])
+			prevalues = append(prevalues, nil)
+			postvalues = append(postvalues, slices.Clone(ins.New[:]))


Weird, so this means that stemdiff is somewhat mutated after affecting the returned Proof?

jsign · 2024-09-11T16:44:21Z

tree.go

@@ -170,6 +170,7 @@ const (
 	leafType       byte = 2
 	eoAccountType  byte = 3
 	singleSlotType byte = 4
+	skipListType   byte = 8


eoAccountType has value 3 which isn't respecting that idea. Is it worth changing now? If not, we're ina a half-baked idea.

jsign · 2024-09-11T16:53:41Z

tree.go

+		for _, gap := range gaps {
+			if gap.Count == 0 {
+				break // skip the last gap as nothing follows
+			}


Could we do:

for j:=0; j<=gapCount; j++ {

and avoid if L1852 ?

not completely, because if Count == 0 it means that the group ends with a gap, wherease if Count != 0 then the group ends with a range. But yeah, no point in going over the whole list, I'll loop on j.

jsign · 2024-09-11T16:57:50Z

encoding.go

+		rangecount := serialized[offset+1]
+		gapsize := serialized[offset]


Can we rename gapsize and valueIdx to rangeSkip and rangeCount, as to use the same names in the serialization Skip and Count?

Using gap or range worlds can be confusing. Using the same names as the serialization makes easier to understand the both ways of the algorithm.

tree.go

gballet · 2024-09-12T18:48:25Z

tree.go

+			leafIdx += int(gap.Skip)
+			result = append(result, byte(gap.Count))


I'm thinking there can't be a 0-length range, so we should encode Count-1 instead of forcing the wraparound, which might not work on every platform.

Suggested change

leafIdx += int(gap.Skip)

result = append(result, byte(gap.Count))

leafIdx += int(gap.Skip)

// count can be 256, bytes(256) == 0 which is used as a marker for a full leaf

result = append(result, byte(gap.Count))

Signed-off-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>

jsign · 2024-09-13T14:37:34Z

encoding.go

@@ -56,6 +56,9 @@ const (
 	leafC1CommitmentOffset = leafCommitmentOffset + banderwagon.UncompressedSize
 	leafC2CommitmentOffset = leafC1CommitmentOffset + banderwagon.UncompressedSize
 	leafChildrenOffset     = leafC2CommitmentOffset + banderwagon.UncompressedSize
+	leafSkipListCOffset    = nodeTypeSize + StemSize


Nit: leafSkipListC0Offset

jsign · 2024-09-13T14:44:53Z

encoding.go

-			values[valueIdx] = serialized[offset : offset+leafSlotSize]
+
+	// shortcut: the leaf is full and so both values are 0.
+	if serialized[offset] == 0 && serialized[offset+1] == 0 {


Is this worth it? If the node is full, doesn't sound we should serialize with skip list really. Sounds wasteful.

Signed-off-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>

use skiplists to compress leave storage

5444cd1

gballet requested a review from jsign September 11, 2024 10:58

gballet commented Sep 11, 2024

View reviewed changes

fix: copy slice instead of reusing buffer

f40bb5a

gballet commented Sep 11, 2024

View reviewed changes

gballet added this to the verkle-gen-devnet-8 milestone Sep 11, 2024

gballet changed the base branch from master to jsign-type-3 September 11, 2024 16:29

jsign approved these changes Sep 11, 2024

View reviewed changes

gballet added 2 commits September 12, 2024 11:47

some fixes found while replaying

3b15f78

fix: support full leaves

fe8246c

gballet commented Sep 12, 2024

View reviewed changes

tree.go Outdated Show resolved Hide resolved

improve Count comment

ee9413f

gballet commented Sep 12, 2024

View reviewed changes

gballet added 2 commits September 13, 2024 13:51

replay fixes

4deabaa

Signed-off-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>

more replay fixes

a41ce08

Signed-off-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>

jsign reviewed Sep 13, 2024

View reviewed changes

fix: +2 offset in no-gap encoding

12343ee

Signed-off-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use skiplists to save on the bitmap size when compressing leaves #454

Use skiplists to save on the bitmap size when compressing leaves #454

gballet commented Sep 11, 2024 •

edited

Loading

gballet Sep 11, 2024

jsign Sep 11, 2024

gballet Sep 11, 2024

gballet Sep 11, 2024

gballet Sep 11, 2024

gballet Sep 11, 2024

jsign Sep 11, 2024

gballet Sep 11, 2024

jsign left a comment

jsign Sep 11, 2024

jsign Sep 11, 2024

jsign Sep 11, 2024

gballet Sep 12, 2024

jsign Sep 11, 2024

gballet Sep 12, 2024

jsign Sep 13, 2024

jsign Sep 13, 2024

		rangecount := serialized[offset+1]
		gapsize := serialized[offset]

		leafIdx += int(gap.Skip)
		result = append(result, byte(gap.Count))

Use skiplists to save on the bitmap size when compressing leaves #454

Are you sure you want to change the base?

Use skiplists to save on the bitmap size when compressing leaves #454

Conversation

gballet commented Sep 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsign left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gballet commented Sep 11, 2024 •

edited

Loading