Merkle Mountain Range & BEEFY integration #2101

tomusdrw · 2020-12-09T16:05:49Z

This is a draft PR of a potential MMR & BEEFY integration with Rococo.

There is a lot of things to polish and finish up here, but opening it up for discussion opportunity and visibility for external teams.

Open questions:

Shall the ETH bridge support Parathreads?
Can we efficiently update the Merkle Tree of ParaHeads?
Shall we lift a particular parachain header digest item to the MMR leaf directly. Currently the bridge needs to include entire ParaHead to prove anything about a parachain. To make these proofs smaller we could create another entry in MmrLeaf and extract a particular digest item from each ParaHead, then merkelize them in addition to headers.

Future Work:

Start BEEFY service
Cache BEEFY merkle tree
Session Keys migration
Westend integration
Update parachains merkle tree efficiently (only changed parachains)

This reverts commit 7f4d41c.

Bump all. Fix missing imports.

tomusdrw · 2021-01-12T13:25:37Z

@bkchr @rphmeier feel free to take a look already, but AFAIU to deploy to Rococo we need to have the session keys migration implemented, right?

The parachain merkle tree construction optimization is something I'd left out for a separate PR, I feel this is not going to be critical for Rococo.

rphmeier · 2021-01-12T23:26:42Z

we need to have the session keys migration implemented, right?

I wrote a Session::update_keys function which makes this pretty easy now, if that helps. I am also open to restarting Rococo after we merge this.

To make these proofs smaller we could create another entry in MmrLeaf and extract a particular digest item from each ParaHead

The parachain headers don't have any particular format. They are just Vec<u8>, so I don't think we can do that, unfortunately.

Can you give some more info on the para-heads merkle tree and what it will be used for?

tomusdrw · 2021-01-14T14:47:16Z

The parachain headers don't have any particular format. They are just Vec, so I don't think we can do that, unfortunately.

Ah, fair enough.

Can you give some more info on the para-heads merkle tree and what it will be used for?

A light client tracking latest MMR root hash, will be able to receive:

A MMR proof of a particular leaf (i.e. one particular relay chain block)
A Merkle Proof of Parachain Header
A parachain-specific proof (for instance storage proof).

Hence a light client with only MMR root hash will be able to efficiently verify any parachain state data if needed.

runtime/common/src/mmr.rs

andresilva

Initial review, mostly just nits about the search-replace on dependencies.

andresilva · 2021-03-30T09:50:22Z

cli/Cargo.toml

@@ -33,7 +33,7 @@ browser-utils = { package = "substrate-browser-utils", git = "https://github.com

 # this crate is used only to enable `trie-memory-tracker` feature
 # see https://github.com/paritytech/substrate/pull/6745
-sp-trie = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
+sp-trie = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }


There are a bunch of changes like this which I assume came from some automated search and replace. Can we revert this?

andresilva · 2021-03-30T09:54:47Z

core-primitives/Cargo.toml

+sp-core = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+sp-std = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+sp-runtime = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }


andresilva · 2021-03-30T09:55:10Z

node/core/approval-voting/Cargo.toml

+sc-client-api = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+sc-keystore = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+sp-consensus-slots = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+sp-blockchain = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+sp-application-crypto = { git = "https://github.com/paritytech/substrate", default-features = false, features = ["full_crypto"] , branch = "master" }
+sp-runtime = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }


andresilva · 2021-03-30T09:55:22Z

node/core/av-store/Cargo.toml

@@ -21,7 +21,7 @@ polkadot-overseer = { path = "../../overseer" }
 polkadot-primitives = { path = "../../../primitives" }
 polkadot-node-primitives = { path = "../../primitives" }

-sc-service = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
+sc-service = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }


andresilva · 2021-03-30T09:55:33Z

node/network/approval-distribution/Cargo.toml

@@ -15,7 +15,7 @@ futures = "0.3.8"
 tracing = "0.1.25"

 [dev-dependencies]
-sp-core = { git = "https://github.com/paritytech/substrate", branch = "master", features = ["std"] }
+sp-core = { git = "https://github.com/paritytech/substrate", features = ["std"] , branch = "master" }


andresilva · 2021-03-30T10:19:13Z

runtime/kusama/Cargo.toml

@@ -48,6 +49,7 @@ pallet-identity = { git = "https://github.com/paritytech/substrate", branch = "m
 pallet-im-online = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
 pallet-indices = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
 pallet-membership = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
+pallet-mmr-primitives = { git = "https://github.com/paritytech/substrate", default-features = false  , branch = "master" }


Missing from std features list below?

andresilva · 2021-03-30T10:19:36Z

runtime/common/Cargo.toml

+sp-api = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+inherents = { package = "sp-inherents", git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+sp-std = { package = "sp-std", git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+sp-io = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+sp-runtime = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+sp-session = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+sp-staking = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }
+sp-core = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }


andresilva · 2021-03-30T10:19:58Z

runtime/common/Cargo.toml

@@ -35,11 +36,14 @@ pallet-offences = { git = "https://github.com/paritytech/substrate", branch = "m
 pallet-transaction-payment = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
 pallet-treasury = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }

+pallet-beefy = { git = "https://github.com/paritytech/grandpa-bridge-gadget", default-features = false , branch = "master" }
+pallet-mmr = { git = "https://github.com/paritytech/substrate", default-features = false , branch = "master" }


Missing from std feature list below?

andresilva · 2021-03-30T10:20:06Z

runtime/common/Cargo.toml

-sp-session = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
-sp-staking = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
-sp-core = { git = "https://github.com/paritytech/substrate", branch = "master", default-features = false }
+beefy-primitives = { git = "https://github.com/paritytech/grandpa-bridge-gadget", default-features = false , branch = "master" }


Missing from std feature list below?

andresilva · 2021-03-30T10:22:00Z

node/service/src/lib.rs

@@ -955,6 +967,19 @@ pub fn new_full<RuntimeApi, Executor>(
 		task_manager.spawn_essential_handle().spawn_blocking("babe", babe);
 	}

+	// Start BEEFY
+	task_manager.spawn_essential_handle().spawn_blocking(


Should we do this only on rococo/westend? Otherwise if BEEFY crashes for some reason the whole node goes down.

tomusdrw · 2021-03-31T14:46:10Z

runtime/rococo/src/lib.rs

+/// Priviledged origin used by propose parachain.
+pub struct PriviledgedOrigin;
+// A wrapper around `babe::CurrentBlockRandomness` that does not return `Option<Random>`.
+pub struct CurrentBlockRandomness;


Suggested change

/// Priviledged origin used by propose parachain.

pub struct PriviledgedOrigin;

// A wrapper around `babe::CurrentBlockRandomness` that does not return `Option<Random>`.

pub struct CurrentBlockRandomness;

/// Priviledged origin used by propose parachain.

pub struct PriviledgedOrigin;

// A wrapper around `babe::CurrentBlockRandomness` that does not return `Option<Random>`.

pub struct CurrentBlockRandomness;

Seems unneeded.

- revert unnecessary changes - add missing /std dependencies - remove unused dependencies

andresilva

I pushed some minor fixes. LGTM.

tomusdrw added 7 commits December 8, 2020 13:37

Switch branch.

7f4d41c

Implement basic MMR leaf.

ccabe2a

Revert "Switch branch."

9b6c57b

This reverts commit 7f4d41c.

Bump substrate.

b2f70b9

Integrate BEEFY.

02a111d

Bump all. Fix missing imports.

Merge branch 'master' into td-mmr

93b17b3

Use beefy pallet to get authorities.

3f266b0

tomusdrw added the A1-onice label Dec 9, 2020

tomusdrw requested a review from andresilva December 9, 2020 16:06

github-actions bot added the A3-in_progress Pull request is in progress. No review needed at this stage. label Dec 9, 2020

paritytech-ci added D2-breaksapi labels Dec 9, 2020

tomusdrw added the B7-runtimenoteworthy label Dec 10, 2020

tomusdrw added 7 commits January 6, 2021 14:30

Merge branch 'master' into td-mmr

7a851ed

Bump BEEFY repo.

f90ca04

Use next authority set instead of the current one.

7645661

Start BEEFY service.

50ae027

Merge branch 'master' into td-mmr

827e9b1

Fix BEEFY start up.

f7b8657

Cache BEEFY authority set.

ac5e220

tomusdrw removed A1-onice A3-in_progress Pull request is in progress. No review needed at this stage. labels Jan 11, 2021

Merge branch 'master' into td-mmr

f62e4f5

tomusdrw added the C1-low PR touches the given topic and has a low impact on builders. label Jan 11, 2021

tomusdrw marked this pull request as ready for review January 12, 2021 12:37

Merge branch 'master' into td-mmr

086a0ae

musnit reviewed Jan 17, 2021

View reviewed changes

runtime/common/src/mmr.rs Show resolved Hide resolved

adoerr added the D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit. label Mar 24, 2021

adoerr added 3 commits March 24, 2021 06:42

resolve merge issue

5ef2183

merge master

616f0ca

fix pallet indices

b8b3653

tomusdrw added D2-notlive 💤 PR contains changes in a runtime directory that is not deployed to a chain that requires an audit. and removed D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit. labels Mar 25, 2021

adoerr and others added 4 commits March 25, 2021 17:47

fix Westend OldSessionKey

c7b87fc

remove unused imports in Westend runtime

8f15405

Fix compilation for Westend.

eb87063

Merge branch 'master' into td-mmr

566b151

andresilva suggested changes Mar 30, 2021

View reviewed changes

adoerr added 3 commits March 30, 2021 15:07

address review

be4e076

start BEEFY gadget conditionally

55f4c8b

start BEEFY gadget conditionally

18e5c33

adoerr requested a review from andresilva March 30, 2021 16:10

adoerr added 7 commits March 31, 2021 15:15

address review again

468ca36

fix typo

8e0f976

merge master

02b068e

remove duplicate

6fa8609

remove another duplicate

f7e09e7

well

a3c93ed

add missing stuff

7001d5a

tomusdrw commented Mar 31, 2021

View reviewed changes

adoerr and others added 4 commits March 31, 2021 20:05

Merge master for availability recovery fix

7e681b4

cleanup Cargo.toml files

c39e469

- revert unnecessary changes - add missing /std dependencies - remove unused dependencies

runtime: remove unused structs from rococo runtime

f414e23

node: cleanup service

67da771

andresilva approved these changes Mar 31, 2021

View reviewed changes

tomusdrw merged commit e805045 into master Apr 1, 2021

tomusdrw deleted the td-mmr branch April 1, 2021 10:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merkle Mountain Range & BEEFY integration #2101

Merkle Mountain Range & BEEFY integration #2101

tomusdrw commented Dec 9, 2020 •

edited

Loading

tomusdrw commented Jan 12, 2021

rphmeier commented Jan 12, 2021 •

edited

Loading

tomusdrw commented Jan 14, 2021 •

edited

Loading

andresilva left a comment

andresilva Mar 30, 2021

andresilva Mar 30, 2021

andresilva Mar 30, 2021

andresilva Mar 30, 2021

andresilva Mar 30, 2021

andresilva Mar 30, 2021

adoerr Mar 31, 2021

andresilva Mar 30, 2021

andresilva Mar 30, 2021

andresilva Mar 30, 2021

andresilva Mar 30, 2021

tomusdrw Mar 31, 2021

andresilva left a comment

Merkle Mountain Range & BEEFY integration #2101

Merkle Mountain Range & BEEFY integration #2101

Conversation

tomusdrw commented Dec 9, 2020 • edited Loading

tomusdrw commented Jan 12, 2021

rphmeier commented Jan 12, 2021 • edited Loading

tomusdrw commented Jan 14, 2021 • edited Loading

andresilva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andresilva left a comment

Choose a reason for hiding this comment

tomusdrw commented Dec 9, 2020 •

edited

Loading

rphmeier commented Jan 12, 2021 •

edited

Loading

tomusdrw commented Jan 14, 2021 •

edited

Loading