Integrate litep2p into Polkadot SDK #2944

altonen · 2024-01-16T14:56:11Z

litep2p is a libp2p-compatible P2P networking library. It supports all of the features of rust-libp2p that are currently being utilized by Polkadot SDK.

Compared to rust-libp2p, litep2p has a quite different architecture which is why the new litep2p network backend is only able to use a little of the existing code in sc-network. The design has been mainly influenced by how we'd wish to structure our networking-related code in Polkadot SDK: independent higher-levels protocols directly communicating with the network over links that support bidirectional backpressure. A good example would be NotificationHandle/RequestResponseHandle abstractions which allow, e.g., SyncingEngine to directly communicate with peers to announce/request blocks.

I've tried running polkadot --network-backend litep2p with a few different peer configurations and there is a noticeable reduction in networking CPU usage. For high load (--out-peers 200), networking CPU usage goes down from ~110% to ~30% (80 pp) and for normal load (--out-peers 40), the usage goes down from ~55% to ~18% (37 pp).

These should not be taken as final numbers because:

a) there are still some low-hanging optimization fruits, such as enabling receive window auto-tuning, integrating Peerset more closely with litep2p or improving memory usage of the WebSocket transport
b) fixing bugs/instabilities that incorrectly cause litep2p to do less work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed

Nevertheless, these numbers should give an early estimate for CPU usage of the new networking backend.

This PR consists of three separate changes:

introduce a generic PeerId (wrapper around Multihash) so that we don't have use NetworkService::PeerId in every part of the code that uses a PeerId
introduce NetworkBackend trait, implement it for the libp2p network stack and make Polkadot SDK generic over NetworkBackend
implement NetworkBackend for litep2p

The new library should be considered experimental which is why rust-libp2p will remain as the default option for the time being. This PR currently depends on the master branch of litep2p but I'll cut a new release for the library once all review comments have been addresses.

Users of `sc-network` cannot directly depend on `libp2p::PeerId` as it won't be the only `PeerId` that will be provided. `sc-network` still re-exports `libp2p::PeerId` as `sc_network::PeerId` but any code using `PeerId` should depend on `sc-network-types::PeerId` as it's agnostic over the underlying networking backend implementation while remaining compatible with `libp2p::PeerId`.

Implement `NetworkBackend` for `NetworkWorker`, introduce more traits related to making notification/request-response protocols generic over the underlying networking backend and start using these traits throughout the codebase. `sc_network::NetworkService` is no longer provided as an object but instead as `Arc<dyn NetworkService>` which abstracts the underlying networking backend without introducing generics across the entire codebase. Notification/request-response protocol configurations are generic over the networking backend as libp2p and litep2p have slightly different ways of providing said protocols and thus require custom ways of initializing them. The network backend be selected with `--network-backend` switch

The new backend can be selected with `--network-backend litep2p`

dmitry-markin · 2024-01-16T16:46:50Z

The PR is already nicely split in three commits as per bullet points, so what about publishing it as three separate PRs to make reviewing easier? Does it make sense from the perspective of LOC in every change (i.e., if 90% of changes are in one of three commits, that likely doesn't make sense)?

In the end it's up to you to decide, because rebasing/merging the ladder of three PRs could be quite tedious.

altonen · 2024-01-17T08:32:44Z

Is there a benefit from splitting the code into three PRs compared to reviewing the three commits separately in this PR? The workload for the review is the same as far as I see it.

dmitry-markin · 2024-01-17T09:10:28Z

A possible benefit I had in mind is that when the code is updated (i.e., the review comments are addressed), it would be instantly visible in separate PRs, but the individual commits would stay the same, so all 9700 LOC of changes should be viewed to get the understanding of what the code is after additional commits are pushed.

dmitry-markin · 2024-01-17T09:12:19Z

I'm not pushing for this, as this would come at the cost of rebasing the PRs

altonen · 2024-01-18T07:21:45Z

Do other people have any opinions? Would you like to see this split into multiple PRs or is it ok to review the three commits separately?

alindima · 2024-01-18T07:27:55Z

I don't mind reviewing as a single PR

koute

Hm... I think actually splitting this PR might be a good idea. Not necessarily because of making it easier to review, but simply because that'd probably speed up how fast we can merge it. The PeerId commit is a good example of something that we could just merge almost immediately.

substrate/client/authority-discovery/src/worker.rs

substrate/client/mixnet/Cargo.toml

substrate/client/mixnet/src/peer_id.rs

substrate/client/network/types/Cargo.toml

altonen · 2024-01-18T13:42:16Z

Hm... I think actually splitting this PR might be a good idea. Not necessarily because of making it easier to review, but simply because that'd probably speed up how fast we can merge it. The PeerId commit is a good example of something that we could just merge almost immediately.

I don't mind if the review takes a bit longer because honestly I see now value in getting the first two commits into master while the third one is still in review.

…itep2p

alindima

Outstanding work! 🚀

polkadot/node/network/gossip-support/Cargo.toml

substrate/client/cli/src/arg_enums.rs

substrate/client/mixnet/src/protocol.rs

substrate/client/network/src/litep2p/shim/notification/mod.rs

substrate/client/network/src/litep2p/service.rs

substrate/client/network/src/service/signature.rs

substrate/client/network/src/service/traits.rs

substrate/client/network/types/Cargo.toml

dmitry-markin

Some comments (mostly nits) for the sc_network_types::PeerId commit (the first commit).

substrate/client/network/src/peer_store.rs

substrate/client/network/src/protocol_controller.rs

substrate/client/network/types/src/peer_id.rs

substrate/client/network/transactions/src/lib.rs

substrate/client/network/src/config.rs

substrate/client/network/src/litep2p/shim/notification/peerset.rs

substrate/client/network/src/litep2p/peerstore.rs

substrate/client/network/src/litep2p/discovery.rs

[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P networking library. It supports all of the features of `rust-libp2p` that are currently being utilized by Polkadot SDK. Compared to `rust-libp2p`, `litep2p` has a quite different architecture which is why the new `litep2p` network backend is only able to use a little of the existing code in `sc-network`. The design has been mainly influenced by how we'd wish to structure our networking-related code in Polkadot SDK: independent higher-levels protocols directly communicating with the network over links that support bidirectional backpressure. A good example would be `NotificationHandle`/`RequestResponseHandle` abstractions which allow, e.g., `SyncingEngine` to directly communicate with peers to announce/request blocks. I've tried running `polkadot --network-backend litep2p` with a few different peer configurations and there is a noticeable reduction in networking CPU usage. For high load (`--out-peers 200`), networking CPU usage goes down from ~110% to ~30% (80 pp) and for normal load (`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp). These should not be taken as final numbers because: a) there are still some low-hanging optimization fruits, such as enabling [receive window auto-tuning](libp2p/rust-yamux#176), integrating `Peerset` more closely with `litep2p` or improving memory usage of the WebSocket transport b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less work will increase the networking CPU usage c) verification in a more diverse set of tests/conditions is needed Nevertheless, these numbers should give an early estimate for CPU usage of the new networking backend. This PR consists of three separate changes: * introduce a generic `PeerId` (wrapper around `Multihash`) so that we don't have use `NetworkService::PeerId` in every part of the code that uses a `PeerId` * introduce `NetworkBackend` trait, implement it for the libp2p network stack and make Polkadot SDK generic over `NetworkBackend` * implement `NetworkBackend` for litep2p The new library should be considered experimental which is why `rust-libp2p` will remain as the default option for the time being. This PR currently depends on the master branch of `litep2p` but I'll cut a new release for the library once all review comments have been addresses. --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: Dmitry Markin <dmitry@markin.tech> Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>

Polkadot-Forum · 2024-05-01T07:32:08Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/paritytech-update-for-april/7646/1

kianenigma · 2024-05-15T09:48:26Z

I understand that this is an important change, and a technological achievement indeed (all judging by the number of emoji reaction 🚀), but I want to ask for future similar changes to
be made with possibly less changes to the node side code (service.rs etc), if possible.

Please see #5, https://forum.polkadot.network/t/polkadot-parachain-omni-node-gathering-ideas-and-feedback/7823 and similar :)

eskimor · 2024-05-15T10:00:07Z

@michalkucharczyk good data point for the improvements we want to make. Networking changes should obviously not affect client code, if they are not customizing things.

bkchr · 2024-05-15T11:09:12Z

I had started locally to fix this up, but have given up. The reviewers should not have accepted it in this state, especially for an experimental feature.

) This PR introduces custom types / substrate wrappers for `Multiaddr`, `multiaddr::Protocol`, `Multihash`, `ed25519::*` and supplementary types like errors and iterators. This is needed to unblock `libp2p` upgrade PR #1631 after #2944 was merged. `libp2p` and `litep2p` currently depend on different versions of `multiaddr` crate, and introduction of this "common ground" types is needed to support independent version upgrades of `multiaddr` and dependent crates in `libp2p` & `litep2p`. While being just convenient to not tie versions of `libp2p` & `litep2p` dependencies together, it's currently not even possible to keep `libp2p` & `litep2p` dependencies updated to the same versions as `multiaddr` in `libp2p` depends on `libp2p-identity` that we can't include as a dependency of `litep2p`, which has it's own `PeerId` type. In the future, to keep things updated on `litep2p` side, we will likely need to fork `multiaddr` and make it use `litep2p` `PeerId` as a payload of `/p2p/...` protocol. With these changes, common code in substrate uses these custom types, and `litep2p` & `libp2p` backends use corresponding libraries types.

…ritytech#4198) This PR introduces custom types / substrate wrappers for `Multiaddr`, `multiaddr::Protocol`, `Multihash`, `ed25519::*` and supplementary types like errors and iterators. This is needed to unblock `libp2p` upgrade PR paritytech#1631 after paritytech#2944 was merged. `libp2p` and `litep2p` currently depend on different versions of `multiaddr` crate, and introduction of this "common ground" types is needed to support independent version upgrades of `multiaddr` and dependent crates in `libp2p` & `litep2p`. While being just convenient to not tie versions of `libp2p` & `litep2p` dependencies together, it's currently not even possible to keep `libp2p` & `litep2p` dependencies updated to the same versions as `multiaddr` in `libp2p` depends on `libp2p-identity` that we can't include as a dependency of `litep2p`, which has it's own `PeerId` type. In the future, to keep things updated on `litep2p` side, we will likely need to fork `multiaddr` and make it use `litep2p` `PeerId` as a payload of `/p2p/...` protocol. With these changes, common code in substrate uses these custom types, and `litep2p` & `libp2p` backends use corresponding libraries types.

- Upgrade Polkadot-sdk 1.10.0 to 1.11.0 - Update weights to reflect the new version. Notable Changes: - [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512) - [Genesis Presets](paritytech/polkadot-sdk#2714) - [Integrate LiteP2P](paritytech/polkadot-sdk#2944) - [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files) For more details, please refer to: [Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0) issue-1957

- Upgrade Polkadot-sdk 1.10.0 to 1.11.0 - Update weights to reflect the new version. Notable Changes: - [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512) - [Genesis Presets](paritytech/polkadot-sdk#2714) - [Integrate LiteP2P](paritytech/polkadot-sdk#2944) - [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files) - [Remove try-runtime-cli](https://github.com/paritytech/polkadot-sdk/pull/4017/files#diff-3a3aa5e088741f8ab1ca38fbe314c7a150d18b7466426f4ad2569113017e8439) For more details, please refer to: [Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0) #1957

- Upgrade Polkadot-sdk 1.10.0 to 1.11.0 - Update weights to reflect the new version. Notable Changes: - [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512) - [Genesis Presets](paritytech/polkadot-sdk#2714) - [Integrate LiteP2P](paritytech/polkadot-sdk#2944) - [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files) - [Remove try-runtime-cli](https://github.com/paritytech/polkadot-sdk/pull/4017/files#diff-3a3aa5e088741f8ab1ca38fbe314c7a150d18b7466426f4ad2569113017e8439) For more details, please refer to: [Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0)

- Upgrade Polkadot-sdk 1.10.0 to 1.11.0 - Update weights to reflect the new version. Notable Changes: - [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512) - [Genesis Presets](paritytech/polkadot-sdk#2714) - [Integrate LiteP2P](paritytech/polkadot-sdk#2944) - [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files) - [Remove try-runtime-cli](https://github.com/paritytech/polkadot-sdk/pull/4017/files#diff-3a3aa5e088741f8ab1ca38fbe314c7a150d18b7466426f4ad2569113017e8439) For more details, please refer to: [Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0) #1957

altonen added 3 commits January 16, 2024 16:22

Implement litep2p network backend for Polkadot SDK

39a6428

The new backend can be selected with `--network-backend litep2p`

altonen added the T0-node This PR/Issue is related to the topic “node”. label Jan 16, 2024

altonen requested review from dmitry-markin, alindima, sandreim and a team January 16, 2024 14:56

altonen requested a review from andresilva as a code owner January 16, 2024 14:56

koute reviewed Jan 18, 2024

View reviewed changes

Fix CI complaints

7472a60

altonen added 2 commits January 18, 2024 16:00

Apply review comments

38f0003

Merge remote-tracking branch 'origin/master' into altonen-integrate-l…

1ad2bdc

…itep2p

alexggh self-requested a review January 22, 2024 08:57

altonen added 3 commits January 22, 2024 11:00

Fix clippy

e6fc19e

more clippy fixes

62aedb3

bind to both ipv4 and ipv6

39dc59c

alindima reviewed Jan 22, 2024

View reviewed changes

dmitry-markin reviewed Jan 22, 2024

View reviewed changes

substrate/client/network/transactions/src/lib.rs Outdated Show resolved Hide resolved

dmitry-markin reviewed Jan 23, 2024

View reviewed changes

substrate/client/network/src/config.rs Outdated Show resolved Hide resolved

dmitry-markin reviewed Jan 23, 2024

View reviewed changes

substrate/client/network/src/config.rs Outdated Show resolved Hide resolved

substrate/client/network/src/litep2p/shim/notification/peerset.rs Outdated Show resolved Hide resolved

dmitry-markin reviewed Jan 23, 2024

View reviewed changes

lexnv added this pull request to the merge queue Apr 8, 2024

Merged via the queue into master with commit 80616f6 Apr 8, 2024
131 of 134 checks passed

lexnv deleted the altonen-integrate-litep2p branch April 8, 2024 17:11

dmitry-markin mentioned this pull request Apr 12, 2024

Ignored block announcement because all validation slots for this peer are occupied #4109

Open

dmitry-markin mentioned this pull request Apr 23, 2024

Replace Multiaddr & related types with substrate-specific types #4198

Merged

kayabaNerve mentioned this pull request May 19, 2024

Move to Polkadot SDK serai-dex/serai#461

Open

ahmadkaouk mentioned this pull request Jun 20, 2024

Upgrade to polkadot-sdk v1.11.0 moonbeam-foundation/moonbeam#2838

Merged

kianenigma mentioned this pull request Jul 30, 2024

network/metrics: Expose number of banned peers from peerstore and enable litep2p metrics #4977

Merged

enddynayn mentioned this pull request Aug 12, 2024

chore(upgrade): v1.10.0 to v1.11.0 frequency-chain/frequency#2117

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate litep2p into Polkadot SDK #2944

Integrate litep2p into Polkadot SDK #2944

altonen commented Jan 16, 2024

dmitry-markin commented Jan 16, 2024

altonen commented Jan 17, 2024

dmitry-markin commented Jan 17, 2024

dmitry-markin commented Jan 17, 2024

altonen commented Jan 18, 2024

alindima commented Jan 18, 2024

koute left a comment

altonen commented Jan 18, 2024

alindima left a comment

dmitry-markin left a comment

Polkadot-Forum commented May 1, 2024

kianenigma commented May 15, 2024 •

edited

Loading

eskimor commented May 15, 2024

bkchr commented May 15, 2024

Integrate litep2p into Polkadot SDK #2944

Integrate litep2p into Polkadot SDK #2944

Conversation

altonen commented Jan 16, 2024

dmitry-markin commented Jan 16, 2024

altonen commented Jan 17, 2024

dmitry-markin commented Jan 17, 2024

dmitry-markin commented Jan 17, 2024

altonen commented Jan 18, 2024

alindima commented Jan 18, 2024

koute left a comment

Choose a reason for hiding this comment

altonen commented Jan 18, 2024

alindima left a comment

Choose a reason for hiding this comment

dmitry-markin left a comment

Choose a reason for hiding this comment

Polkadot-Forum commented May 1, 2024

kianenigma commented May 15, 2024 • edited Loading

eskimor commented May 15, 2024

bkchr commented May 15, 2024

kianenigma commented May 15, 2024 •

edited

Loading