Fix weights for backed candidates #6929

eskimor · 2023-03-21T13:35:22Z

In particular take into account ump and hrmp messages.

Update:

This does not work. Benchmarks are all over the place. This PR would result in a massive overestimate of the actual weight. In reality there is not much computation done based on these parameters, it is mostly block size that is affected - which is not measured by weight.

In particular the constant term gets massively over estimated and the linear terms also fluctuate a lot from run to run.

Anyhow, given that the constant term changes by a factor of 41 based on the upper bound of those linear parameters, some significant effect on weight seems to be there regardless - it likely is not linear though and the current benchmarking system has trouble providing meaningful/useful results.

For now limiting candidates based on size (to be done) will need to suffice.

Can hopefully be revived once paritytech/polkadot-sdk#194 is done.
With this in place as well we can also properly track size via v2 weights.

Another Update and Plan forward

Benchmarks are still bonkers, but I managed to get acceptable values (not too crazy high for the base case). Constant factors are mostly ignored now for CPU time, this should be acceptable if we limit by size.

Plan: Get size limiting, then merge this.

eskimor · 2023-03-21T13:45:21Z

bot bench $ runtime kusama runtime_parachains::paras_inherent

command-bot · 2023-03-21T13:45:27Z

@eskimor "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" runtime kusama runtime_parachains::paras_inherent (https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/2562026) was cancelled in #6929 (comment)

eskimor · 2023-03-21T13:46:22Z

bot cancel

command-bot · 2023-03-21T13:46:39Z

@eskimor Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" runtime kusama runtime_parachains::paras_inherent has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/2562026 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/2562026/artifacts/download.

eskimor · 2023-03-22T10:07:13Z

bot bench-all $ polkadot-all kusama

command-bot · 2023-03-22T10:07:19Z

@eskimor https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/2567452 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench-all/bench-all.sh" polkadot-all kusama. Check out https://gitlab.parity.io/parity/mirrors/polkadot/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 5-3e09b7e0-145e-4c7f-bcd2-77f3d8d25eb4 to cancel this command or bot cancel to cancel all commands in this pull request.

eskimor · 2023-03-22T10:09:22Z

runtime/parachains/src/builder.rs

@@ -116,7 +123,6 @@ impl<T: paras_inherent::Config> BenchBuilder<T> {
 			dispute_statements: BTreeMap::new(),
 			dispute_sessions: Default::default(),
 			backed_and_concluding_cores: Default::default(),
-			code_upgrade: None,


Is now part of the candidates themselves.

eskimor · 2023-03-22T10:10:03Z

runtime/parachains/src/builder.rs

@@ -221,8 +220,9 @@ impl<T: paras_inherent::Config> BenchBuilder<T> {

 	/// Get the minimum number of validity votes in order for a backed candidate to be included.
 	#[cfg(feature = "runtime-benchmarks")]
-	pub(crate) fn fallback_min_validity_votes() -> u32 {
-		(Self::fallback_max_validators() / 2) + 1
+	pub(crate) fn fallback_min_backing_votes() -> u32 {


Changed name, because backing votes is less ambiguous than validity_votes.

eskimor · 2023-03-22T10:11:24Z

runtime/parachains/src/paras_inherent/mod.rs

@@ -895,7 +895,7 @@ fn apply_weight_limit<T: Config + inclusion::Config>(
 		let (acc_candidate_weight, indices) =
 			random_sel::<BackedCandidate<<T as frame_system::Config>::Hash>, _>(
 				rng,
-				candidates.clone(),
+				&candidates,


Not sure why this was cloned. Can be quite expensive for candidates.

command-bot · 2023-03-22T10:15:27Z

@eskimor Command "$PIPELINE_SCRIPTS_DIR/commands/bench-all/bench-all.sh" polkadot-all kusama has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/2567452 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/2567452/artifacts/download.

runtime/parachains/src/builder.rs

runtime/parachains/src/paras_inherent/weights.rs

runtime/parachains/src/builder.rs

eskimor · 2023-03-24T17:09:30Z

bot bench $ runtime kusama runtime_parachains::paras_inherent

command-bot · 2023-03-24T17:09:35Z

"$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" runtime kusama runtime_parachains::paras_inherent was queued.

Comment bot cancel 41-8ede39f0-18e3-413d-ab7d-c0f1b5966091 to cancel this command or bot cancel to cancel all commands in this pull request.

command-bot · 2023-03-24T17:09:41Z

@eskimor Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" runtime kusama runtime_parachains::paras_inherent has finished. Result:

ValidationError: "id" is required
ValidationError: "id" is required
{"message":{"base":["Reference not found"]}}

…ains::paras_inherent

command-bot · 2023-04-25T20:02:39Z

@eskimor Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" runtime kusama runtime_parachains::paras_inherent has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/2728288 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/2728288/artifacts/download.

eskimor · 2023-04-26T10:18:51Z