fix(`/blocks`): cache extrinsic base weight constants to improve performance #478

TarikGul · 2021-03-17T23:26:26Z

This PR aims to fix a regression that happened when v2.1.2 was released. ie: commit.

Update on benchmarks: After integrating getBlockWeights I made sure to do some benchmarking. I chose to target block 1907499 which has a staking bondExtra tx (Note I am querying a single block here). I ran a 30s Benchmark with 4 threads and 30 concurrent connections. (This was also done on my local Macbook pro which is extremely fast)

Before this PR:                     After:
--------------------- |           |----------------------
Avg Latency    : 1.13s            | Avg Latency   : 92.11ms
Completed Req  : 312              | Completed Req : 7392

To wrap this PR I ran a new set of benchmarks from our standard collection of 39 blocks that are used in Sidecar-Bench. I also ran it inside of our GCP instance, to recreate a standard environment. This benchmark is 2 minute's with 4 threads and 12 concurrent connections (Which is our standard test for Sidecar in GCP).

v4.0.1:                           This PR:
--------------------- |           |----------------------
Avg Latency    : 9.72s            | Avg Latency   : 5.28s
Completed Req  : 140              | Completed Req : 268

…ons for extrinsicBaseweight metadata

src/chains-config/metadata-consts/index.ts

TarikGul · 2021-03-29T14:28:17Z

Benchmarks for the final additions.

/blocks

v4.0.1:                           The latest commits:
--------------------- |           |----------------------
Avg Latency    : 9.72s            | Avg Latency   : 4.48s
Completed Req  : 140              | Completed Req : 329

src/chains-config/metadata-consts/kusamaConsts.ts

src/chains-config/metadata-consts/substrateConsts.ts

src/controllers/blocks/BlocksExtrinsicsController.ts

src/services/blocks/BlocksService.ts

dvdplm · 2021-03-29T14:30:02Z

src/services/blocks/BlocksService.ts

 				const len = block.extrinsics[idx].encodedLength;
 				const weight = weightInfo.weight;

 				const partialFee = calcFee.calc_fee(
 					BigInt(weight.toString()),
 					len,
-					extrinsicBaseWeight.toBigInt()
+					calcFeeExBaseWeight
 				);

 				extrinsics[idx].info = api.createType('RuntimeDispatchInfo', {


@TarikGul ^^^ didn't pan out? Or am I wrong?

src/types/chains-config/ControllerConfig.ts

src/services/blocks/BlocksService.ts

src/test-helpers/metadata/metadata.ts

src/test-helpers/registries/polkadotRegistry.ts

dvdplm · 2021-03-29T16:03:53Z

src/types/util/Option.ts

@@ -0,0 +1 @@
+export type Option<T> = T | null;


So there is no such type in Typescript itself?

Yea it doesn't exist out of the box.

Maybe we should make a PR for Option to TypeScript ;) Native support for this would be very nice.

Co-authored-by: David <dvdplm@gmail.com>

emostov

LMK when its ready for a thorough review - just mostly focused on responding to open comments for this one

src/test-helpers/metadata/polkadotV29Metadata.ts

src/chains-config/metadata-consts/index.ts

src/chains-config/metadata-consts/substrateConsts.ts

src/controllers/blocks/BlocksExtrinsicsController.ts

emostov · 2021-03-29T17:51:08Z

src/types/util/Option.ts

@@ -0,0 +1 @@
+export type Option<T> = T | null;


Maybe we should make a PR for Option to TypeScript ;) Native support for this would be very nice.

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>

emostov · 2021-03-29T19:00:30Z

src/services/blocks/BlocksService.ts

-			extrinsicBaseWeightExists === undefined ||
+			!perByte ||
+			!extrinsicBaseWeightExists ||
+			(this.minCalcFeeRuntime && specVersion < this.minCalcFeeRuntime) ||
 			typeof api.query.transactionPayment?.nextFeeMultiplier?.at !== 'function'


We try and query this on line 491 and with the ? it should just be undefined if it does not exist, so we can should just be able to check multiplier

emostov · 2021-03-29T19:07:09Z

src/services/blocks/BlocksService.ts

+		const metadata = await api.rpc.state.getMetadata(blockHash);
+		const {
+			consts: { system },
+		} = expandMetadata(api.registry, metadata);


We should keep an eye on what registry we use here in the future. There is a chance it may be better to pull the registry off block we get from the rpc response since that should be guranteed to have the exact types for the runtime. For now I think this is fine.

emostov

Everything looks good to me.

I tried making suggestions via the github vscode extension but botched it somewhere along the way and ended up just making a commit: 20feb8e

Mostly just updates to comments, and variable names. But I did take out that giant try catch when we loop through extrinsics. I honestly think if that is catching we are not doing the correct checks prior so my proposal is to remove it and if it becomes a production issue we can address it

dvdplm · 2021-03-29T19:40:21Z

src/services/blocks/BlocksService.ts

@@ -508,7 +503,7 @@ export class BlocksService extends AbstractService {
 			!perByte ||
 			!extrinsicBaseWeightExists ||
 			(this.minCalcFeeRuntime && specVersion < this.minCalcFeeRuntime) ||
-			typeof api.query.transactionPayment?.nextFeeMultiplier?.at !== 'function'
+			!multiplier


dvdplm · 2021-03-29T19:40:35Z

src/test-helpers/metadata/metadata.ts

+import { polkadotMetadataRpcV16 } from './polkadotV16Metadata';
+import { polkadotMetadataRpcV29 } from './polkadotV29Metadata';


dvdplm · 2021-03-29T19:41:09Z

src/types/util/Option.ts

@@ -1 +1,2 @@
-export type Option<T> = T | null;
+// Not to be confused with the polkadot-js `Codec` extending `Option` type.


dvdplm · 2021-03-29T19:41:41Z

src/types/util/Option.ts

@@ -1 +1,2 @@
-export type Option<T> = T | null;
+// Not to be confused with the polkadot-js `Codec` extending `Option` type.
+export type IOption<T> = T | null;


So what is the TS convention for when to use the I prefix?

I thinks its more just a me convention that I have implicitly picked up from reading other TS code - I is really for interface, but I use it both for interface or types to avoid name collisions with classes / functions etc.

So in typescript (this is quite the debate apparently), "I" should refer to interface's only, and when its not there it will refer to an impl. Even though this example doesn't fit those parameters the use of 'I' here is to make sure there is a distinguishing difference as stated in the comment.

dvdplm

lgtm, good job.

emostov · 2021-03-29T19:52:06Z

@lovesh you should take a look at the dock controller configs to make sure they are still valid. Additionally you may be interested in adding the equivalent of this for the dock chains.

lovesh · 2021-04-08T12:50:22Z

@emostov Thanks for the notification. Submitted change at #511

TarikGul added 2 commits March 17, 2021 17:43

feat: metadata-consts folder holding the polkadot and kusama definiti…

1076f6c

…ons for extrinsicBaseweight metadata

feat: types for polkadot kusama definitions

b43d95c

TarikGul added A3 - In Progress PR in progress, not ready for review I7 - Optimization 🚴 Make Sidecar drive faster labels Mar 17, 2021

TarikGul added 2 commits March 17, 2021 19:27

fix: types on return add | Object

4703ea3

lint

eaa8885

emostov reviewed Mar 18, 2021

View reviewed changes

src/chains-config/metadata-consts/index.ts Outdated Show resolved Hide resolved

TarikGul added 19 commits March 17, 2021 21:20

fix: types

8efb3e2

remove object.freeze

8162959

fix: specName specVersion bug fixed, remove runtimeDoesNotMatch

e10c681

fix: update block service to include the new extrinsicBaseWeight logic

d454d80

fix: cleanup types

db7c46a

fix: change keys to distinguish extrinsicBaseWeight

d6ce0a6

fix: appropiate weights to each chains file

67f1cd6

fix: add caching for decorated metadata inside BlocksController

a20a230

add comment to That

9814272

remove consoles

3a3b7d5

fix: refactor Cache to only have necessary block weight data

813d2c3

fix: put cache inside AbstractController, instead of BlocksController

34d94b9

fix: give getExtrinsicByTimepoint access to the cache

8f60f55

remove commented import

42d5177

fix: runtimeDoesNotMatch bug, that => undefined type block

87a8988

feat: add tests

6c02bf2

refactor, and cleanup types, imports, comments

c707121

fix: lint

ad80e8b

remove unnecessary consts

ce63f83

TarikGul requested review from emostov, dvdplm, maciejhirsz and joepetrowski March 22, 2021 13:31

TarikGul added 2 commits March 29, 2021 00:39

remove console

902ffb8

fix: add weight validators and cleanup

52a8c63

dvdplm reviewed Mar 29, 2021

View reviewed changes

create option type

92f5c19

dvdplm reviewed Mar 29, 2021

View reviewed changes

TarikGul and others added 3 commits March 29, 2021 12:10

fix naming

83a3294

Update src/services/blocks/BlocksService.ts

5c3fb1d

Co-authored-by: David <dvdplm@gmail.com>

add westend definitions for calc fee

6da1db6

emostov reviewed Mar 29, 2021

View reviewed changes

TarikGul and others added 2 commits March 29, 2021 14:03

Update src/test-helpers/metadata/polkadotV29Metadata.ts

05b7aca

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>

Update src/chains-config/metadata-consts/index.ts

a0afa29

Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com>

emostov self-requested a review March 29, 2021 18:47

emostov reviewed Mar 29, 2021

View reviewed changes

zeke: polishing review suggestions

20feb8e

emostov self-requested a review March 29, 2021 19:18

emostov approved these changes Mar 29, 2021

View reviewed changes

emostov requested a review from dvdplm March 29, 2021 19:24

emostov changed the title ~~fix: update reading decorated metadata to increase performance~~ fix(/blocks): cache extrinsic base weight constants in order to improve performance Mar 29, 2021

emostov changed the title ~~fix(/blocks): cache extrinsic base weight constants in order to improve performance~~ fix(/blocks): cache extrinsic base weight constants to improve performance Mar 29, 2021

dvdplm reviewed Mar 29, 2021

View reviewed changes

dvdplm approved these changes Mar 29, 2021

View reviewed changes

emostov merged commit 610db42 into master Mar 29, 2021

emostov deleted the fix-metadata-regression branch March 29, 2021 19:48

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(`/blocks`): cache extrinsic base weight constants to improve performance #478

fix(`/blocks`): cache extrinsic base weight constants to improve performance #478

TarikGul commented Mar 17, 2021 •

edited

Loading

TarikGul commented Mar 29, 2021

dvdplm Mar 29, 2021

dvdplm Mar 29, 2021

TarikGul Mar 29, 2021

emostov Mar 29, 2021

emostov left a comment

emostov Mar 29, 2021

emostov Mar 29, 2021 •

edited

Loading

emostov Mar 29, 2021

emostov left a comment

dvdplm Mar 29, 2021

dvdplm Mar 29, 2021

dvdplm Mar 29, 2021

dvdplm Mar 29, 2021

emostov Mar 29, 2021

TarikGul Mar 29, 2021

dvdplm left a comment

emostov commented Mar 29, 2021

lovesh commented Apr 8, 2021

		import { polkadotMetadataRpcV16 } from './polkadotV16Metadata';
		import { polkadotMetadataRpcV29 } from './polkadotV29Metadata';

		@@ -1 +1,2 @@
		export type Option<T> = T \| null;
		// Not to be confused with the polkadot-js `Codec` extending `Option` type.

fix(/blocks): cache extrinsic base weight constants to improve performance #478

fix(/blocks): cache extrinsic base weight constants to improve performance #478

Conversation

TarikGul commented Mar 17, 2021 • edited Loading

TarikGul commented Mar 29, 2021

/blocks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emostov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emostov Mar 29, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emostov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dvdplm left a comment

Choose a reason for hiding this comment

emostov commented Mar 29, 2021

lovesh commented Apr 8, 2021

fix(`/blocks`): cache extrinsic base weight constants to improve performance #478

fix(`/blocks`): cache extrinsic base weight constants to improve performance #478

TarikGul commented Mar 17, 2021 •

edited

Loading

emostov Mar 29, 2021 •

edited

Loading