feat: add decoded XCM data in `blocks` endpoint #1364

Imod7 · 2023-12-06T03:12:35Z

Relates to: #732

Description

This PR adds the following 2 query parameters:

decodedXcmMsgs : of type boolean
paraId : of type number

to the /blocks/{blockNumber} endpoint.

Expected Behaviour

When the query param decodedXcmMsgs is set to true, the endpoint is expected to return the XCM messages decoded in a specific block as follows :

If Sidecar is connected to a parachain, then it returns the decoded XCM messages found in :
- downwardMessages
- horizontalMessages
If Sidecar is connected to a relay chain, then it returns the decoded XCM messages found in :
- upwardMessages

When the query param paraId is also set to a specific number/parachain ID, it will only check if there are any XCM messages that are related to that parachain ID. If there are additional XCM messages related to other parachains, they will be ignored and not returned in the response.

Sample Response per Endpoint

Request
http://127.0.0.1:8080/blocks/4777964?decodedXcmMsgs=true

while connected to Kusama Asset Hub

Response
At the end of the usual response from /blocks, the item decodedXcmMsgs is added that looks as follows :

"decodedXcmMsgs": {
    "horizontalMessages": [],
    "downwardMessages": [
      {
        "sentAt": "18510973",
        "msg": "0x03100204000100000700b2383d410a13000100000700b2383d41000d010204000101009ec43214602db413abd2dd38bf27fc7fc76be1715f3a2a53e0c15f0be434a323",
        "data": [
          {
            "v3": [
              {
                "receiveTeleportedAsset": [
                  {
                    "id": {
                      "concrete": {
                        "parents": "1",
                        "interior": {
                          "here": null
                        }
                      }
                    },
                    "fun": {
                      "fungible": "280200000000"
                    }
                  }
    ...
    ...
    ],
    "upwardMessages": []
  }

The object decodedXcmMsgs will always return 3 arrays (horizontalMessages, downwardMessages, upwardMessages`), one for each direction and depending on the XCM message it will fill the corresponding array(s). The rest it will leave empty.

Request
http://127.0.0.1:8080/blocks/18468942?decodedXcmMsgs=true&paraId=2000

while connected to Polkadot Relay chain, it will return the XCM message related to parachain = 2000. In this specific block, there is also another XCM message (for parachain 2012) that will not appear.

"decodedXcmMsgs": {
    "horizontalMessages": [],
    "downwardMessages": [],
    "upwardMessages": [
      {
        "paraId": "2000",
        "data": {
          "v3": [
            {
              "withdrawAsset": [
                {
                  "id": {
                    "concrete": {
                      "parents": "0",
                      "interior": {
                        "here": null
                      }
                    }
                  },
                  "fun": {
                    "fungible": "1545324000000"
                  }
    ...
    ...
    ]
  }

Testing

This script was created to test multilple blocks from different relay chains and parachains and check the corresponding decoded XCM messages that are returned.

Notes

To be able to decode horizontal messages when connected to Moonbeam, you need to make the changes that are shown in this commit. In the current PR these changes were removed since Sidecar should be as generic as possible and not include changes that are parachain specific.

Todos

Credits

Many credits to :

@IkerAlus and @TarikGul for the research on the different XCM messages, different directions and where we should look for what (so that I can retrieve them in this PR).
@TarikGul for crafting the design and structure of the XcmDecoder class and all the necessary properties and methods it should have.
@marshacb for the savior commit where he added of all the types necessary to fix the lint errors.

- rebuilt docs - removed moonbeam specific code from xcm decoder

- updated docs in blocks controller comments

- in block 18468942 we have 2 upward msgs from two different parachains so with the same mocked data we can add 2 tests - added a test to check the query param paraId is working as expected

- in block 3356195 we have downward and horizontal msgs so with the same mocked data we can test two different directions

- changed structure of the decodedMsgs response so that it always returns three arrays (filled or empty), one for each direction. - removed an if statement as unnecessary in XCMDecoder - updated corresponding tests with the new structure of the response

…onnected to parachain

…izontal msgs

docs/src/openapi-v1.yaml

src/controllers/blocks/BlocksController.ts

src/services/blocks/XCMDecoder.ts

src/services/test-helpers/mock/mockAssetHubKusamaApiBlock3356195.ts

src/types/responses/SanitizedBackedCandidate.ts

TarikGul

Overall, really amazing stuff. There are some key changes needed in the syntax for the XCMDecoder class. The docs need to have a the decoded messages in the response section, and there are a few more nits here and there.

Really amazing feature, really excited to get this in.

- the response is a combination of Block and BlockXCM response with the use of allOf - added schema for Liquidity Pool because the swagger was complaining - changed the order of a hash field so that it aligns of it actually appears in the corresponding response

- changed type of paraId after validation - renamed boolean option for decodedXcmMsgs so it is shorter

- changes in imports - replaced class name with the keyword 'this' - replaced static with readonly for class properties - replaced static with private for class methods - changed specname to lowercase

- added the 'type' keyword in more imports

src/services/blocks/BlocksService.ts

TarikGul · 2024-01-23T14:08:55Z

src/services/test-helpers/mock/mockAssetHubKusamaApi.ts

-						reserved: null,
-					},
-				},
+				perClass: Promise.resolve().then(() => {


This doesn't need to be a promise since its a const.

perClass: defaultMockApi.consts.system.blockWeights.perClass

Should suffice.

TarikGul · 2024-01-23T14:10:21Z

src/services/test-helpers/mock/mockAssetHubKusamaApiBlock3356195.ts

+			blockWeights: {
+				baseBlock: new BN(5481991000),
+				maxBlock: assetHubKusamaRegistryV1000000.createType('u64', 15),
+				perClass: Promise.resolve().then(() => {


Same with this, this doesn't need to be a promise. It can just be:

perClass: defaultMockApi.consts.system.blockWeights.perClass

TarikGul · 2024-01-23T14:10:54Z

src/services/test-helpers/mock/mockAssetHubKusamaApiBlock6202603.ts

+			blockWeights: {
+				baseBlock: new BN(5481991000),
+				maxBlock: assetHubKusamaRegistryV1000000b.createType('u64', 15),
+				perClass: Promise.resolve().then(() => {


This can also just be:

perClass: defaultMockApi.consts.system.blockWeights.perClass

TarikGul · 2024-01-23T14:11:43Z

src/services/test-helpers/mock/mockAssetHubWestendApi.ts

-						reserved: null,
-					},
-				},
+				perClass: Promise.resolve().then(() => {


This can also just be:

perClass: defaultMockApi.consts.system.blockWeights.perClass

TarikGul

Few small nits, but once those are wrapped up the PR should be good to merge, Great Job!

Co-authored-by: Tarik Gul <47201679+TarikGul@users.noreply.github.com>

TarikGul

LGTM 🚀

feat: add decoded xcm data in blocks endpoint

394e889

Imod7 requested a review from a team as a code owner December 6, 2023 03:12

marshacb and others added 21 commits December 6, 2023 14:01

add types to fix any lint errors

2f937f3

update copyright year

5945988

fix paraId filter for horizontal msgs

5f26396

fix xcm decoding for moonbeam horizontal msg

8647bf9

add yarn.lock file

34efc92

update docs

85fa529

removed moonbeam package

b8e28e8

merged master & resolved conflicts

68afb73

- rebuilt docs - removed moonbeam specific code from xcm decoder

adding back yarn.lock

9dae51f

changed variable to const

236d0fd

retrieved yarn.lock from master branch

7339cab

added a test for decoding upward msg in polkadot block

23cad30

added test for decoding horizontal msg in KAH block

1f7f478

Merged master, resolved conflicts & rebuilt docs

2090b46

updated cacheKey with query params of decodedXcmMsgs and paraId

6ad6e16

- updated docs in blocks controller comments

replaced mocked data of block 18207445 with those of block 18468942

3e9982d

- in block 18468942 we have 2 upward msgs from two different parachains so with the same mocked data we can add 2 tests - added a test to check the query param paraId is working as expected

replaced mocked data of block 5831776 with those of block 3356195

ba354f7

- in block 3356195 we have downward and horizontal msgs so with the same mocked data we can test two different directions

reintroduced the if statement so that paraId query param works when c…

bd35e99

…onnected to parachain

added test to check if query param paraId works correctly also in hor…

3b037d4

…izontal msgs

run linter

d49db0e