chore: publish metadata as gitlab artifact

[ggera]

fixes KILTProtocol/ticket#2241

Generates metadata and publishes json value to s3 bucket
Sample here https://kilt-protocol.org/peregrine/peregrine-metadata.json

Pending

• SRTool Docker image that includes the new subwasm 0.19 with support for json+scale output format. PR: chore: update subwasm dependency to 0.19 paritytech/srtool#60

[ntn-x2]

As far as I can see this is definitely good. But I cannot get the SDK to use this properly inside the augment-api package. @rflechtner do you have any insights into how we could use this directly instead of pulling the metadata from a running node?

[ntn-x2]

Can we just use the value of "$1" as part of the path?

[ggera]

already added $1 == spiritnet

[ntn-x2]

Probably scope of a different PR, but subwasm also allows to run diffs between two metadata. We can discuss what is the best strategy to execute those checks, but I would say that running it when a PR is merged into master might be a good time.

[ggera]

As far as I can see this is definitely good. But I cannot get the SDK to use this properly inside the augment-api package. @rflechtner do you have any insights into how we could use this directly instead of pulling the metadata from a running node?

This is expected metadata as json https://kilt-protocol.org/peregrine/peregrine-metadata.json

[rflechtner]

As far as I can see this is definitely good. But I cannot get the SDK to use this properly inside the augment-api package. @rflechtner do you have any insights into how we could use this directly instead of pulling the metadata from a running node?

The augment-api script expects the result of the state_getMetadata rpc call, which as far as I can see a JSON object with a result property whose value is the scale encoded metadata in hex encoding.
I don't know what the best way to scale-encode the JSON example Gera posted is; in theory it should work with

import { Metadata, TypeRegistry} from '@polkadot/types';
reg = new TypeRegistry()
meta = new Metadata(reg, json)
meta.toHex()

Doesn't really work for me though rn, but that does seem to do the trick:

import { TypeRegistry } from '@polkadot/types';
meta = new TypeRegistry().createType('MetadataV14', json.V14)
meta.toHex()

[rflechtner]

I assume subwasm also has a binary export format for the metadata if you're passing the --json flag? That could be just what we need. Should we try uploading that?
You could also try updating subwasm; something's definitely off with the JSON export. The interface for the V14 data model for the metadata looks as follows:

export interface MetadataV14 extends Struct {
    readonly lookup: PortableRegistry;
    readonly pallets: Vec<PalletMetadataV14>;
    readonly extrinsic: ExtrinsicMetadataV14;
    readonly type: SiLookupTypeId;
}

But in the JSON file we've got the keys types, pallets, extrinsics, and ty instead.

[ntn-x2]

I have been playing on the rust server with the latest version of subwasm, and I also get a JSON with a single V14 key with properties types, pallets, extrinsic, and ty. I think the difference could be in how the RPC nodes returns this information. There's also such a thing as a magicNumber, which I guess the script expects when querying from the RPC node state_getMetadata. We should bypass this and simply parse a metadata document somehow. I will dedicate a bit more time to that this morning, and see if I make any progress on that front.

[ntn-x2]

Asked this on StackExchange as well: https://substrate.stackexchange.com/questions/6988/how-to-generate-augmented-types-from-v14-metadata-generated-with-subwasm

[ntn-x2]

Another related StackExchange issue: polkadot-js/api#4293

[weichweich]

it might be enough if we have them as an artifact. Not sure if we need to upload them to s3. Gitlab also stores them for a while.

[trusch]

https://github.com/chevdor/subwasm/releases/tag/v0.19.0 contains the needed functionality to output all kinds of metadata formats.

Especially there is --format=json+scale which produces the metadata as polkadotjs expects it and --format=scale which outputs the format that subxt needs.

[ntn-x2]

@trusch so as a quick recap, does --format=json+scale produce a file with the metadata? That we can serve directly as an artifact? Can we output multiple things at the same time? Can we also easily check what changed between two metadata versions?

[trusch]

@ntn-x2 the format flag just specifies the format, but there is now also a --output flag where you can specify a file.

It's not possible to output two different formats at the same time, but running the command is fast enough to do it two times.

The diff subcommand was not changed since the last release.

[ntn-x2]

@trusch thanks for the clarifications. I would say that we can update this PR to export the new format so that the SDK can use it. We will tackle metadata diffs in a separate ticket.

[ntn-x2]

@ggera as far as I know, CI_COMMIT_TAG could be empty if the job was simply triggered by a commit push. Would it make sense to have the same logic we have for Docker image tagging? Or maybe push something like <branch-commit> if a regular push and <tag> if a tag.

[ntn-x2]

I pushed a new change with the new format for subwasm, but we still need to wait for a new SRtool release that includes the new subwasm 0.19.

[ntn-x2]

Tracking PR: paritytech/srtool#60

[ggera]

it might be enough if we have them as an artifact. Not sure if we need to upload them to s3. Gitlab also stores them for a while.

Removed the s3 upload and updated srtool since new image is pushed @ntn-x2 @trusch

[ntn-x2]

The artifacts only include *.wasm outputs. Should also the *.json be added as artifacts? So that we have a URL to pull the metadata from.

[ntn-x2]

Should be good enough! Let's give it a try!