archive: Implement archive_unstable_storage #1846
Conversation
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
lexnv
added
A0-needs_burnin
|
|
||
| let mut storage_results = Vec::with_capacity(items.len()); | ||
| for item in items { | ||
| if !is_key_queryable(&item.key.0) { |
There was a problem hiding this comment.
I am not exactly sure about the semantics we want to have here. If the key is not queryable should it be counted as discarded item?
There was a problem hiding this comment.
Indeed this is a bit left to interpretation, I had to revisit the spec to get a better idea of it:
discardedItems is an integer indicating the number of items at the back of the array of the items parameters that couldn't be processed.
I'll add a small PR to clarify this behavior, both in chainHead and archive
There was a problem hiding this comment.
I've removed the concept of non-querieable keys, as per comment: paritytech/json-rpc-interface-spec#118 (comment)
| } | ||
| } | ||
|
|
||
| let discarded_items = items.len().saturating_sub(self.storage_max_queried_items); |
There was a problem hiding this comment.
Is self.storage_max_queried_items part of the spec? What is the value we use in production for this?
There was a problem hiding this comment.
As far as I can understand the spec, there's no explicit mention of an upper limit before the pagination kicks in. This is by design, since smoldot / other implementation might be running under hardware constraints.
For chainHead_storage we are using 5 items, but we haven't collect yet any data to make a better informed decision:
The archive class is not exposed by substrate yet, but would keep both classes in sync
There was a problem hiding this comment.
It would be to good to add a prometheus metric for that or some annoying log for us to know whether this limit is exceeded often or not ^^
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
|
We have merged: paritytech/json-rpc-interface-spec#94 into the spec, which backports the Would love to get some reviews on this 🙏 One other thing that is in flight wrt this PR is: paritytech/json-rpc-interface-spec#118, since we have not enabled the |
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
This PR implements the `archive_unstable_storage` method that offers support for: - fetching values - fetching hashes - iterating over keys and values - iterating over keys and hashes - fetching merkle values from the trie-db A common component dedicated to RPC-V2 storage queries is created to bridge the gap between `chainHead/storage` and `archive/storage`. Query pagination is supported by `paginationStartKey`, similar to the old APIs. Similarly to the `chainHead/storage`, the `archive/storage` method accepts a maximum number of queried items. The design builds upon: paritytech/json-rpc-interface-spec#94. Closes paritytech#1512. cc @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
This PR implements the
archive_unstable_storagemethod that offers support for:A common component dedicated to RPC-V2 storage queries is created to bridge the gap between
chainHead/storageandarchive/storage.Query pagination is supported by
paginationStartKey, similar to the old APIs.Similarly to the
chainHead/storage, thearchive/storagemethod accepts a maximum number of queried items.The design builds upon: paritytech/json-rpc-interface-spec#94.
Closes #1512.
cc @paritytech/subxt-team