fix: claimed in staking info endpoint #1445

Imod7 · 2024-06-04T12:05:58Z

Description

This PR restores the field claimed (under staking) in the staking-info endpoint's response which was previously replaced by legacyClaimedRewards. The root cause of this is explained here.

Suggested Fix

The suggested solution includes :

Adding an IStakingLedger interface to have more control on the response fields under staking.
Retrieving the claimed eras from :
- 1. stakingLedger.legacyClaimedRewards OR stakingLedger.claimedRewards (depending on what call is available every time)
- 1. AND staking.claimedRewards (if available)

Checks needed per Type of Account

Validator Account

If the account queried in the endpoint is a validator account, the following logic is implemented:

A validator gets two types of reward, one for their own stake/contribution, and one for the commission. The total commission is not paid out all at once but in parts/per page. So for each page, part of the commission is paid out to the validator. The total commission is paid out when all pages are claimed.
- The "reward of their own stake" + "reward from commission for 1st page" is claimed when page 0 is claimed.
- For the rest of the pages (when there are more than 1 page), only "reward from commission for that page" is paid out. The reward for their own stake is disregarded (relevant code).
The payout for the validator is done here.
So, for a validator's rewards we have 3 potential statuses (with the new calls) :
- claimed if all pages of erasStakersPaged were claimed.
- unclaimed if no pages were claimed.
- partially claimed if some of the pages were claimed.

Nominator Account

Logic implemented

1st Check

In early blocks/eras there is lastReward instead of claimedRewards under stakingLedger. In this case, we set as claimed the era mentioned in the lastReward so:

if lastReward == 552
- then we return in the response claimedRewards: {era: 552, status : claimed
This is tested in the existing Kusama historical test (after it was updated accordingly) for:
- block height : 1500000 and
- stash account : HP8qJ8P4u4W2QgsJ8jzVuSsjfFTT6orQomFD6eTRSGEbiTK
Before in the response we were returning lastReward instead of claimedRewards. This has been changed so that we always return claimedRewards. Depending on which call is available, values (era & status) in claimedRewards are updated according to the logic described in this PR.

2nd Check

If stakingLedger.legacyClaimedRewards call is available, retrieve claimed information from that call
If not, retrieve claimed info from stakingLedger.claimedRewards
The resulting output is an array of all eras claimed up to that block height which is then transformed into the final output format which is an array of objects of type {era: eraNumber, status : claimed | unclaimed | partially claimed}
*** Note : the 2 calls mentioned above are per specific block height and stash account.

3rd Check

Then, independently of the previous check, check also:

If query.staking.claimedRewards call is available
- If yes, then retrieve the claimed information from that call but only for the eras that we are still missing and complete the missing info
- If the call is not available, no additional check is performed beyond the previous check
  *** Note : this call is per specific block height, era and stash account.

More specifically :

We calculate with the depth and current_era the eras we are missing
Then for each era:
- If staking.erasStakersOverview.pageCount == query.staking.claimedRewards -> then we set the queried era as claimed
- If staking.erasStakersOverview.pageCount != query.staking.claimedRewards. length -> we set the era as partially claimed
- If overview == null && erasStakers > 0 -> this means that pageCount = 1
  - so then it depends again on the query.staking.claimedRewards value to see if era claimed or unclaimed.
The resulting output is the same as in the 1st check, e.g. { "era": "6453", "status": "claimed" },
*** Note : the output from query.staking.claimedRewards is of format [0] if only one page was claimed or [0, 1] if 2 pages were claimed depending on how many pages the stakers of the specific validator are split into (shown from staking.erasStakersOverview.pageCount).

Different Cases

Case `staking.erasStakersOverview.pageCount` == `staking.claimedRewards.length`

This is the case when

Validator has
- 1 or more pages of nominators (erasStakersOverview -> pageCount) per era and
- these pages are equal to the length of the contents found in claimedRewards array

Example

Account DteShXKaQQy2un2VizKwwhViN5e7F47UrkAZDkxgK22LdBv - subscan
Era 6577 in Kusama chain
at block = 23032300
block hash = 0xf9362e71ed123c3a057b75bce389a4c0758ad405556125fee00529569a433898
pjs apps
- staking.claimedRewards = [0]
- staking.erasStakersOverview.pageCount = 1

Sidecar http://127.0.0.1:8080/accounts/DteShXKaQQy2un2VizKwwhViN5e7F47UrkAZDkxgK22LdBv/staking-info?at=23032300
- Era 6577 - Claimed

Case Handled

This case is handled by setting queried era as claimed.

Case `staking.erasStakersOverview.pageCount` != `staking.claimedRewards.length`

This is the case when a validator has multiple pages of nominators and only some of these pages were claimed.

Example

Account 11VR4pF6c7kfBhfmuwwjWY3FodeYBKWx7ix2rsRCU2q6hqJ
Era 1470 in Polkadot chain
pjs apps
- era 1468
- at block = 21157800
- block hash = 0x59de258cf9999635c866df7bc5f397d152892827f887d3629344cb3cebba134f
claimedRewards = [0]
erasStakersOverview = 2
Sidecar http://127.0.0.1:8080/accounts/11VR4pF6c7kfBhfmuwwjWY3FodeYBKWx7ix2rsRCU2q6hqJ/staking-info?at=21157800
- Era 1468 - Partially Claimed

Case handled

This case is handled currently by setting queried era as partially claimed.

Case `staking.erasStakersOverview` = `null` & `staking.claimedRewards` = `[0]`

This is a case where staking.erasStakersOverview is null so the information about the pageCount (in how many pages the stakers are divided into) needs to be retrieved from erasStakers and then compare it with staking.claimedRewards.

Example

Account F2VckTExmProzJnwNaN3YVqDoBPS1LyNVmyG8HUAygtDV3T
pjs apps
- era 6513 in Kusama chain
- at block = 22939322
- block hash = 0x1ef674fffb042c9016987e0e3995a36401a7e2b66e0b6c0bb111a0b049857098
claimedRewards = [0]
erasStakersOverview = null

Sidecar http://127.0.0.1:8080/accounts/F2VckTExmProzJnwNaN3YVqDoBPS1LyNVmyG8HUAygtDV3T/staking-info?at=22939322
- Era 6513 - Claimed

Case handled

This case is handled by checking if erasStakers > 0 (hence pageCount == 1) && staking.claimedRewards is equal to [0] (hence length is 1) -> which is true for era 6513 so era is set to claimed.

Case `staking.erasStakersOverview` = `null` & `erasStakers` = 0

Based on this comment, exposure is not found so we cannot conclude in a status for this specific era. An example is the below request in which only 48 eras are returned since for the rest 36 -> staking.erasStakersOverview = null & erasStakers = 0.

Response (old code)

http://127.0.0.1:8080/accounts/CmjHFdR59QAZMuyjDF5Sn4mwTgGbKMH2cErUFuf6UT51UwS/staking-info?at=22869643

{
  "at": {
    ...
    ...
  "staking": {
    ...
    ...
    "legacyClaimedRewards": [
      "6462",
      "6463",
      "6467",
      "6468",
      "6469",
      "6470",
      "6471",
      "6472",
      "6473",
      "6474",
      "6495",
      "6496",
      "6497",
      "6498",
      "6499",
      "6503",
      "6504",
      "6505",
      "6506",
      "6507",
      "6508",
      "6509",
      "6510",
      "6511",
      "6512"
    ]
  }
}

Response (new code)

{
  "at": {
    "hash": "0xa43364b7a138ec47bd80f09e480a1599f622405add19c6c0913624ab0bb0a96e",
    "height": "22869643"
    ...
    ...
  "staking": {
    ...
    ...
    "claimedRewards": [
      {
        "era": "6462",
        "status": "claimed"
      },
      {
        "era": "6463",
        "status": "claimed"
      },
      {
        "era": "6467",
        "status": "claimed"
      },
      {
        "era": "6468",
        "status": "claimed"
      },
      {
        "era": "6469",
        "status": "claimed"
      },
      {
        "era": "6470",
        "status": "claimed"
      },
      {
        "era": "6471",
        "status": "claimed"
      },
      {
        "era": "6472",
        "status": "claimed"
      },
      {
        "era": "6473",
        "status": "claimed"
      },
      {
        "era": "6474",
        "status": "claimed"
      },
      {
        "era": "6495",
        "status": "claimed"
      },
      {
        "era": "6496",
        "status": "claimed"
      },
      {
        "era": "6497",
        "status": "claimed"
      },
      {
        "era": "6498",
        "status": "claimed"
      },
      {
        "era": "6499",
        "status": "claimed"
      },
      {
        "era": "6503",
        "status": "claimed"
      },
      {
        "era": "6504",
        "status": "claimed"
      },
      {
        "era": "6505",
        "status": "claimed"
      },
      {
        "era": "6506",
        "status": "claimed"
      },
      {
        "era": "6507",
        "status": "claimed"
      },
      {
        "era": "6508",
        "status": "claimed"
      },
      {
        "era": "6509",
        "status": "claimed"
      },
      {
        "era": "6510",
        "status": "claimed"
      },
      {
        "era": "6511",
        "status": "claimed"
      },
      {
        "era": "6512",
        "status": "claimed"
      },
      {
        "era": "6513",
        "status": "claimed"
      },
      {
        "era": "6514",
        "status": "claimed"
      },
      {
        "era": "6515",
        "status": "claimed"
      },
      {
        "era": "6516",
        "status": "claimed"
      },
      {
        "era": "6517",
        "status": "claimed"
      },
      {
        "era": "6518",
        "status": "claimed"
      },
      {
        "era": "6519",
        "status": "claimed"
      },
      {
        "era": "6520",
        "status": "claimed"
      },
      {
        "era": "6521",
        "status": "claimed"
      },
      {
        "era": "6522",
        "status": "claimed"
      },
      {
        "era": "6523",
        "status": "claimed"
      },
      {
        "era": "6524",
        "status": "claimed"
      },
      {
        "era": "6525",
        "status": "claimed"
      },
      {
        "era": "6526",
        "status": "claimed"
      },
      {
        "era": "6527",
        "status": "claimed"
      },
      {
        "era": "6528",
        "status": "unclaimed"
      },
      {
        "era": "6529",
        "status": "unclaimed"
      },
      {
        "era": "6530",
        "status": "unclaimed"
      },
      {
        "era": "6531",
        "status": "unclaimed"
      },
      {
        "era": "6532",
        "status": "unclaimed"
      },
      {
        "era": "6533",
        "status": "unclaimed"
      },
      {
        "era": "6534",
        "status": "unclaimed"
      },
      {
        "era": "6535",
        "status": "unclaimed"
      }
   ]
  }
}

Testing

Tested for account and blocks that include the eras before and after the migration (when the logic changed).

🗒️ Implementation Notes

‼️ The response time of the endpoint is significantly slower now.
- Possible Solution : Add a query parameter as flag to enable or not the claimed rewards. It can default to true so that it does not break the current functionality of the endpoint. Implementation in a separate PR.
I use current era (and not active era) to calculate the depth.
The total of eras returned are equal to the current depth. Depending on the calls available it is either :
- current era - depth if only the new calls are available or
- array.length from result of old calls up until depth. So if array.length = 20 and depth = 84, we will check 64 more eras after the last era found in the array (from the old calls).

Todos

Nice to have

A test for the case staking.erasStakersOverview = null & erasStakers = 0 which can be implemented by mocking this request http://127.0.0.1:8080/accounts/CmjHFdR59QAZMuyjDF5Sn4mwTgGbKMH2cErUFuf6UT51UwS/staking-info?at=22869643

Changelog

This PR introduces breaking changes to the staking-info endpoint response by :

removing the returned field lastReward for early eras and adding claimedRewards updated accordingly.
changing the type of the returned claimedRewards field (under staking)
Hence, it should be explicitly mentioned in the corresponding Changelog and Release Notes of the next release.

IkerAlus · 2024-06-07T08:56:47Z

The change is reasonable. Sidecar user does not care of internal renaming changes since the retrieved info has the same meaning.

Is this PR ready for review or are we waiting for the Todos list to be checked?

Imod7 · 2024-06-07T10:07:00Z

The change is reasonable. Sidecar user does not care of internal renaming changes since the retrieved info has the same meaning.

Is this PR ready for review or are we waiting for the Todos list to be checked?

Yes, ideally the todos needs to be completed before a review. However, if any review will speed things up then please feel free to add them.

Imod7 · 2024-06-13T15:41:57Z

Update
After getting feedback from @Ank4n, for the 2nd check mentioned in the logic section above, I concluded that the logic and thus the implementation in this PR should change to the following:

If staking.erasStakersOverview.pageCount == query.staking.claimedRewards -> era claimed
If staking.erasStakersOverview.pageCount != query.staking.claimedRewards. length -> era partially claimed
If overview == null && erasStakers > 0 -> pageCount = 1 : so then it depends on query.staking.claimedRewards value to see if era claimed or unclaimed

The response will also needs to be changed from a Vec of eras to a Vec of objects of type {era: eraNumber, status : claimed | unclaimed | partially claimed}

Related Resource
polkadot-js/api#5859 (comment)

src/services/accounts/AccountsStakingInfoService.ts

marshacb

LGTM! Just had a question about unwraps

IkerAlus · 2024-06-21T15:55:44Z

docs/src/openapi-v1.yaml

+            for _validators._ This array is populated with values from `stakingLedger.legacyClaimedRewards`
+            or `stakingLedger.claimedRewards`, as well as the `query.staking.claimedRewards` call, depending
+            on whether the queried block is before or after the migration. For more details on the implementation
+            and the migration, refer to the related PR and linked issue.


Where/how can the user find the relevant linked info?

Added specific links of the PR and linked issue. Let me know if this covers your question. I also updated the schema description since I no longer return lastReward. Changes included in this commit.

IkerAlus · 2024-06-21T16:02:58Z

The response time of the endpoint is significantly slower now

Do we have specific data for this statement? How long does it take to return the staking info in the any worst case scenario (for example, 84 eras in different pages and/or with different logic)?

Imod7 · 2024-06-24T07:52:07Z

The response time of the endpoint is significantly slower now

Do we have specific data for this statement? How long does it take to return the staking info in the any worst case scenario (for example, 84 eras in different pages and/or with different logic)?

You are right, I didn't have any formal data when I wrote that. It was based on my observations from the tests I did and also on my knowledge of the calculations of each implementation.
After adding some simple code to measure the execution time of fetchAccountStakingInfo, it looks like there are cases with similar runtimes (old code vs new code) and others have significant differences. However, these results can be influenced by factors such as the connection and the machine used. I am including below some results but to back this statement I need to run proper benchmarks.

http://127.0.0.1:8080/accounts/GpyTMuLmG3ADWRxhZpHQh5rqMgNpFoNUyxA1DJAXfvsQ2Ly/staking-info?at=16869643 - kusama
- old code : 608 ms
- new code : 1.612 ms
http://127.0.0.1:8080/accounts/CmjHFdR59QAZMuyjDF5Sn4mwTgGbKMH2cErUFuf6UT51UwS/staking-info?at=22869643 - kusama
- old code : 372 ms
- new code : 5.312 ms [this is an edge case- goes super slow]
http://127.0.0.1:8080/accounts/11VR4pF6c7kfBhfmuwwjWY3FodeYBKWx7ix2rsRCU2q6hqJ/staking-info?at=18157800 - polkadot
- old code : 433 ms
- new code : 408 ms

Imod7 · 2024-06-26T10:58:57Z

@IkerAlus After our last convo on whether the status partially claimed is valid or not and some more feedback from the staking master @Ank4n :

For a validator to get all commission, they should claim reward for all pages of an era.

Here is the next round of conclusions about claimed:

When a user queries the staking-info endpoint with a validator address and at a specific era&block
- The possible status is still : claimed, partially claimed or unclaimed
When a user queries a nominator address and at a specific era&block
- @IkerAlus you were right, the possible status can only be : claimed or unclaimed

Currently, I am also considering that in both scenarios, it is useful to add the undefined status which is the case when we only have lastReward available (early eras). In such cases, we cannot determine if the queried era was claimed unless it matches the era noted in lastReward. If the queried era does not correspond to the lastReward era then claimed should be set as undefined.

- removed `partially claimed` value - added `undefined` value - updated tests

- bringing back `partially claimed` for validator

fix: claimed in staking info endpoint

c65010a

Imod7 requested a review from a team as a code owner June 4, 2024 12:05

fix existing tests

50943e1

IkerAlus requested a review from bee344 June 4, 2024 15:21

Imod7 added 2 commits June 7, 2024 21:09

add test for kusama (includes era 6508)

82457aa

updated docs

9012e24

Imod7 mentioned this pull request Jun 13, 2024

Use pending_rewards for claimed info in related endpoints #1452

Open

Imod7 added 6 commits June 14, 2024 20:06

implement new logic & update test responses

f320c08

add test for partially claimed & overview = null

2aa5996

updated docs

de06c6b

output up until 84 eras & update tests

8c47eda

merge master, update historical tests & code for lastReward

56e3738

Remove yarn.lock files from PR

9e838e8

marshacb reviewed Jun 20, 2024

View reviewed changes

src/services/accounts/AccountsStakingInfoService.ts Outdated Show resolved Hide resolved

marshacb reviewed Jun 20, 2024

View reviewed changes

src/services/accounts/AccountsStakingInfoService.ts Outdated Show resolved Hide resolved

marshacb approved these changes Jun 20, 2024

View reviewed changes

Imod7 added 2 commits June 21, 2024 10:34

check before unwrapping & fix types

753f3ee

run linter

7ac0236

IkerAlus reviewed Jun 21, 2024

View reviewed changes

docs: added links & updated schema description

171e10a

Imod7 mentioned this pull request Jun 24, 2024

fix: staking payouts claimed with new logic & statuses #1457

Open

1 task

Imod7 added 3 commits August 5, 2024 18:41

updated logic of claimed field for validator account

88be07e

- removed `partially claimed` value - added `undefined` value - updated tests

fix for claimed in case of validator account

cd66735

- bringing back `partially claimed` for validator

moved validator logic in separate function

00a54a0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: claimed in staking info endpoint #1445

fix: claimed in staking info endpoint #1445

Imod7 commented Jun 4, 2024 •

edited

Loading

IkerAlus commented Jun 7, 2024

Imod7 commented Jun 7, 2024

Imod7 commented Jun 13, 2024 •

edited

Loading

marshacb left a comment

IkerAlus Jun 21, 2024 •

edited

Loading

Imod7 Jun 24, 2024

IkerAlus commented Jun 21, 2024

Imod7 commented Jun 24, 2024

Imod7 commented Jun 26, 2024

fix: claimed in staking info endpoint #1445

Are you sure you want to change the base?

fix: claimed in staking info endpoint #1445

Conversation

Imod7 commented Jun 4, 2024 • edited Loading

Description

Suggested Fix

Checks needed per Type of Account

Validator Account

Nominator Account

Logic implemented

1st Check

2nd Check

3rd Check

Different Cases

Case staking.erasStakersOverview.pageCount == staking.claimedRewards.length

Example

Case Handled

Case staking.erasStakersOverview.pageCount != staking.claimedRewards.length

Example

Case handled

Case staking.erasStakersOverview = null & staking.claimedRewards = [0]

Example

Case handled

Case staking.erasStakersOverview = null & erasStakers = 0

Response (old code)

Response (new code)

Testing

🗒️ Implementation Notes

Todos

Nice to have

Changelog

IkerAlus commented Jun 7, 2024

Imod7 commented Jun 7, 2024

Imod7 commented Jun 13, 2024 • edited Loading

marshacb left a comment

Choose a reason for hiding this comment

IkerAlus Jun 21, 2024 • edited Loading

Choose a reason for hiding this comment

Imod7 Jun 24, 2024

Choose a reason for hiding this comment

IkerAlus commented Jun 21, 2024

Imod7 commented Jun 24, 2024

Imod7 commented Jun 26, 2024

Imod7 commented Jun 4, 2024 •

edited

Loading

Case `staking.erasStakersOverview.pageCount` == `staking.claimedRewards.length`

Case `staking.erasStakersOverview.pageCount` != `staking.claimedRewards.length`

Case `staking.erasStakersOverview` = `null` & `staking.claimedRewards` = `[0]`

Case `staking.erasStakersOverview` = `null` & `erasStakers` = 0

Imod7 commented Jun 13, 2024 •

edited

Loading

IkerAlus Jun 21, 2024 •

edited

Loading