'StorageData:: Unable to decode storage system.events:: ..' after runtime version update

[paperfrog95]

Hello, We are coin wallet developer, and Of course, Polkadot exchange is being offered to customers.

Description

Our environment

We are operating three sidecars on a separate server.

• Sidecar001
• Sidecar002
• Sidecar003

Bug report

The bug occurred after the runtime version was updated.
Only sidecar003 has caused problems since the version was updated.

Error occured info

• Network : mainnet (Polkadot)
• Polkadot node version : v0.9.36
• Sidecar version : v14.2.3
• BlockNumber : 14188817 (Spec version 9340)
• Error extrinsic

...
  "era": {
        "immortalEra": "0x00"
      },
      "events": [],
      "success": "Unable to fetch Events, cannot confirm extrinsic status. Check pruning settings on the node.",
      "paysFee": false
    },
...

We're definitely using pruning mode, which keeps only 100,000 blocks. ( option --pruning=100000)
However, the above error occurs only in Sidecar003 . It works normally on Sidecar001 and 002. (use the same pruning mode!)

Ours guess and log

• Sidecar001 , 002 logs

2023-02-10 16:10:56 [info]: API/INIT: Runtime version updated to spec=9360, tx=19

• Sidecar003 log!

2023-02-10 16:10:56 [info]: API/INIT: Runtime version updated to spec=9360, tx=19
2023-02-10 16:11:37 [error]: RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: Unable to decode storage system.events:: createType(Vec<FrameSystemEventRecord>):: decodeU8aVec: failed at 0x0002000000070bd70300001baa453966… (index 31/953): {"phase":"FrameSystemPhase","event":"Event","topics":"Vec<H256>"}:: decodeU8aStruct: failed at 0x57ac09cc00000200000005071baa4539… on topics (index 3/3): Vec<H256>:: Compact input is > Number.MAX_SAFE_INTEGER

Looking at this log, I think there was a conflict in Sidecar003 during the runtime upgrade.

Steps to Reproduce

Maybe, impossible..

I think it will be fixed if Sidecar is finished and restarted. But we're staying the same for disability identification.
Of course, the normal extrinsic is still not viewed only in Sidecar003.

Is this PR related to the problem?
I need to bring my boss the possibility of a recurrence and the solution in the solution.

I look forward to your kind cooperation.

[TarikGul]

Hey @paperfrog95,

First, I tried I reproduce the issue given the same specs as above (well almost). I have a fully synced archive node for polkadot on v9360, and set sidecar the v14.2.3. Unfortunately there was no reproduction. You are right in the sense that these errors can be really difficult to debug for many reasons, especially when a full restart solves the issue.

NOTE: There is one caveat that if you are using sidecar to query a currently syncing node (not in idle syncing), we have noticed some unknown weird behavior can occur.

Is this #1205 related to the problem?

No, so this PR has to do with the /pallets/* endpoint and type safety. It doesn't affect the blocks endpoint. To be sure as well, the query that is giving you issues is /blocks/{blockId}?

There is one constant that I would say avoids these issues. Staying up to date with packages and deps. We stay up to date with polkadot-js pretty religiously. This ensures we are up to date with the substrate types that may be added or updated. It also ensures we are compatible with the latest runtime.

I know it can be tough especially in Infra to ensure things are always updated because that is not realistic in all cases, when downtime needs to be at a minimum for services.

I do think there is a decent hint here in your logs:

2023-02-10 16:10:56 [info]: API/INIT: Runtime version updated to spec=9360, tx=19
2023-02-10 16:11:37 [error]: RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: Unable to decode storage system.events:: createType(Vec<FrameSystemEventRecord>):: decodeU8aVec: failed at 0x0002000000070bd70300001baa453966… (index 31/953): {"phase":"FrameSystemPhase","event":"Event","topics":"Vec<H256>"}:: decodeU8aStruct: failed at 0x57ac09cc00000200000005071baa4539… on topics (index 3/3): Vec<H256>:: Compact input is > Number.MAX_SAFE_INTEGER

I can't promise it will be quick, but I can investigate the actual course of the server along the change of a runtime, and how the types can be affected :).