Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Parachain node stalled syncing with error StateDatabase("Too many sibling blocks inserted") #10476

Closed
xlc opened this issue Dec 12, 2021 · 5 comments
Closed

Parachain node stalled syncing with error StateDatabase("Too many sibling blocks inserted") #10476

xlc opened this issue Dec 12, 2021 · 5 comments

Comments

@xlc
Copy link
Contributor

xlc commented Dec 12, 2021 •
edited
Loading

Node stalled syncing with this error

2021-12-12 20:42:47.916  WARN tokio-runtime-worker sync: [Parachain] 💔 Error importing block 0xb084174ffe4172f2557cd1ca2757910d1f87445cc1342c29731b4e9abfb36da2: Err(Other(ClientImport("State Database error: Too many sibling blocks inserted")))
StateDatabase("Too many sibling blocks inserted")

This happened on one of the node for our internal parachain testnet. Also received reports for Karura.

More logs:

2021-12-12 20:43:10.858  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (7 peers), best: #467774 (0xd7a5…c0d1), finalized #467771 (0xd93c…df88), ⬇ 2.4kiB/s ⬆ 1.5kiB/s
2021-12-12 20:43:07.737  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 0.5kiB/s ⬆0.9kiB/s
2021-12-12 20:43:06.217  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467774 (0xd7a5…c0d1)
2021-12-12 20:43:05.857  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (7 peers), best: #467773 (0x2e8f…6c3e), finalized #467771 (0xd93c…df88), ⬇ 1.8kiB/s ⬆ 2.1kiB/s
2021-12-12 20:43:02.736  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 1.6kiB/s ⬆2.4kiB/s
2021-12-12 20:43:00.856  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (7 peers), best: #467773 (0x2e8f…6c3e), finalized #467770 (0x04f7…087b), ⬇ 3.7kiB/s ⬆ 2.5kiB/s
2021-12-12 20:43:00.218  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467773 (0x2e8f…6c3e)
2021-12-12 20:42:57.735  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 0.8kiB/s ⬆0.9kiB/s
2021-12-12 20:42:55.855  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (6 peers), best: #467772 (0x42ae…7578), finalized #467769 (0xc2f7…12b4), ⬇ 2.9kiB/s ⬆ 2.0kiB/s
2021-12-12 20:42:54.219  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467772 (0x42ae…7578)
2021-12-12 20:42:52.734  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 71.0kiB/s ⬆ 1.1kiB/s
2021-12-12 20:42:51.289  INFO tokio-runtime-worker afg: [Relaychain] 👴 Applying GRANDPA set change to new set [(Public(88dc3417d5058ec4b4503e0c12ea1a0a89be200fe98922423d4334014fa6b0ee (5FA9nQDV...)), 1), (Public(d17c2d7823ebf260fd138f2d7e27d114c0145d968b5ff5006125f2414fadae69 (5GoNkf6W...)), 1)]
2021-12-12 20:42:51.289  INFO tokio-runtime-worker afg: [Relaychain] 👴 Applying authority set change scheduled at block #467769
2021-12-12 20:42:50.854  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (6 peers), best: #467771 (0xd93c…df88), finalized #467768 (0x9871…19f8), ⬇ 3.5kiB/s ⬆ 2.5kiB/s
2021-12-12 20:42:48.924  WARN tokio-runtime-worker telemetry: [Parachain] ❌ Error while dialing /dns/telemetry.polkadot.io/tcp/443/x-parity-wss/%2Fsubmit%2F:Custom { kind: Other, error: Timeout }
2021-12-12 20:42:48.924  WARN    async-std/runtime trust_dns_proto::xfer::dns_exchange: failed to associate send_message response to the sender
2021-12-12 20:42:48.215  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467771 (0xd93c…df88)
2021-12-12 20:42:47.916  WARN tokio-runtime-worker sync: [Parachain] 💔 Error importing block 0xb084174ffe4172f2557cd1ca2757910d1f87445cc1342c29731b4e9abfb36da2: Err(Other(ClientImport("State Database error: Too many sibling blocks inserted")))
StateDatabase("Too many sibling blocks inserted")
2021-12-12 20:42:47.915  WARN tokio-runtime-worker sc_service::client::client: [Parachain] Block import error:
2021-12-12 20:42:47.733  INFO tokio-runtime-worker substrate: [Parachain] ⚙️  Syncing  0.0 bps, target=#74000 (2 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 71.3kiB/s ⬆ 2.7kiB/s
2021-12-12 20:42:47.521  WARN tokio-runtime-worker sync: [Parachain] 💔 Error importing block 0xb084174ffe4172f2557cd1ca2757910d1f87445cc1342c29731b4e9abfb36da2: Err(Other(ClientImport("State Database error: Too many sibling blocks inserted")))
StateDatabase("Too many sibling blocks inserted")
2021-12-12 20:42:47.520  WARN tokio-runtime-worker sc_service::client::client: [Parachain] Block import error:
2021-12-12 20:42:45.854  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (6 peers), best: #467770 (0x04f7…087b), finalized #467767 (0x372d…e63a), ⬇ 3.0kiB/s ⬆ 4.3kiB/s
2021-12-12 20:42:42.732  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 141.0kiB/s ⬆ 2.4kiB/s
2021-12-12 20:42:42.224  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467770 (0x04f7…087b)
2021-12-12 20:42:40.853  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (7 peers), best: #467769 (0xc2f7…12b4), finalized #467767 (0x372d…e63a), ⬇15.4kiB/s ⬆ 8.8kiB/s
2021-12-12 20:42:39.018  WARN tokio-runtime-worker sync: [Parachain] 💔 Error importing block 0xb084174ffe4172f2557cd1ca2757910d1f87445cc1342c29731b4e9abfb36da2: Err(Other(ClientImport("State Database error: Too many sibling blocks inserted")))
StateDatabase("Too many sibling blocks inserted")
2021-12-12 20:42:39.018  WARN tokio-runtime-worker sc_service::client::client: [Parachain] Block import error:
2021-12-12 20:42:38.598  WARN tokio-runtime-worker sync: [Parachain] 💔 Error importing block 0xb084174ffe4172f2557cd1ca2757910d1f87445cc1342c29731b4e9abfb36da2: Err(Other(ClientImport("State Database error: Too many sibling blocks inserted")))
StateDatabase("Too many sibling blocks inserted")
2021-12-12 20:42:38.597  WARN tokio-runtime-worker sc_service::client::client: [Parachain] Block import error:
2021-12-12 20:42:37.732  INFO tokio-runtime-worker substrate: [Parachain] ⚙️  Syncing  0.0 bps, target=#74000 (2 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 1.0kiB/s ⬆ 1.6kiB/s
2021-12-12 20:42:36.332  INFO tokio-runtime-worker parachain::approval-voting: [Relaychain] Advanced session window for approvals update=Advanced { prev_window_start: 46793, prev_window_end: 46798, new_window_start: 46794, new_window_end: 46799 }
2021-12-12 20:42:36.314  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467769 (0xc2f7…12b4)
2021-12-12 20:42:36.228  INFO tokio-runtime-worker babe: [Relaychain] 👶 Next epoch starts at slot 273223636
2021-12-12 20:42:36.228  INFO tokio-runtime-worker babe: [Relaychain] 👶 New epoch 46799 launching at block 0xc2f7…12b4 (block slot 273223626 >= start slot 273223626).
2021-12-12 20:42:35.852  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (7 peers), best: #467768 (0x9871…19f8), finalized #467766 (0x2606…18b6), ⬇10.4kiB/s ⬆ 6.8kiB/s
2021-12-12 20:42:32.730  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 1.6kiB/s ⬆2.4kiB/s
2021-12-12 20:42:30.851  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (7 peers), best: #467768 (0x9871…19f8), finalized #467765 (0xd067…6029), ⬇15.9kiB/s ⬆ 7.1kiB/s
2021-12-12 20:42:30.219  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467768 (0x9871…19f8)
2021-12-12 20:42:27.730  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 1.0kiB/s ⬆1.7kiB/s
2021-12-12 20:42:25.850  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (7 peers), best: #467767 (0x372d…e63a), finalized #467764 (0x410b…4c63), ⬇ 4.9kiB/s ⬆ 4.6kiB/s
2021-12-12 20:42:25.673  INFO tokio-runtime-worker sub-libp2p: [Relaychain] 🔍 Discovered new external address for our node: /ip4/192.168.120.128/tcp/30336/ws/p2p/12D3KooWCu7G3ogEQMhcJKqpyp3wMz4h3Px98a7QSE8Z29YfxziK
2021-12-12 20:42:25.673  INFO tokio-runtime-worker sub-libp2p: [Relaychain] 🔍 Discovered new external address for our node: /ip4/192.168.120.128/tcp/30335/p2p/12D3KooWCu7G3ogEQMhcJKqpyp3wMz4h3Px98a7QSE8Z29YfxziK
2021-12-12 20:42:24.222  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467767 (0x372d…e63a)
2021-12-12 20:42:22.729  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 1.4kiB/s ⬆2.0kiB/s
2021-12-12 20:42:20.849  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (6 peers), best: #467766 (0x2606…18b6), finalized #467763 (0xc2e0…9481), ⬇10.0kiB/s ⬆ 3.9kiB/s
2021-12-12 20:42:18.261  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467766 (0x3f6e…db6e)
2021-12-12 20:42:18.224  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467766 (0x2606…18b6)
2021-12-12 20:42:17.728  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 3.0kiB/s ⬆2.6kiB/s
2021-12-12 20:42:15.848  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (6 peers), best: #467765 (0xd067…6029), finalized #467763 (0xc2e0…9481), ⬇ 7.3kiB/s ⬆ 6.3kiB/s
2021-12-12 20:42:12.727  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 141.2kiB/s ⬆ 2.5kiB/s
2021-12-12 20:42:12.219  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467765 (0xd067…6029)
2021-12-12 20:42:10.847  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (6 peers), best: #467764 (0x410b…4c63), finalized #467761 (0xa25e…165c), ⬇ 2.9kiB/s ⬆ 1.6kiB/s
2021-12-12 20:42:09.576  WARN tokio-runtime-worker sync: [Parachain] 💔 Error importing block 0xb084174ffe4172f2557cd1ca2757910d1f87445cc1342c29731b4e9abfb36da2: Err(Other(ClientImport("State Database error: Too many sibling blocks inserted")))
StateDatabase("Too many sibling blocks inserted")
2021-12-12 20:42:09.576  WARN tokio-runtime-worker sc_service::client::client: [Parachain] Block import error:
2021-12-12 20:42:09.216  WARN tokio-runtime-worker sync: [Parachain] 💔 Error importing block 0xb084174ffe4172f2557cd1ca2757910d1f87445cc1342c29731b4e9abfb36da2: Err(Other(ClientImport("State Database error: Too many sibling blocks inserted")))
StateDatabase("Too many sibling blocks inserted")
2021-12-12 20:42:09.216  WARN tokio-runtime-worker sc_service::client::client: [Parachain] Block import error:
2021-12-12 20:42:08.917  WARN tokio-runtime-worker telemetry: [Parachain] ❌ Error while dialing /dns/telemetry.polkadot.io/tcp/443/x-parity-wss/%2Fsubmit%2F:Custom { kind: Other, error: Timeout }
2021-12-12 20:42:07.726  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 1.0kiB/s ⬆1.5kiB/s
2021-12-12 20:42:06.285  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467764 (0x410b…4c63)
2021-12-12 20:42:06.284  INFO tokio-runtime-worker sc_informant: [Relaychain] ♻️  Reorg on #467764,0x2db9…9b7c to #467764,0x410b…4c63, common ancestor #467763,0xc2e0…9481
2021-12-12 20:42:06.233  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467764 (0x2db9…9b7c)
2021-12-12 20:42:05.846  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (6 peers), best: #467763 (0xc2e0…9481), finalized #467761 (0xa25e…165c), ⬇ 1.3kiB/s ⬆ 1.6kiB/s
2021-12-12 20:42:02.725  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 141.0kiB/s ⬆ 2.6kiB/s
2021-12-12 20:42:01.744  WARN tokio-runtime-worker sync: [Parachain] 💔 Error importing block 0xb084174ffe4172f2557cd1ca2757910d1f87445cc1342c29731b4e9abfb36da2: Err(Other(ClientImport("State Database error: Too many sibling blocks inserted")))
StateDatabase("Too many sibling blocks inserted")
2021-12-12 20:42:01.743  WARN tokio-runtime-worker sc_service::client::client: [Parachain] Block import error:
2021-12-12 20:42:01.457  WARN tokio-runtime-worker sync: [Parachain] 💔 Error importing block 0xb084174ffe4172f2557cd1ca2757910d1f87445cc1342c29731b4e9abfb36da2: Err(Other(ClientImport("State Database error: Too many sibling blocks inserted")))
StateDatabase("Too many sibling blocks inserted")
2021-12-12 20:42:01.457  WARN tokio-runtime-worker sc_service::client::client: [Parachain] Block import error:
2021-12-12 20:42:00.845  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (6 peers), best: #467763 (0xc2e0…9481), finalized #467760 (0xea0c…6716), ⬇ 2.0kiB/s ⬆ 1.4kiB/s
2021-12-12 20:42:00.219  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467763 (0xc2e0…9481)
2021-12-12 20:41:57.724  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 141.5kiB/s ⬆ 3.2kiB/s
2021-12-12 20:41:55.845  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (6 peers), best: #467762 (0x4018…2215), finalized #467759 (0xc17a…eacc), ⬇ 2.5kiB/s ⬆ 2.3kiB/s
2021-12-12 20:41:55.269  WARN tokio-runtime-worker sync: [Parachain] 💔 Error importing block 0xb084174ffe4172f2557cd1ca2757910d1f87445cc1342c29731b4e9abfb36da2: Err(Other(ClientImport("State Database error: Too many sibling blocks inserted")))
StateDatabase("Too many sibling blocks inserted")
2021-12-12 20:41:55.269  WARN tokio-runtime-worker sc_service::client::client: [Parachain] Block import error:
2021-12-12 20:41:54.458  WARN tokio-runtime-worker sync: [Parachain] 💔 Error importing block 0xb084174ffe4172f2557cd1ca2757910d1f87445cc1342c29731b4e9abfb36da2: Err(Other(ClientImport("State Database error: Too many sibling blocks inserted")))
StateDatabase("Too many sibling blocks inserted")
2021-12-12 20:41:54.457  WARN tokio-runtime-worker sc_service::client::client: [Parachain] Block import error:
2021-12-12 20:41:54.219  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467762 (0x4018…2215)
2021-12-12 20:41:52.723  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 0.8kiB/s ⬆1.2kiB/s
2021-12-12 20:41:51.229  INFO tokio-runtime-worker afg: [Relaychain] 👴 Applying GRANDPA set change to new set [(Public(88dc3417d5058ec4b4503e0c12ea1a0a89be200fe98922423d4334014fa6b0ee (5FA9nQDV...)), 1), (Public(d17c2d7823ebf260fd138f2d7e27d114c0145d968b5ff5006125f2414fadae69 (5GoNkf6W...)), 1)]
2021-12-12 20:41:51.228  INFO tokio-runtime-worker afg: [Relaychain] 👴 Applying authority set change scheduled at block #467759
2021-12-12 20:41:50.844  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (7 peers), best: #467761 (0xa25e…165c), finalized #467758 (0xfcff…6ba9), ⬇ 2.9kiB/s ⬆ 2.1kiB/s
2021-12-12 20:41:48.211  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467761 (0xa25e…165c)
2021-12-12 20:41:47.722  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 1.4kiB/s ⬆1.6kiB/s
2021-12-12 20:41:45.843  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (7 peers), best: #467760 (0xea0c…6716), finalized #467757 (0x2ab7…b08a), ⬇ 2.2kiB/s ⬆ 2.4kiB/s
2021-12-12 20:41:42.721  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 0.9kiB/s ⬆1.0kiB/s
2021-12-12 20:41:42.222  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467760 (0xea0c…6716)
2021-12-12 20:41:40.842  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (7 peers), best: #467759 (0xc17a…eacc), finalized #467757 (0x2ab7…b08a), ⬇ 3.4kiB/s ⬆ 1.9kiB/s
2021-12-12 20:41:37.720  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 1.5kiB/s ⬆1.9kiB/s
2021-12-12 20:41:36.323  INFO tokio-runtime-worker parachain::approval-voting: [Relaychain] Advanced session window for approvals update=Advanced { prev_window_start: 46792, prev_window_end: 46797, new_window_start: 46793, new_window_end: 46798 }
2021-12-12 20:41:36.312  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #467759 (0xc17a…eacc)
2021-12-12 20:41:36.224  INFO tokio-runtime-worker babe: [Relaychain] 👶 Next epoch starts at slot 273223626
2021-12-12 20:41:36.224  INFO tokio-runtime-worker babe: [Relaychain] 👶 New epoch 46798 launching at block 0xc17a…eacc (block slot 273223616 >= start slot 273223616).
2021-12-12 20:41:35.841  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (7 peers), best: #467758 (0xfcff…6ba9), finalized #467756 (0x079f…15ec), ⬇ 1.8kiB/s ⬆ 1.7kiB/s
2021-12-12 20:41:32.719  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #36189 (0x422a…7226), finalized #36189 (0x422a…7226), ⬇ 0.8kiB/s ⬆1.0kiB/s
@bkchr
Copy link
Member

bkchr commented Dec 12, 2021

Was the parachain stalled?

Temporary solution is to revert some blocks.

@xlc
Copy link
Contributor Author

xlc commented Dec 13, 2021 •
edited
Loading

The parachain was having block production issue. Restart some nodes got it back working.

I guess it could be indeed too many sibling blocks due to the block production issue, but it shouldn't prevent syncing old blocks.

@bkchr
Copy link
Member

bkchr commented Dec 13, 2021

I guess it could be indeed too many sibling blocks due to the block production issue, but it shouldn't prevent syncing old blocks.

If the synced block is on the same height, it could be. However, the underlying problem of not pruning of blocks that are "dead" is known and needs to be fixed.

@xlc
Copy link
Contributor Author

xlc commented Dec 13, 2021

Ok. Is there a tracking issue of the not pruning blocks bug that I can send people to?

@bkchr
Copy link
Member

bkchr commented Dec 13, 2021

paritytech/cumulus#432

@xlc xlc closed this as completed Dec 13, 2021
@davxy davxy mentioned this issue Sep 23, 2022
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@xlc @bkchr