Unnamed tasks leaking on RPC nodes

[tomaka]

Here is a screenshot of the number of unnamed tasks on our RPC nodes.
As you can see, we're at around 25000 such tasks at the moment, and it is increasing, which I suppose isn't normal.

An "unnamed task" in this context is a Future that we spawn through the futures::task::Spawn trait. If I'm not mistaken, we don't have any unnamed task in Substrate anymore, so this is probably coming from somewhere in Polkadot.
cc #1244

This leak apparently happens on our two Polkadot RPC nodes and Kusama RPC node, but not our bootnode nor on my own node that has been running for two days. Also doesn't happen on any Kusama non-RPC node.
The flags on these RPC nodes are --ws-port 5556 --unsafe-ws-external --ws-max-connections 5000 --rpc-cors '*'.

[tomaka]

Memory consumption seems to increase by around 100 MiB/day on these nodes, but it's unclear how much that is due to these tasks.

[bkchr]

I think this could be fixed by this pr as well: paritytech/substrate#6341

It lead to transaction notifiers not being closed down, but this would mean the connection stayed open all the time or there is also a bug on the RPC not closing tasks properly. Maybe worth to monitor with 0.8.8.

[tomaka]

Still leaking on 0.8.10-183848b6

[tomaka]

Around 150,000 such tasks alive on each RPC node at the moment.

[tomaka]

The tasks are still unnamed despite running 0.8.14.
We probably have to do #1378 to continue here.

[tomaka]

The leaking task is still unnamed on 0.8.17, despite #1430 being in

The number of tasks seems to stay between 300k and 500k.

[tomaka]

We have reached a peak of 772k tasks a bit earlier

[tomaka]

As Rob suggested, this might be the JSON-RPC code registering finality and chain heads notifications for when users subscribe.

[tomaka]

The leaking task is substrate_rpc_subscription

[tomaka]

Tasks are now named. A new issue should be opened if a task is leaking.