bug: duplicate messages on the network

[NagyZoltanPeter]

Problem

It has been found that in some cases messages can appear multiple times in the network out of the gossip sub layer's duplicate check time window.
With liteprotocoltester the issue is reproduced in separated simulation environment with high stress conditions but verified the on TWN messages are regularly duplicated in the network multiple times.

Impact

Growing bandwidth usage on the network with these "spam" messages as they are re-delivered to every node again and again.
One may can use this bug to flood the network maliciously.

To reproduce

Liteprotocoltester run on TWN.
With waku-sim it is harder to reproduce due the need of just enough high load.

This thread describes the issue and ways to reproduce it: https://discord.com/channels/1110799176264056863/1269986371142160478

Expected behavior

Messages shall not appear at a node twice for processing and relaying. Duplicates must be clearly filtered within the used time window of 2 min.

nwaku version/commit hash

Comes continuously since first tested and seen on v0.31.0 (probably not related to that release just we have not tested this way before)

Additional context

We have some hypotheses on how it can happen. Enough to have one malfunctioning / slow node - maybe due to low resources - that has stale async operation, specially on processing the message sending queue that would flash the message to the wire.
If that process is starving due to resource conditions, it can happen that message gets send out of the dup-check time window thus other nodes will not recognise it as duplicate and normal propagation will take place.

[SionoiS]

may be related libp2p/rust-libp2p#5504 (comment)

[NagyZoltanPeter]

may be related libp2p/rust-libp2p#5504 (comment)

Thanks for it, it is exactly we are facing with!