Presence is worse on 0.99.3

[turt2live]

Possibly related: #4713
Related: #3971

https://snapshot.raintank.io/dashboard/snapshot/7FO0m6GfHbrA183bQ60ukgl7VQU8lJy2

At roughly 19:00 UTC in that graph I upgraded Synapse from 0.99.0 to 0.99.3 on t2bot.io - shortly after the server started melting due to presence EDU spam. Traffic in rooms does not appear to be any different, but going from under 1Hz of EDUs outbound to 25Hz+ is a bit drastic.

[turt2live]

This might actually be a change in how the metrics are populated? It lines up perfectly with the outgoing transaction rate after 19:00, but not before. 40Hz+ of outgoing transactions is normal for t2bot.io, but there is an unexplained gap between the number of PDUs and EDUs going out versus the transaction rate.

Edit: The graph in question being https://snapshot.raintank.io/dashboard/snapshot/80FmwXBJSMHBDeq01EKhB8KqHFAbNcPr

[erikjohnston]

I'd be interested in what synapse_federation_client_sent_edus_by_type metric looked like in that time

[turt2live]

Was m.presence, which has the same line as EDUs from the snapshots.

I think it's just recording it differently now, which means that presence gets counted.

[turt2live]

after some research, it is indeed the metrics being recorded differently. Not sure when it changed, but the outbound traffic is the same - it just looks different in grafana.