ram: high traffic sites appear to have memory leak

[stephen-soltesz]

Watching the topk dashboard, I noticed that very high traffic sites, maa.. mnl.., appear to have a gradual memory leak for all sidecar services that is not observed at other sites.

Image include reset during rollout on 07/08

[nkinkade]

There may indeed be a memory leak, but one other factor probably contributing to the unusually high load at MAA01, is that MAA02 has been down for the past couple weeks, meaning that MAA01 has to carry the whole load for that region.

[stephen-soltesz]

gru is also one of our busiest metros. If there is a leak, it seems more likely to be triggered in busy locations. It's possible the leak is very slow at other locations.

[laiyi-ohlsen]

@stephen-soltesz

[stephen-soltesz]

There is a growth curve. It's possible the curve is leveling off as it approaches 1GB to 1.5GB?

The highest memory users are:

• tcp-info
• traceroute-caller

This requires pprof heap traces to investigate further. It may also require running a service for longer than 30days.