Abnormal metrics reported by tiflash-proxy

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Deploy a v6.4.0 cluster and run high pressure update workloads.

2. What did you expect to see? (Required)

3. What did you see instead (Required)

The CPU and memory reported by tiflash-proxy is abnormal. However, the metrics reported by tiflash is OK

4. What is your TiFlash version? (Required)

v6.4.0

[hehechen]

This is because the profile takes more time than expected, causing the status_server thread to be stuck and unable to process the http request from prometheus.
For example: A 10s profile takes more than a minute

The reason why profile took longer than expected is still under investigation.

[hehechen]

It seems that the overhead of dwarf is larger than that of frame pointer. I tried to revert this PR and it will be fine.

[hehechen]

I add some logs in https://github.com/rust-lang/backtrace-rs/blob/a59e64f50c571d9675616321b22c2953d70d1a8b/src/symbolize/gimli.rs#L338, and found out that the number of dwarf symbols is more than frame pointer.
dwarf:

frame pointer:

So the overhead of dwarf resolving symbol is larger than that of frame pointer.

[hehechen]

The root cause is that the backtrace of DWARF is deeper than that of frame pointer. So DWARF need to resolve more shared libraries when resolving symbols. But the capacity of lib cache in backtrace-rs is only 4, so cache miss will occurs frequently in DWARF scenarios.
After I increased the capacity of lib cache in backtrace-rs to 10, the 10 seconds profile took less than 11 seconds, and the metrics became normal.

one backtrace comparison:
frame pointer:

DWARF:

I added a log when inserting lib cache, and the log showed that:
Libraries need to resolve in DWARF scenarios :

Libraries need to resolve in frame pointer scenarios :

The flame graph of DWARF when resolving symbols: