Clamp GOMAXPROCS when higher than runtime.NumCPU

[dimitarvdimitrov]

Background

We are trying to automatically set GOMAXPROCS based on the number of CPUs that an ingester pod requests in Kubernetes. We're going with 2x the requested cores. The reason for this is that the default values of GOMAXPROCS is NumCPU. When running on large nodes and only utilizing a small % of the underlying node results in high scheduling overhead.

Problem

Sometimes the setting of GOMAXPROCS might exceed the number of cores of the node. We also don't want to restrict the nodes on which pods run. In those cases setting GOMAXPROCS to a higher value than NumCPU has the opposite effect - it increases scheduling overhead instead of reducing it.

What this PR does

Clamps the value of GOMAXPROCS to runtime.NumCPU. The idea of this PR is to basically make automating the GOMAXPROCS setting in deployment tooling easier by having some support from the code.

Considerations

It's possible that the original NumCPU is not correctly detected or that the operator intended to run with higher GOMAXPROCS. I didn't want to add more configuration options, so instead we're logging a warning message. If there is a real use case for GOMAXPROCS > NumCPU, then we can add the configuration option later.

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[dimitarvdimitrov]

cc @agardiman since you raised this at the last community call

[pracucci]

LGTM. I'm wondering if there should me a minimum GOMAXPROCS for components using mmap (ingesters), regardless the number of CPU cores in the node, due to the "golang mmap issue".

[dimitarvdimitrov]

AFAIK we have no visibility into how much of the process capacity mmap might be using, so not sure how to make this call - is 2 enough? 10? Do you have any ideas?

[dimitarvdimitrov]

golang mmap issue

I had a look at a single ingester process via Linux's pprof. I was looking at the proportion of time spent in handle_mm_fault (60s profiling @ 500Hz: perf record -F 500 -g -p <PID> -- sleep 60). Looking at perf script there are only 18 stack traces that include handle_mm_fault out of 44146 total stacks: 0.04%.

Flamegraph & `perf script` output

out.script.zip

Unfortunately, this is an extremely narrow view of ingester usage. But the method with perf doesn't scale at all. At least it suggests that in a stable state the overhead of mmap can be negligible.

I'll merge this PR since I don't think the clamping is likely to be a negative change. Happy to continue discussing about mmap and GOMAXPROCS.