Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix resource detection for unset CPU and memory limits in cgroup v2 #5267

Conversation

mobratil
Copy link
Contributor

@mobratil mobratil commented Jul 4, 2024

Address the issue where /sys/fs/cgroup/cpu.max and /sys/fs/cgroup/memory.max report "max", indicating unset limits in cgroup v2.

Changes Made:

  • When /sys/fs/cgroup/cpu.max reads "max", the number of CPU cores is now derived from the host's /proc/stat file.
  • Similarly, available memory is determined from the host's /proc/meminfo file if /sys/fs/cgroup/memory.max contains "max".
Microsoft Reviewers: Open in CodeFlow

Address the issue where `/sys/fs/cgroup/cpu.max` and `/sys/fs/cgroup/memory.max` report "max", indicating unset limits in cgroup v2. Now, system resources (CPU cores and memory) are fetched directly from `/proc/stat` and `/proc/meminfo` limits are not set..
@gitmln
Copy link
Contributor

gitmln commented Jul 12, 2024

Unfortunately, this pull request does not fix either of the two issues.

  1. CPU limits
    LinuxUtilizationProvider can't handle 'max' in /sys/fs/cgroup/memory.max #5241
    Could not parse '/sys/fs/cgroup/cpu.max'. Expected an integer but got: 'max 100000'.
    if (quotaBuffer.Equals("max", StringComparison.InvariantCulture))
    quotaBuffer in the above case contains "max 100000", not "max" string,
  2. Memory limits
    This one is more tricky.
    System.InvalidOperationException : Could not parse '/sys/fs/cgroup/memory.max' content. Expected to find available memory in bytes but got 'max ' instead #5161
    Could not parse '/sys/fs/cgroup/memory.max' content. Expected to find available memory in bytes but got 'max
    ' instead.

    if (memoryBuffer.Equals("max", StringComparison.InvariantCulture))
    Again, you are comparing the wrong strings. The correct one is "max\n" (note the new line at the end!).

@mobratil mobratil marked this pull request as ready for review July 22, 2024 12:06
@mobratil
Copy link
Contributor Author

@gitmln thanks for pointing that out. The PR was just a draft because i didn't have a real installation of Linux with cgroup v2 on which I could test it.
Now I finally tested the PR with proper modifications.

@RussKie
Copy link
Member

RussKie commented Jul 22, 2024

@gitmln how does the current implementation look to you?

@gitmln
Copy link
Contributor

gitmln commented Jul 23, 2024

@RussKie It looks perfect to me. This is the code that I've already thought about and tested successfully using a generic web app with mcr.microsoft.com/dotnet/runtime-deps:8.0 image on my AKS cluster.

@mobratil
Copy link
Contributor Author

@gitmln thanks for checking. Do not hesitate to create a PR next time 😊

@mobratil mobratil merged commit 6a3cb23 into main Jul 23, 2024
6 checks passed
@mobratil mobratil deleted the 5241-linuxutilizationprovider-cant-handle-max-in-sysfscgroupmemorymax branch July 23, 2024 10:00
@RussKie
Copy link
Member

RussKie commented Jul 25, 2024

Thank you @mobratil

@github-actions github-actions bot locked and limited conversation to collaborators Aug 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

LinuxUtilizationProvider can't handle 'max' in /sys/fs/cgroup/memory.max
4 participants