Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarity required around Logging Architecture of Kubelet. #51

Open
javadevmtl opened this issue Jul 15, 2024 · 0 comments
Open

Clarity required around Logging Architecture of Kubelet. #51

javadevmtl opened this issue Jul 15, 2024 · 0 comments

Comments

@javadevmtl
Copy link

Running Kubernetes 1.27.6

I have some performance issues regarding logrotation where I miss logs, I'm not here blaming Kubernetes, but there needs some clarification around the Log Architecture documents here Loggin Architecture or more details.

We have set containerLogMaxSize and containerLogMaxFiles which works fine and is rotating logs as per the settings. We are trying to debug an issue where we miss logs when there is large volume.

Now here is the disparity.

  1. Based on the linked document above and the 2 settings we are lead to believe that Kubelet is doing the rotation, while many online resources, forums, seem to be saying it's logrotate.
  2. Looking at the /var/log/pods/ folder indeed the logs are being rotate as per the 2 settings, but they are also gzipped. Nowhere does the document explain this and there's no settings indicated anywhere for compressing the logs. Which leads us to belive that the file is being zipped before the file agent can finish reading all the contents of the file.
  3. If logrotate is truly used does Kubelet pass these 2 settings containerLogMaxSize and containerLogMaxFiles to logrotate somehow and if so how (We looked in /etc/logrotate.d/ and /etc/logrotate.conf and found no indication of a Kubenrnetes config pointing to /var/log/pods/)?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant