Expose validation webhook warnings in Argo CD UI

[Sergey-Kizimov]

Summary

We implemented a warning message in the validation webhook, but Argo CD UI doesn't show a warning when syncing or creating resources. A warning message only logged into the application-controller log

Motivation

In our case, we want to see a warning validation webhook message in resource events in the Argo CD UI

Related links: https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#:~:text=Starting%20in%20v1.19%2C%20admission%20webhooks%20can%20optionally%20return%20warning%20messages%20that%20are%20returned%20to%20the%20requesting%20client%20in%20HTTP%20Warning%20headers%20with%20a%20warning%20code%20of%20299.%20Warnings%20can%20be%20sent%20with%20allowed%20or%20rejected%20admission%20responses.

[gmcbrien]

We have a similar requirement. We use OPA Gatekeeper to enforce policies in higher environments. We want to set the enforcement policy to 'Warn' in lower, development environments. Trouble is, the developers don't see these warnings in ArgoCD.

[nerzhul]

Hello,
I just fall on the same usecase on my side :) i use gatekeeper and i try to find the warnings somewhere but i don't find them

[nikolay-te]

I have the same issue, and I was thinking of probably exposing the warnings also as kubernetes resource Events, but the trouble is the warnings are generated in the admission webhook before the resource exists in the cluster, so I either have to find a way to pass them to the controller or re-run the whole validation again after the resource is applied.

[jdoylei]

We have OPA Gatekeeper and admission webhooks in general in our road map for adopting Kubernetes, for which this admission webhook warning support in Argo CD would be useful.

In the nearer term, we are at least looking to enable Pod Security Admission in our clusters. Currently the Pod Security Admission warnings are ignored by Argo CD, the same as admission webhook warnings. If I understand it right, warnings from both Pod Security Admission and admission webhooks are based on the same general API server warning response header mechanism, and I suspect the Pull Requests already submitted for this issue probably work with that mechanism and address both at the same time, which would be great.

As users move to more recent K8s versions and leverage Pod Security Admission, having this support in Argo CD becomes really crucial to Pod Security Admission being useful with GitOps and Argo CD.

The proposed interface for presenting the warnings in the GUI, as @Sergey-Kizimov shows in the Pull Requests, looks great to me!

[brunomiguel-teixeira]

Is there any news regarding this topic? It would be extremely helpful to warn clients of upcoming changes (ex: new APIs)

[crenshaw-dev]

I think this PR needs some work to help us move forward. @Sergey-Kizimov is unresponsive (no worries, life gets busy!). If anyone has time to pick up the PRs, I'd happily help push this forward.

[donflavour]

Are there any updates or any plans to move forward in this case? We would really appreciate it..

[crenshaw-dev]

@donflavour this feature needs a new champion. There are two open PRs and some discussion about how to move forward with them. Just need someone who has time to pick them up and work on them.

[wangli1030]

Hi @crenshaw-dev, I have the same issue and would like to pick this up. Any guideline or direction I should follow to move this forward?

[wangli1030]

Hi @crenshaw-dev and @Sergey-Kizimov , regarding this PR and issue @Sergey-Kizimov mentioned I have dived into the code and found that rest.SetDefaultWarningHandler(rest.NewWarningWriter(warn, rest.WarningWriterOptions{})) will set the default warning handler for all the clients. Even if it has been set to different object when applying resources, since applying is in parallel, normally the last setting will take effect and all the warning messages will go the last resource. That is why when multiple resources causing the warning, only one (normally it is last applied) resource will get the warning message. And if I understand the code correctly, it may also result when syncing multiple Argo applications at the same time, the warning messages for different Argo applications may only show in single resources.
So after I dig into the code, I found the client will be created for every apply. So I think the WarningHandler could be created and set for every client and in this way, the warning message for each resource will only exist in its writer/buffer. However, when I tested this approach, I found the client still does not use the WarningHandler passed in from the k.config, it still uses the default one. Is there anything I am missing or better approach we could achieve this? Really think this feature would benefit a lot and need help here. Thank you. Also, I could post my code if it is needed. Thank you.

[wangli1030]

After my investigation, there might be a bug in kubectl client and causing the customized WarningHandler is not used. I have raised an issue in kubectl project.