Add ability to disable policy per namespace

[phenixblue]

What would you like to be added:

It would be nice to have the ability to disable individual policies on a per-namespace basis.

Why is this needed:

This allows for flexibility to granularly disable policies without the need to completely remove a policy from a cluster, or having to lower the severity level of the policy at a global cluster level to meet the needs of a specific namespace.

Ideally I think something along the lines of a label on the namespace could work. Something like:

k8s.t-mobile.com/magtape-disable-<policy_name>: true

If the label is found for a specific policy we need to add logic to skip deny's, but still track failures for alerting/event creation. Because this is primarily valid in environments where end-users can't manipulate their assigned namespace resource, this should also be a global toggle with an ENV var similar to MAGTAPE_ENABLE_NS_TOGGLE

[phenixblue]

Could also be additional metadata in the policy definition. This could potentially support both includeNamespaces and excludeNamespaces without requiring an API call or cache lookup.

[ilrudie]

FWIW I lean towards some implementation that would not allow a misbehaving user who is clever to exempt themselves by altering their namespace. For that reason of the two ideas noted thus far having exemption metadata in the policy sounds like a better option to me.

[ilrudie]

Could we just store the operating mode (i.e. if magtape enforces by default and exempts versus permitting by default and enforcing selectively), plus a list of name spaces that you want to exempt/permit in a configmap and mount that into the webhook pod where we can read it? This prevents end users from honking up a label/annotation in their own name space but allows us to use tools like Kustomize to manage this data per cluster instead of having it globally in the rego or having to worry about modifying the rego itself in pipeline somehow to annotate it per cluster. Also maintains better separation between the data and the policy definition.

[phenixblue]

In general, it's somewhat assumed that end-users can't manipulate the namespace resource itself, including labels/annotations. I'd love to be able to move away from that assumption, but since the Admission Webhooks are primitively founded on namespace labels for enabling/disabling the webhook scope, I can't think of a way we can fully move away from that.

We have more freedom with controlling policy level granularity since policy assessment happens inside the webhook.

[phenixblue]

@ilrudie Circling back around on this idea, and maybe we can add to the policy metadata like this:

policy_metadata = {
	# Set MagTape Policy Info
	"name": "policy-emptydir",
	"severity": "MED",
	"errcode": "MT1009",
	"targets": {"Pod"},
        "disable_namespaces": {"namespace1", "namespace2", "namespace3"},
}

Thoughts?

[ilrudie]

This seems good to me.