Proposal: Validation for CustomResources. #708

nikhita · 2017-06-12T16:47:02Z

ThirdPartyResource (TPR) is deprecated and CustomResourceDefinition (CRD) is the successor which solves the fundamental issues of TPRs to form a stable base for further features.

Currently we do not provide validation for CustomResources (CR). This proposal proposes the design and describes a way to add JSON-Schema based validation for CustomResources.

cc @sttts @deads2k @enisoc @xiao-zhou @erictune @lavalamp @brendanburns @philips @fabxc @frankgreco @sdminonne

sttts · 2017-06-12T16:49:39Z

/cc @kubernetes/sig-api-machinery-api-reviews @kubernetes/sig-api-machinery-misc

deads2k · 2017-06-12T17:10:46Z

@sdminonne

fabiand · 2017-06-12T18:39:35Z

contributors/design-proposals/customresources-validation.md

+        1. [Direct Embedding of the Schema into the Spec](#direct-embed)
+        2. [External CustomResourceSchema Type](#external-type)
+
+## <a name="overview"></a>Overview


You might want to stick to markdown only, thus remove the anchor.

enisoc · 2017-06-12T18:58:45Z

cc @mbohlool

enisoc

Looks great to me overall.

enisoc · 2017-06-12T18:59:28Z

contributors/design-proposals/customresources-validation.md

+
+We do a schema pass there using the https://github.com/go-openapi/validate validator with the provided schema in the corresponding CRD. Validation errors are returned to the caller as for native resources.
+
+JSON-Schema also allows us to reject additional fields that are not defined in the schema and only allow the fields that are specified. This can be achieved by using `"additionalProperties": false` in the schema.


I'm not sure if there's a defined policy, but it seems like native k8s objects simply ignore unrecognized fields, at least on the server side. Is there any danger in even allowing CRD authors to violate this expectation/convention?

Server-side validation needs to be optional:
kubernetes/kubernetes#5889 (comment)

That is, schema validation needs to be optional. Value validation does not.

To elaborate, my interpretation of the chain of comments @bgrant0607 linked is: Yes, there is danger in allowing CRD authors to set additionalProperties: false because it breaks version skew (new client sends new optional field to old server). At most, the client should be able to opt-in to "reject unrecognized fields" on a per-request basis. However, there is currently no such standard query param (AFAIK), so we should not allow CRD authors to set additionalProperties: false.

Updated to not allow CRD authors to set additionalProperties: false.

enisoc · 2017-06-12T20:19:59Z

contributors/design-proposals/customresources-validation.md

+
+Note: A reflective test to check for drift between the types here and the OpenAPI types for runtime usage will be added.
+
+## JSON-Schema Examples


Could you also include some examples of what validation errors would look like in this proposal? For example, does the description field allow the author to inject hints into the error message?

One trade-off between this approach (using a JSON-Schema library) and alternatives (e.g. webhooks) is that the CRD author has less control over the content of validation error messages. Of course we're not limited to just printing whatever the library spits out, but if more work will be needed to improve on what the library gives us, we should scope it out here since telling the user how to fix the problem is arguably the most important part of validation.

Added examples of validation errors.

enisoc · 2017-06-12T20:20:25Z

contributors/design-proposals/customresources-validation.md

+
+This is especially true in situations when CRs are used by components that are out of the control of the admin. Example: A user can create a database CR for a Database-As-A-Service. In this case, only server-side validation can give confidence that the CRs are well formed.
+
+### Existing Instances and changing the Schema


In the Google Doc, there was some discussion about ways to deal with CRs that exist before the introduction of validation on a given CRD for the first time. Can you summarize the findings in this proposal?

enisoc · 2017-06-12T20:20:45Z

contributors/design-proposals/customresources-validation.md

+
+Note: this is a workaround while we do not support multiple versions and conversion for custom resources.
+
+### JSON-Schema Validation Runtime Complexity


In general, it seems like this will be slower than native validation. We should consult with @kubernetes/sig-scalability-proprosals to define the performance bar this needs to meet and create regression tests for large numbers of objects.

This is certainly slower than hand crafted validation. The main point of this paragraph is not that's it might be slower by a factor of 10 (a guess), but that it cannot be used to bring down an API server because the validation algorithm is quadratic of even exponential in the input size.

But of course, having a performance bar with tests would be great.

enisoc · 2017-06-12T20:20:57Z

contributors/design-proposals/customresources-validation.md

+
+The CRD JSON-Schema will be validated to have neither recursion, nor `uniqueItems=true` being set.
+
+### Alternatives


I think many people will see imperative validation (custom code inserted in the request path somehow) as the main alternative to this proposal. You mentioned above they are not mutually exclusive, but it would be good to document here why we think it makes sense to put our effort in creating the declarative version first.

Schema validation performed by kubectl is driven by Swagger/OpenAPI models. An equivalent capability needs to move to the server (kubernetes/kubernetes#5889) as part of the general effort to move functionality to the server in order to improve extensibility and to simplify clients (kubernetes/kubernetes#12143).

I would like to see us move towards declarative validation generally for the API, as much as possible:
kubernetes/kubernetes#25460

@mbohlool may have already started on a proposal for that.

Also, ideally, I'd like the same spec to be usable to serve an OpenAPI spec for the CRD APIs, which would mean it would have to be an OpenAPI-compatible flavor of JSON schema. I really don't want multiple different schema languages in the system.

As for a hook-based approach, (potentially multiple) special-purpose hooks per resource would be harder to understand, as well as being more fragile, slower, etc. I'd prefer to find a way to use admission-control extension for that use case, so as not to add yet another hook mechanism:

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_extension.md

No need for another hook mechanism. We don't want that. I verified that CRDs play nicely with initializers in 1.7, same should be true for admission webhooks (not verified).

Added a note about intializers.

Commented above. The commentary assumes/implies that webhooks are harder and less useful than json-schema. I think exactly the opposite is true.

Personally, I would argue for webhook validation in the CRD resource because conceptually it is part of the resource. While it is possible to do it in an extension admission controller, the user experience is way worse, because suddenly I have to create/manage/synchronize two different objects to implement my desired experience.

Instead, I think we should add webhooks to the CRD, and then implement a controller that reflects that webhook into an admission control extension. (similar to how ReplicaSet creates Pod objects, etc)

IMHO we shouldn't mix our desires around implementation complexity (only one kind of webhook) with our designs around API complexity (everything relating to a particular object should be in that object)

enisoc · 2017-06-12T20:59:30Z

contributors/design-proposals/customresources-validation.md

+
+Currently we do not provide validation for CustomResources (CR), i.e. the CR payload is free-form JSON. However, one of the most requested [[1](https://github.com/kubernetes/features/issues/95#issuecomment-296416969)][[2](https://github.com/kubernetes/features/issues/95#issuecomment-298791881)] features is validation and this proposal seeks to add it.
+
+## Goals


I just realized there are examples of giving fields default values below. Is it a goal of this proposal to support defaulting? Either way, defaulting should be added to either Goals or Non-Goals to make it clear.

I went into go-openapi this morning and prototyped defaulting support: go-openapi/validate#27.

So this is not available currently upstream, but is feasible. The only support that exists is that errors of required fields are surpressed if a default value is given. My PR adds support to actually take that default and write it into the field.

@nikhita has added a link to that prototype below.

bgrant0607 · 2017-06-13T02:47:20Z

Original description of this GSOC project:
https://summerofcode.withgoogle.com/organizations/6540924424290304/#5982049109278720

bgrant0607 · 2017-06-13T02:53:25Z

Ref kubernetes/kubernetes#38117

bgrant0607 · 2017-06-13T03:04:35Z

@kubernetes/sig-api-machinery-proposals

fabiand · 2017-06-13T06:33:15Z

contributors/design-proposals/customresources-validation.md

+## Goals
+
+1. To provide validation for CustomResources using a declarative specification language for JSON data.
+2. To keep open the door to add other validation mechanisms later.<sup id="f2">[2](#footnote2)</sup>


You might want to drop the <sup> tag

Github Flavored Markdown does not support footnotes without the <sup> tags.

Please let me know if this is indeed possible and I'm missing something here!

fabiand · 2017-06-13T06:48:04Z

contributors/design-proposals/customresources-validation.md

+
+The implementation is planned in the following steps:
+
+1. Add the proposed types to the v1beta1<sup id="f3">[3](#footnote3)</sup> version of the CRD type.


You probably want to drop <sup> here as well.

deads2k · 2017-06-14T13:33:22Z

contributors/design-proposals/customresources-validation.md

+	MaxItems             *int64                     `json:"maxItems,omitempty"`
+	MinItems             *int64                     `json:"minItems,omitempty"`
+	// disable uniqueItems for now because it can cause the validation runtime
+	// complexity to become quadratic.


CRDs are inherently a cluster-admin resource since they affect the resources for the entire cluster. I don't feel bad about allowing a cluster-admin to make an expensive resource.

Will cluster-admins understand complexity?

Will cluster-admins understand complexity?

Will they care? I suspect that first line of defense would simply be to scale out the masters.

deads2k · 2017-06-14T13:35:06Z

contributors/design-proposals/customresources-validation.md

+
+The client-side validation is carried out before sending the request to the api-server, or even completely offline. This can be achieved while creating resources through the client i.e. kubectl using the --validate option.
+
+If the API type serves the JSON-Schema in the swagger spec, the existing kubectl code will already be able to also validate CRs. This will be achieved as a follow-up.


@mbohlool this is going to require changes to the openapi aggregation triggers.

Sorry that I see this late. I have two points here:

If we register CRDs with aggregation server (which I think we should), this will be aggregated automatically.

OpenAPI supports a subset of json schema. I think instead of asking users to provide JSON Schema we should ask them to provide OpenAPI spec that has a Definition model in it (that is a subset of JSON Schema).

about 2) why should we restrict ourselfes to openapi? The openapi spec is a side-product of this validation and I don't want to loose expressivity because openapi hasn't updated to the JSON schema draft 4 yet.

Moreover, some day openapi v3 is finalized, we will support v3 and v2 for backward compatibility. Do you suggest that we stay at the common denominator, i.e. to that JSON schema subset supported by v2? In other words, at some point we will have to diverge anyway.

Is there anything that stops us from stripping the given spec from unsupported features for consumation by the openapi aggregator? As far as I see all JSON Schema fields are monotonic in the sense that stripping fields makes the spec weaker. That's what we need for easy openapi consumation.

deads2k · 2017-06-14T13:37:25Z

contributors/design-proposals/customresources-validation.md

+
+1. This is the same behavior that we require for native resources. Validation cannot be made stricter in later Kubernetes versions without breaking compatibility.
+
+2. For migration of CRDs with no validation to CRDs with validation, we can create a controller that will validate and annotate invalid CRs once the spec changes, so that the custom controller can choose to delete them (this is also essentially the status condition of the CRD). This can be achieved, but it is not part of the proposal.


I'm not really a fan of this. It seems like it should mirror kube-apis: it's the responsibility of the APi author and fixups are performed client-side.

deads2k · 2017-06-14T13:38:50Z

contributors/design-proposals/customresources-validation.md

+The implementation is planned in the following steps:
+
+1. Add the proposed types to the v1beta1<sup id="f3">[3](#footnote3)</sup> version of the CRD type.
+2. Add a validation step to the CREATE and UPDATE REST handlers of the apiextensions-apiserver.


I think you actually just want to update the strategy.

yes, that's actually meant here.

Yes, will be passing the CRD to the strategy.

deads2k · 2017-06-14T13:40:03Z

minor comments. This lgtm.

sdminonne · 2017-06-14T14:09:51Z

I liked the declarative approach defining validation but I would love to have a way to reuse definitions similar to swagger multiple files
Even better would be to reuse swagger as already reported by @bgrant0607-nocc

The same mechanism should be used to reference native resources like PodSpec.
To be more clear: If my CDR contains a v1.PodSpec I would like references the swagger definition of the io.k8s.kubernetes.pkg.api.v1.PodSpec avoiding cut & paste of the swagger.json.

According to @sttts this mechanism may be added in a second step (brief discussion on slack). So I'm in general OK with this approach.

sttts · 2017-06-14T14:34:56Z

Adding to @sdminonne's comment:

The languages of native kube object specs in swagger and those proposed here are the same. Hence, a schema reference could be used to link a CRD to a native object spec, using a JSON-Schema reference with a certain URL.

The main conceptual problem to solve is to define this URL namespace. Note that we are coupling non-kube-CRDs with the kube type universe, maybe even with a number of user provided API servers. One could use the service names of those API servers to reference their specs, e.g. https://kubernetes/api/v1/pods/schema.json and https://kubernetes/apis/you.group.com/your-custom-resources/schema.json. But this needs deep thought as we are coupling independent components, after splitting them apart into single API servers with a lot of effort.

Hence, I see this as a follow-up to this proposal. Right now, we will keep schema references local and the schemas self-contained.

deads2k · 2017-06-19T12:04:41Z

Right, but my point is that I may actually want multiple web hooks, or a
JSON schema and a webhook. I think an array captures this more elegantly
than a single top-level struct.

The current struct does not preclude introducing a slice of webhooks. Your example also highlights a deficiency of an array of validators, since some combinations like multiple JSON-schemas don't make sense. Rather than enforce that in resource validation, I would make the struct represent the valid combinations.

agonzalezro · 2017-06-20T16:05:35Z

If multiple validations are added (as @brendanburns mentioned) I suppose it's going to be a new struct field here: CRD.Spec.Validation.*.

Is the order of this validations important? For example, somebody would like to run JSON Schema validations before calling a webhook.

In case it's important what do you think about something like this (please, read this as pseudo code 😬):

type CustomResourceDefinitionSpec struct {
   ...
   Validations []CustomResourceValidation `json:"validation,omitempty"`
}

type CustomResourceValidation interface{
  Validate() []error
}

This would allow us to even repeat validations which can be good and bad at the same time:

good: we could validate with JSON Schema, webhook A (microservice from dev team), webhook B (microservice from finance team -> billing).
bad: you could as well add several JSON Schemas validation to the same CRD that could end up fighting between them (one returning true and the other false).

jamiehannaford · 2017-06-22T12:09:18Z

contributors/design-proposals/customresources-validation.md

+## Non-Goals
+
+1. The JSON-Schema specs can be used for creating OpenAPI documentation for CRs. The format is compatible but we won’t propose an implementation for that.
+2. A turing-complete specification language is not proposed. Instead a declarative way is proposed to express 90% of validations.


Is 90% literal here, or just an example?

Just an example.

Ah okay, can we change to express the vast majority of validations or something? Just for folks who might take this literally

jamiehannaford · 2017-06-22T12:09:42Z

contributors/design-proposals/customresources-validation.md

+
+## Background
+
+ThirdPartyResource (TPR) is deprecated and CustomResourceDefinition (CRD) is the successor which solves the fundamental issues of TPRs to form a stable base for further features.


Is it outside the scope of this doc to briefly mention what those issues are? Perhaps just as links

Will add them. 👍

jamiehannaford · 2017-06-22T12:11:53Z

contributors/design-proposals/customresources-validation.md

+
+## API Types
+
+The schema is referenced in [`CustomResourceDefinitionSpec`](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1/types.go#L22). `Validation` is of the type `CustomResourceValidation`. The JSON-Schema is stored in a field of `Validation`. This way we can make the validation generic and add other validations in the future as well.


Nit: maybe we can pin to a specific commit in the URL? I've lost track of the amount of times I've followed master links and the lines are outdated 😄

good point. :)
will do.

jamiehannaford · 2017-06-22T12:14:32Z

contributors/design-proposals/customresources-validation.md

+}
+
+// JSONSchemaProps is a JSON-Schema following Specification Draft 4 (http://json-schema.org/).
+type JSONSchemaProps struct {


Could we use a json schema lib like https://github.com/xeipuuv/gojsonschema instead of defining this ourselves? I know folks are wary of more imports but just thought I'd float the idea

I believe we don't really want to import here and end up making our types depend on external types.

Had considered using https://github.com/xeipuuv/gojsonschema but instead decided to go with go-openapi because we want to be able to express the schema via openapi docs.

Yeah, it's also defined here. Just wondering if we can avoid duplication and the risk of feature drift.

@liggitt should we be avoiding external types here?

external types don't have the protobuf annotations needed by our API types. tying our API to an external API is not something we want to do.

nikhita · 2017-06-22T12:45:00Z

@agonzalezro That looks like a nice approach but what I am concerned about is that we can end up with multiple JSON schemas and allow invalid combinations. We need the JSON schema to only have one entry point (not multiple).

Address comments

sttts · 2017-07-03T13:00:33Z

contributors/design-proposals/customresources-validation.md

+
+## API Types
+
+The schema is referenced in [`CustomResourceDefinitionSpec`](https://github.com/kubernetes/kubernetes/commit/0304ef60a210758ab4ac43a468f8a5e19f39ff5a#diff-0e64a9ef2cf809a2a611b16fd44d22f8). `Validation` is of the type `CustomResourceValidation`. The JSON-Schema is stored in a field of `Validation`. This way we can make the validation generic and add other validations in the future as well.


To address @brendanburns's comment #708 (comment):

This way we can make the validation generic and add other validation options in the future. Even multiple validation options could be used at the same time within one CustomResourceDefinition.

sttts · 2017-07-03T14:12:30Z

I can see value in @agonzalezro's example with multiple webhooks. Also the possibility to rely on static JSON-schema validation before calling a webhook sounds reasonable.

Though I am not so sure about introducing any more structure, i.e. a whole slice of validation mechanisms. As the next step we would want to define the parallalism, i.e. sequential or in parallel, eventually leading to complete pipeline semantics. I agree with @deads2k that having a struct { JSONSchema, []WebHook } is a good line to draw. If some day we have some embedded interpreter of some sort, we could have struct { JSONSchema, Script, []WebHook }.

agonzalezro · 2017-07-03T14:17:27Z

I see struct { JSONSchema, []WebHook } reasonable. In the case that it's EXTREMELY important for somebody to to run the webhooks before the schema they could implement the schema validation in their latest webhook. Of course, this case would be like the 0.1% of the people :)

Looks pretty good, good job you all.

deads2k · 2017-07-05T14:18:00Z

/lgtm

frankgreco · 2017-07-05T17:47:39Z

I am very interested in contributing to this change. Is this the right issue to be following for that. I assume at some point this thread will turn into more of implementation discussion once the final design is finished.

enisoc · 2017-07-05T17:53:17Z

@frankgreco There is work in progress here: kubernetes/kubernetes#47263

deads2k · 2017-07-05T19:34:19Z

I am very interested in contributing to this change. Is this the right issue to be following for that. I assume at some point this thread will turn into more of implementation discussion once the final design is finished.

I figured I'd give a day for closing arguments and merge this tomorrow.

deads2k · 2017-07-06T11:47:21Z

I figured I'd give a day for closing arguments and merge this tomorrow.

Ok. I'll merge this as a starting point. Specific issues are welcome as are comments on the WIP kubernetes/kubernetes#47263

mbohlool · 2017-07-31T15:14:18Z

OpenAPI v3 is release and we are going to support it but even v3 Schema is a subset of JSON schema. I don't think OpenAPI supports complete JSON schema any time soon if ever. If we provide JSON schema for CRDs there is no path forward for converting it to OpenAPI (hence at least the proposal's assertion that it is compatible with OpenAPI is false). If we are thinking of also asking users to provide OpenAPI spec that would be a bad design in my opinion. Two very similar things. I don't think OpenAPI's Schema is that restrictive and I do believe we should use that as it widely accepted (not that JSON schema is not but for the puropose of describing API objects, OpenAPI designed for that).

…

On Jul 31, 2017 8:04 AM, "Dr. Stefan Schimanski" ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In contributors/design-proposals/customresources-validation.md <#708 (comment)>: > + +ObjectMeta and TypeMeta are implicitly specified. They do not have to be added to the JSON-Schema of a CRD. The validation already happens today as part of the apiextensions-apiserver REST handlers. + +### Server-Side Validation + +The server-side validation is carried out after sending the request to the apiextensions-apiserver, i.e. inside the CREATE and UPDATE handlers for CRs. + +We do a schema pass there using the https://github.com/go-openapi/validate validator with the provided schema in the corresponding CRD. Validation errors are returned to the caller as for native resources. + +JSON-Schema also allows us to reject additional fields that are not defined in the schema and only allow the fields that are specified. This can be achieved by using `"additionalProperties": false` in the schema. However, there is danger in allowing CRD authors to set `"additionalProperties": false` because it breaks version skew (new client can send new optional fields to the old server). So we should not allow CRD authors to set `"additionalProperties": false`. + +### Client-Side Validation + +The client-side validation is carried out before sending the request to the api-server, or even completely offline. This can be achieved while creating resources through the client i.e. kubectl using the --validate option. + +If the API type serves the JSON-Schema in the swagger spec, the existing kubectl code will already be able to also validate CRs. This will be achieved as a follow-up. about 2) why should we restrict ourselfes to openapi? The openapi spec is a side-product of this validation and I don't want to loose expressivity because openapi hasn't updated to the JSON schema draft 4 yet. Moreover, some day openapi v3 is finalized, we will support v3 and v2 for backward compatibility. Do you suggest that we stay at the common denominator, i.e. to that JSON schema subset supported by v2? In other words, at some point we will have to diverge anyway. Is there anything that stops us from stripping the given spec from unsupported features for consumation by the openapi aggregator? As far as I see all JSON Schema fields are monotonic in the sense that stripping fields makes the spec weaker. That's what we need for easy openapi consumation. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#708 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABic4ExRsgiRbp5YqRtIJG3ejb2nRX4Eks5sTezkgaJpZM4N3YUj> .

sttts · 2017-07-31T15:16:47Z

@mbohlool let's continue in the issue kubernetes/kubernetes#49879

@sttts

Automatic merge from submit-queue apiextensions: validation for customresources - [x] Add types for validation of CustomResources - [x] Fix conversion-gen: #49747 - [x] Fix defaulter-gen: kubernetes/gengo#61 - [x] Convert to OpenAPI types - [x] Validate CR using go-openapi - [x] Validate CRD Schema - [x] Add integration tests - [x] Fix round trip tests: #51204 - [x] Add custom fuzzer functions - [x] Add custom conversion functions - [x] Fix data race while updating CRD: #50098 - [x] Add feature gate for CustomResourceValidation - [x] Fix protobuf generation Proposal: kubernetes/community#708 Additional discussion: #49879, #50625 **Release note**: ```release-note Add validation for CustomResources via JSON Schema. ``` /cc @sttts @deads2k

@sttts

Automatic merge from submit-queue apiextensions: validation for customresources - [x] Add types for validation of CustomResources - [x] Fix conversion-gen: #49747 - [x] Fix defaulter-gen: kubernetes/gengo#61 - [x] Convert to OpenAPI types - [x] Validate CR using go-openapi - [x] Validate CRD Schema - [x] Add integration tests - [x] Fix round trip tests: #51204 - [x] Add custom fuzzer functions - [x] Add custom conversion functions - [x] Fix data race while updating CRD: #50098 - [x] Add feature gate for CustomResourceValidation - [x] Fix protobuf generation Proposal: kubernetes/community#708 Additional discussion: kubernetes/kubernetes#49879, kubernetes/kubernetes#50625 **Release note**: ```release-note Add validation for CustomResources via JSON Schema. ``` /cc @sttts @deads2k Kubernetes-commit: 4457e43e7b789586096bfb564330295cf0438e70

…tion Proposal: Validation for CustomResources.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 12, 2017

sttts requested review from sttts, enisoc and deads2k June 12, 2017 16:50

This was referenced Jun 12, 2017

CustomResourceDefinitions kubernetes/enhancements#95

Closed

Validation for Custom Resource contents kubernetes/kubernetes#38117

Closed

fabiand reviewed Jun 12, 2017

View reviewed changes

nikhita force-pushed the customresources-validation branch from 59be5b4 to e7dbf3a Compare June 12, 2017 20:13

enisoc reviewed Jun 12, 2017

View reviewed changes

nikhita mentioned this pull request Jun 12, 2017

proposal: Address reviews in the PR nikhita/gsoc-meta-k8s#20

Closed

fabiand reviewed Jun 13, 2017

View reviewed changes

sttts mentioned this pull request Jun 13, 2017

Prototype: Add defaulter support go-openapi/validate#27

Merged

nikhita force-pushed the customresources-validation branch 3 times, most recently from 337eaa5 to 037a1ec Compare June 14, 2017 13:13

deads2k reviewed Jun 14, 2017

View reviewed changes

jamiehannaford reviewed Jun 22, 2017

View reviewed changes

Add proposal for validation of customresources

6188a9b

Address comments

nikhita force-pushed the customresources-validation branch from fded486 to 6188a9b Compare June 25, 2017 21:20

deads2k mentioned this pull request Jun 27, 2017

Remove TPR resource in favor of CRD kubernetes/kubernetes#48152

Closed

sttts reviewed Jul 3, 2017

View reviewed changes

k8s-ci-robot assigned deads2k Jul 5, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 5, 2017

deads2k merged commit 05d4e60 into kubernetes:master Jul 6, 2017

sttts mentioned this pull request Jul 31, 2017

Accept OpenAPI spec instead of JSON Schema for CRD validation kubernetes/kubernetes#49879

Closed

nikhita mentioned this pull request Aug 7, 2017

apiextensions: validation for customresources kubernetes/kubernetes#47263

Merged

13 tasks

nikhita mentioned this pull request Sep 4, 2017

Add docs for CustomResource validation kubernetes/website#5290

Merged

colemickens mentioned this pull request Oct 12, 2017

Getting CRD Validation to Beta kubernetes/kubernetes#53829

Closed

philipgough mentioned this pull request Nov 18, 2021

Ensure group_by values are unique in alertmanager config prometheus-operator/prometheus-operator#4413

Merged

5 tasks

MadhavJivrajani pushed a commit to MadhavJivrajani/community that referenced this pull request Nov 30, 2021

Merge pull request kubernetes#708 from nikhita/customresources-valida…

9ab370b

…tion Proposal: Validation for CustomResources.

xiao-jay mentioned this pull request Mar 19, 2023

Unable to use +kubebuilder:validation:UniqueItems=true kubernetes-sigs/kubebuilder#3300

Closed

danehans pushed a commit to danehans/community that referenced this pull request Jul 18, 2023

add ycai-aspen to member list (kubernetes#708)

5b645cc


		We do a schema pass there using the https://github.com/go-openapi/validate validator with the provided schema in the corresponding CRD. Validation errors are returned to the caller as for native resources.

		JSON-Schema also allows us to reject additional fields that are not defined in the schema and only allow the fields that are specified. This can be achieved by using `"additionalProperties": false` in the schema.


		Note: A reflective test to check for drift between the types here and the OpenAPI types for runtime usage will be added.

		## JSON-Schema Examples


		This is especially true in situations when CRs are used by components that are out of the control of the admin. Example: A user can create a database CR for a Database-As-A-Service. In this case, only server-side validation can give confidence that the CRs are well formed.

		### Existing Instances and changing the Schema


		Note: this is a workaround while we do not support multiple versions and conversion for custom resources.

		### JSON-Schema Validation Runtime Complexity


		The CRD JSON-Schema will be validated to have neither recursion, nor `uniqueItems=true` being set.

		### Alternatives


		Currently we do not provide validation for CustomResources (CR), i.e. the CR payload is free-form JSON. However, one of the most requested [[1](https://github.com/kubernetes/features/issues/95#issuecomment-296416969)][[2](https://github.com/kubernetes/features/issues/95#issuecomment-298791881)] features is validation and this proposal seeks to add it.

		## Goals


		The implementation is planned in the following steps:

		1. Add the proposed types to the v1beta1<sup id="f3">[3](#footnote3)</sup> version of the CRD type.


		The client-side validation is carried out before sending the request to the api-server, or even completely offline. This can be achieved while creating resources through the client i.e. kubectl using the --validate option.

		If the API type serves the JSON-Schema in the swagger spec, the existing kubectl code will already be able to also validate CRs. This will be achieved as a follow-up.


		1. This is the same behavior that we require for native resources. Validation cannot be made stricter in later Kubernetes versions without breaking compatibility.

		2. For migration of CRDs with no validation to CRDs with validation, we can create a controller that will validate and annotate invalid CRs once the spec changes, so that the custom controller can choose to delete them (this is also essentially the status condition of the CRD). This can be achieved, but it is not part of the proposal.


		## Background

		ThirdPartyResource (TPR) is deprecated and CustomResourceDefinition (CRD) is the successor which solves the fundamental issues of TPRs to form a stable base for further features.


		## API Types

		The schema is referenced in [`CustomResourceDefinitionSpec`](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1/types.go#L22). `Validation` is of the type `CustomResourceValidation`. The JSON-Schema is stored in a field of `Validation`. This way we can make the validation generic and add other validations in the future as well.

Proposal: Validation for CustomResources. #708

Proposal: Validation for CustomResources. #708

Conversation

nikhita commented Jun 12, 2017 • edited Loading

sttts commented Jun 12, 2017

deads2k commented Jun 12, 2017

Choose a reason for hiding this comment

enisoc commented Jun 12, 2017

enisoc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts Jun 13, 2017 • edited Loading

Choose a reason for hiding this comment

bgrant0607 commented Jun 13, 2017

bgrant0607 commented Jun 13, 2017

bgrant0607 commented Jun 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deads2k commented Jun 14, 2017

sdminonne commented Jun 14, 2017

sttts commented Jun 14, 2017

deads2k commented Jun 19, 2017

agonzalezro commented Jun 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikhita commented Jun 22, 2017

Choose a reason for hiding this comment

sttts commented Jul 3, 2017

agonzalezro commented Jul 3, 2017

deads2k commented Jul 5, 2017

frankgreco commented Jul 5, 2017

enisoc commented Jul 5, 2017

deads2k commented Jul 5, 2017

deads2k commented Jul 6, 2017

mbohlool commented Jul 31, 2017 via email

sttts commented Jul 31, 2017

nikhita commented Jun 12, 2017 •

edited

Loading

sttts Jun 13, 2017 •

edited

Loading

agonzalezro commented Jun 20, 2017 •

edited

Loading