protect kubernetes community owned API groups in CRDs #1111

deads2k · 2019-06-24T13:48:32Z

API groups are organized by namespace, similar to java packages. authorization.k8s.io is one example. When users create
CRDs, they get to specify an API group and their type will be injected into that group by the kube-apiserver.

The *.k8s.io or *.kubernetes.io groups are owned by the Kubernetes community and protected by API review (see What APIs need to be reviewed,
to ensure consistency and quality. To avoid confusion in our API groups and prevent accidentally claiming a
space inside of the kubernetes API groups, the kube-apiserver needs to be updated to protect these reserved API groups.

This KEP proposes adding an api-approved.kubernetes.io annotation to CustomResourceDefinition. This is only needed if
the CRD group is k8s.io, kubernetes.io, or ends with .k8s.io, .kubernetes.io. The value should be a link to the
pull request where the API has been approved.

metadata:
  annotations:
    "api-approved.kubernetes.io": "https://github.com/kubernetes/kubernetes/pull/78458"

/assign @jpbetz @liggitt @sttts

@kubernetes/sig-api-machinery-api-reviews @kubernetes/sig-architecture-api-reviews

fejta-bot · 2019-06-24T14:37:03Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

keps/sig-api-machinery/20190612-crd-group-protection.md

sttts · 2019-06-25T08:19:16Z

Sgtm. Some clarification of the PR link would be good.

keps/sig-api-machinery/20190612-crd-group-protection.md

deads2k · 2019-06-26T17:50:48Z

updated for comments.

keps/sig-api-machinery/20190612-crd-group-protection.md

liggitt · 2019-07-02T13:58:15Z

one addition requested in #1111 (comment)

overall approach looks good to me

timothysc

Not really a fan of yet another annotation

keps/sig-api-machinery/20190612-crd-group-protection.md

timothysc · 2019-07-02T17:22:24Z

TL;DR this should really be policy, then enforcement can be made from said policy.

deads2k · 2019-07-02T17:48:52Z

TL;DR this should really be policy, then enforcement can be made from said policy.

This seems unnecessary if the above policy is spelled out in full.
Right now I think we need well defined policy in the community.

I think that is exactly what this KEP is and does.

We've had a well-defined policy for 11 months here: kubernetes/community#2433 . It was written, discussed, and reviewed thoroughly. The fact that it's apparently not well known seems to be an issue of enforcement.

This KEP takes the policy and provides a simple enforcement mechanism that makes these standards unignore-able without adding significant friction along the way.

timothysc · 2019-07-02T17:54:54Z

We've had a well-defined policy for 11 months here: kubernetes/community#2433 . It was written, discussed, and reviewed thoroughly. The fact that it's apparently not well known seems to be an issue of enforcement.

It is well defined for core api's but it negates the larger ecosystem, which is where I think policy should be written then made enforceable.

We currently do not have any recommended guidelines for the community of non-core CRDs that are being published around k8s core.

liggitt · 2019-07-02T17:57:32Z

It is well defined for core api's but it negates the larger ecosystem, which is where I think policy should be written then made enforceable.

https://github.com/kubernetes/community/blob/master/sig-architecture/api-review-process.md#what-apis-need-to-be-reviewed is the written policy that covers *.k8s.io APIs, regardless of whether they are CRD-based or not.

This is the proposal for the mechanism to make that policy enforceable.

timothysc · 2019-07-02T18:04:25Z

https://github.com/kubernetes/community/blob/master/sig-architecture/api-review-process.md#what-apis-need-to-be-reviewed is the written policy that covers *.k8s.io APIs, regardless of whether they are CRD-based or not.

Then how does this impact the rest of the community that may overlap with that namespace, but have not changed yet? What are the recommendations wrt to naming?

If the current policy does not take that question into account, which it does not, I'd assert it requires refinement.

BenTheElder · 2019-07-02T19:14:30Z

This is a really interesting idea... I have some questions / concerns:

What do we do about projects that unknowingly violated this?

How can we make this more discoverable? I had no idea that this review guidelines doc existed, I suspect many others in the project don't either, especially those working on SIG projects.

Can we recommend namespacing guidelines for SIG projects that are unlikely to be core APIs but might want to use CRD storage and might not be the best use of time to review?

EG if I add some CRDs to sigs.k8s.io/slack-infra for some kubernetes.slack.com configuration, or the "component config" style config for sigs.k8s.io/kind... I sort of doubt API reviewing these is the most productive route vs some alternate API group, but I don't know what the correct namespace would be would be.

…#1111.

Release 0.9.0 and change the kongs CRD API group from charts.helm.k8s.io to charts.konghq.com. The new group is not in one of the protected groups established by kubernetes/enhancements#1111. This operator CRD should not use a protected group as it is not a core part of the Kubernetes project. This change makes the CRD compatible with Kubernetes >=1.22. However, it breaks compatibility with previous versions of the operator. As such, 0.9.0 has no replace version: it requires a fresh operator install and a fresh set of Kong CRs.

* charts(kong): update to kong-2.4.0 generator command: 'kong-2.4.0' generator command version: 2b9dc2a * release(v0.9.0) update operator metadata and docs Release 0.9.0 and change the kongs CRD API group from charts.helm.k8s.io to charts.konghq.com. The new group is not in one of the protected groups established by kubernetes/enhancements#1111. This operator CRD should not use a protected group as it is not a core part of the Kubernetes project. This change makes the CRD compatible with Kubernetes >=1.22. However, it breaks compatibility with previous versions of the operator. As such, 0.9.0 has no replace version: it requires a fresh operator install and a fresh set of Kong CRs. * test: update microk8s to 1.22 * test: update kubectl to 1.22.2 * test: update Ingress API version * feat: support Ingress v1 * test: remove Ingress waits Remove the Ingress status waits and add retry configuration to curl when validating the Ingress configuration. KIC 2.0+ handles status updates for non-LoadBalancer Services differently than earlier versions. Previously, KIC would set a status with a 0-length list of ingresses if the proxy Service was not type Loadbalancer, e.g. status: loadBalancer: ingress: - {} As of KIC 2.0, no status is set if the Service is not type LoadBalancer, e.g. status: loadBalancer: {} This change to the operator tests confirms that Ingress configuration was successfully applied to the proxy using requests through the proxy only. These now run immediately after the upstream Deployment becomes available, however, so they may run before the controller has ingested Ingress configuration or observed Endpoint updates. To account for this, the curl checks are now wrapped in wait_for to allow a reasonable amount of time for the controller to update configuration. * fix: update ingress example in README.md to v1 * feat: update OLM maintainer info Co-authored-by: Shane Utt <shaneutt@linux.com> Co-authored-by: Michał Flendrich <michal@flendrich.pro>

apiextensions.k8s.io/v1beta1 is removed as of K8s v1.22 [1], so all CRDs have to be updated to apiextensions.k8s.io/v1. This commit does the upgrade for the turndownschedule CRD. As part of the API updates, K8s is enforcing things grouped under *.k8s.io to be approved [2] because they are actually supposed to be Kuberenetes community-managed APIs [3]. So this commit also changes the CRD from: turndownschedules.kubecost.k8s.io to turndownschedules.kubecost.com This is in-line with K8s rules and links to our main domain. Tested by applying cluster-turndown-full.yaml and example-schedule.yaml successfully. [1] https://cloud.google.com/kubernetes-engine/docs/deprecations/apis-1-22 [2] kubernetes/enhancements#1111 [3] kubernetes/enhancements#1111 (comment)

lowang-bh · 2023-03-28T08:36:30Z

Hi， I have an old crd with group "scheduling.incubator.k8s.io", how can i pass the check and work for me? I add annotation api-approved.kubernetes.io: "unapproved, experimental-only", is it ok?

following is my part of crd definations

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: queues.scheduling.incubator.k8s.io
  annotations:
    api-approved.kubernetes.io: "unapproved, experimental-only"
spec:
  group: scheduling.incubator.k8s.io
  names:
    kind: Queue
    listKind: QueueList
    plural: queues
    shortNames:
    - q
    singular: queue
  scope: Cluster

Not exporting the cluster-scoped part yet, not sure about ability to support it. The CRDs and APIExports were produced as follows. These were produced by the following bashery. The following function converts a `kubectl api-resources` listing into a listing of arguments to the kcp crd-puller. ```bash function rejigger() { if [[ $# -eq 4 ]] then gv="$2" else gv="$3" fi case "$gv" in (*/*) group=.$(echo "$gv" | cut -f1 -d/) ;; (*) group="" esac echo "${1}$group" } ``` With `kubectl` configured to manipulate a kcp workspace, the following command captures the listing of resources built into that kcp workspace. ```bash kubectl api-resources | grep -v APIVERSION | while read line; do rejigger $line; done > /tmp/kcp-rgs.txt ``` With `kubectl` configured to manipulate a kind cluster, the following commands capture the resource listing split into namespaced and cluster-scoped. ```bash kubectl api-resources | grep -v APIVERSION | grep -w true | while read line; do rejigger $line; done > /tmp/kind-ns-rgs.txt kubectl api-resources | grep -v APIVERSION | grep -w false | while read line; do rejigger $line; done > /tmp/kind-cs-rgs.txt ``` With CWD=config/kube/exports/namespaced, ```bash crd-puller --kubeconfig $KUBECONFIG $(grep -v -f /tmp/kcp-rgs.txt /tmp/kind-ns-rgs.txt) ``` With CWD=config/kube/exports/cluster-scoped, ```bash crd-puller --kubeconfig $KUBECONFIG $(grep -v -f /tmp/kcp-rgs.txt /tmp/kind-cs-rgs.txt) ``` Sadly, kubernetes/kubernetes#118698 is a thing. So I manually hacked the CRD for jobs. Sadly, the filenames produced by the crd-puller are not loved by apigen. The following function renames one file as needed. ```bash function fixname() { rg=${1%%.yaml} case $rg in (*.*) g=$(echo $rg | cut -d. -f2-) r=$(echo $rg | cut -d. -f1);; (*) g=core.k8s.io r=$rg;; esac mv ${rg}.yaml ${g}_${r}.yaml } ``` In each of those CRD directories, ```bash for fn in *.yaml; do fixname $fn; done ``` Penultimately, with CWD=config/kube, ```bash ../../hack/tools/apigen --input-dir crds/namespaced --output-dir exports/namespaced ../../hack/tools/apigen --input-dir crds/cluster-scoped --output-dir exports/cluster-scoped ``` Finally, kubernetes/enhancements#1111 applies to APIExport/APIBinding as well as to CRDs. And the CRD puller does not know anything about this (not that it would help?). I manually hacked the namespaced APIResource files that needed it to have an `api-approved.kubernetes.io` annotation. It turns out that the checking in the apiserver only requires that the annotation's value parse as a URL (any URL will do). Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

Not exporting the cluster-scoped part yet, not sure about ability to support it. Updated example1 to exercise this by switching the common workload from a Deployment object to a ReplicaSet object. The CRDs and APIExports were produced as follows. These were produced by the following bashery. The following function converts a `kubectl api-resources` listing into a listing of arguments to the kcp crd-puller. ```bash function rejigger() { if [[ $# -eq 4 ]] then gv="$2" else gv="$3" fi case "$gv" in (*/*) group=.$(echo "$gv" | cut -f1 -d/) ;; (*) group="" esac echo "${1}$group" } ``` With `kubectl` configured to manipulate a kcp workspace, the following command captures the listing of resources built into that kcp workspace. ```bash kubectl api-resources | grep -v APIVERSION | while read line; do rejigger $line; done > /tmp/kcp-rgs.txt ``` With `kubectl` configured to manipulate a kind cluster, the following commands capture the resource listing split into namespaced and cluster-scoped. ```bash kubectl api-resources | grep -v APIVERSION | grep -w true | while read line; do rejigger $line; done > /tmp/kind-ns-rgs.txt kubectl api-resources | grep -v APIVERSION | grep -w false | while read line; do rejigger $line; done > /tmp/kind-cs-rgs.txt ``` With CWD=config/kube/exports/namespaced, ```bash crd-puller --kubeconfig $KUBECONFIG $(grep -v -f /tmp/kcp-rgs.txt /tmp/kind-ns-rgs.txt) ``` With CWD=config/kube/exports/cluster-scoped, ```bash crd-puller --kubeconfig $KUBECONFIG $(grep -v -f /tmp/kcp-rgs.txt /tmp/kind-cs-rgs.txt) ``` Sadly, kubernetes/kubernetes#118698 is a thing. So I manually hacked the CRD for jobs. Sadly, the filenames produced by the crd-puller are not loved by apigen. The following function renames one file as needed. ```bash function fixname() { rg=${1%%.yaml} case $rg in (*.*) g=$(echo $rg | cut -d. -f2-) r=$(echo $rg | cut -d. -f1);; (*) g=core.k8s.io r=$rg;; esac mv ${rg}.yaml ${g}_${r}.yaml } ``` In each of those CRD directories, ```bash for fn in *.yaml; do fixname $fn; done ``` Penultimately, with CWD=config/kube, ```bash ../../hack/tools/apigen --input-dir crds/namespaced --output-dir exports/namespaced ../../hack/tools/apigen --input-dir crds/cluster-scoped --output-dir exports/cluster-scoped ``` Finally, kubernetes/enhancements#1111 applies to APIExport/APIBinding as well as to CRDs. And the CRD puller does not know anything about this (not that it would help?). I manually hacked the namespaced APIResource files that needed it to have an `api-approved.kubernetes.io` annotation. It turns out that the checking in the apiserver only requires that the annotation's value parse as a URL (any URL will do). Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

Not exporting the cluster-scoped part yet, not sure about ability to support it. Updated example1 to exercise this by switching the common workload from a Deployment object to a ReplicaSet object. Also updated example1 to use `kubestellar init` because that now does a lot more than just create one workspace and do one `kubectl apply`. The CRDs and APIExports were produced as follows. These were produced by the following bashery. The following function converts a `kubectl api-resources` listing into a listing of arguments to the kcp crd-puller. ```bash function rejigger() { if [[ $# -eq 4 ]] then gv="$2" else gv="$3" fi case "$gv" in (*/*) group=.$(echo "$gv" | cut -f1 -d/) ;; (*) group="" esac echo "${1}$group" } ``` With `kubectl` configured to manipulate a kcp workspace, the following command captures the listing of resources built into that kcp workspace. ```bash kubectl api-resources | grep -v APIVERSION | while read line; do rejigger $line; done > /tmp/kcp-rgs.txt ``` With `kubectl` configured to manipulate a kind cluster, the following commands capture the resource listing split into namespaced and cluster-scoped. ```bash kubectl api-resources | grep -v APIVERSION | grep -w true | while read line; do rejigger $line; done > /tmp/kind-ns-rgs.txt kubectl api-resources | grep -v APIVERSION | grep -w false | while read line; do rejigger $line; done > /tmp/kind-cs-rgs.txt ``` With CWD=config/kube/exports/namespaced, ```bash crd-puller --kubeconfig $KUBECONFIG $(grep -v -f /tmp/kcp-rgs.txt /tmp/kind-ns-rgs.txt) ``` With CWD=config/kube/exports/cluster-scoped, ```bash crd-puller --kubeconfig $KUBECONFIG $(grep -v -f /tmp/kcp-rgs.txt /tmp/kind-cs-rgs.txt) ``` I manually deleted the four CRDs from https://github.com/kcp-dev/kcp/tree/v0.11.0/config/rootcompute/kube-1.24 . Sadly, kubernetes/kubernetes#118698 is a thing. So I manually hacked the CRD for jobs. Sadly, the filenames produced by the crd-puller are not loved by apigen. The following function renames one file as needed. ```bash function fixname() { rg=${1%%.yaml} case $rg in (*.*) g=$(echo $rg | cut -d. -f2-) r=$(echo $rg | cut -d. -f1);; (*) g=core.k8s.io r=$rg;; esac mv ${rg}.yaml ${g}_${r}.yaml } ``` In each of those CRD directories, ```bash for fn in *.yaml; do fixname $fn; done ``` Penultimately, with CWD=config/kube, ```bash ../../hack/tools/apigen --input-dir crds/namespaced --output-dir exports/namespaced ../../hack/tools/apigen --input-dir crds/cluster-scoped --output-dir exports/cluster-scoped ``` Finally, kubernetes/enhancements#1111 applies to APIExport/APIBinding as well as to CRDs. And the CRD puller does not know anything about this (not that it would help?). I manually hacked the namespaced APIResource files that needed it to have an `api-approved.kubernetes.io` annotation. It turns out that the checking in the apiserver only requires that the annotation's value parse as a URL (any URL will do). Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

palnabarun · 2023-10-27T15:16:29Z

For anyone reading this in future, the KEP now lives here: https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/2337-k8s.io-group-protection

protect kubernetes community owned API groups in CRDs

0ddc786

k8s-ci-robot assigned jpbetz, liggitt and sttts Jun 24, 2019

k8s-ci-robot requested review from derekwaynecarr and jdumars June 24, 2019 13:48

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 24, 2019

sttts reviewed Jun 25, 2019

View reviewed changes

keps/sig-api-machinery/20190612-crd-group-protection.md Outdated Show resolved Hide resolved

liggitt reviewed Jun 26, 2019

View reviewed changes

keps/sig-api-machinery/20190612-crd-group-protection.md Show resolved Hide resolved

clarify the URL for CRD

c80a1d1

liggitt reviewed Jul 2, 2019

View reviewed changes

keps/sig-api-machinery/20190612-crd-group-protection.md Show resolved Hide resolved

liggitt mentioned this pull request Jul 2, 2019

promote CRD to v1 kubernetes/kubernetes#79604

Merged

CRD group protection: add details about how to handle a conflict

4dfe009

timothysc reviewed Jul 2, 2019

View reviewed changes

keps/sig-api-machinery/20190612-crd-group-protection.md Show resolved Hide resolved

keps/sig-api-machinery/20190612-crd-group-protection.md Show resolved Hide resolved

sabw8217 pushed a commit to sabw8217/flink-on-k8s-operator that referenced this pull request Oct 8, 2021

Add api-approved.kubernetes.io annotation per kubernetes/enhancements…

ea82e62

…#1111.

sabw8217 mentioned this pull request Oct 8, 2021

CRD does not include an api-approved.kubernetes.io annotation. spotify/flink-on-k8s-operator#125

Closed

rainest mentioned this pull request Oct 16, 2021

operators kong (0.9.0) k8s-operatorhub/community-operators#300

Merged

19 tasks

rbogendoerfer mentioned this pull request Jan 13, 2022

addons deployment (1.9.1) fails on Kubernetes version 1.22 pegasystems/pega-helm-charts#376

Closed

alculquicondor mentioned this pull request Feb 18, 2022

Graduate API to beta kubernetes-sigs/kueue#23

Closed

4 tasks

kvaps mentioned this pull request Mar 28, 2022

[snapshot-controller] New module deckhouse/deckhouse#1068

Merged

ringtail mentioned this pull request Apr 2, 2022

[BUGFIX] Fix v1.22 crd validation AliyunContainerService/spark-on-k8s-operator#39

Merged

astefanutti mentioned this pull request May 19, 2022

api: Enables the binding of APIs in protected groups kcp-dev/kcp#1104

Merged

4 tasks

michaelmdresser mentioned this pull request May 27, 2022

Upgrade turndownschedules CRD to apiextensions v1 kubecost/cluster-turndown#44

Merged

BenTheElder mentioned this pull request Aug 18, 2022

Add extra labels section for per-node options kubernetes-sigs/kind#2877

Merged

tssurya mentioned this pull request Sep 13, 2022

instructions for installing the CRDs kubernetes-sigs/network-policy-api#38

Closed

npintaux mentioned this pull request Mar 28, 2023

Published CRD yaml file is missing annotation kubernetes-sigs/about-api#17

Open

junm-cloudnatix mentioned this pull request May 23, 2023

Switch to apiextensions/v1. cloudnatix/k8s-rds#1

Merged

MikeSpreitzer mentioned this pull request Jun 16, 2023

🐛 Make root:compute export the rest of namespaced kube API kubestellar/kubestellar#612

Merged

github-actions bot mentioned this pull request Oct 13, 2023

operator [R] cockroachdb (2.0.9 2.1.1 2.1.11 3.0.7) redhat-openshift-ecosystem/community-operators-pipeline#428

Closed

MikeSpreitzer mentioned this pull request Dec 4, 2023

✨ Use crds when ensuring wmw kubestellar/kubestellar#1367

Merged

danilo-gemoli mentioned this pull request Mar 8, 2024

scheduler: update ProwJob CRD kubernetes/test-infra#32211

Merged

pinkfloydx33 mentioned this pull request May 17, 2024

[bitnami/external-dns] External-DNS Chart Broken with 7.3.3 due to removal of protected CRD group annotation bitnami/charts#25967

Open

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

protect kubernetes community owned API groups in CRDs #1111

protect kubernetes community owned API groups in CRDs #1111

deads2k commented Jun 24, 2019

fejta-bot commented Jun 24, 2019

sttts commented Jun 25, 2019

deads2k commented Jun 26, 2019

liggitt commented Jul 2, 2019

timothysc left a comment

timothysc commented Jul 2, 2019

deads2k commented Jul 2, 2019

timothysc commented Jul 2, 2019 •

edited

Loading

liggitt commented Jul 2, 2019

timothysc commented Jul 2, 2019 •

edited

Loading

BenTheElder commented Jul 2, 2019

lowang-bh commented Mar 28, 2023 •

edited

Loading

palnabarun commented Oct 27, 2023

protect kubernetes community owned API groups in CRDs #1111

protect kubernetes community owned API groups in CRDs #1111

Conversation

deads2k commented Jun 24, 2019

fejta-bot commented Jun 24, 2019

sttts commented Jun 25, 2019

deads2k commented Jun 26, 2019

liggitt commented Jul 2, 2019

timothysc left a comment

Choose a reason for hiding this comment

timothysc commented Jul 2, 2019

deads2k commented Jul 2, 2019

timothysc commented Jul 2, 2019 • edited Loading

liggitt commented Jul 2, 2019

timothysc commented Jul 2, 2019 • edited Loading

BenTheElder commented Jul 2, 2019

lowang-bh commented Mar 28, 2023 • edited Loading

palnabarun commented Oct 27, 2023

timothysc commented Jul 2, 2019 •

edited

Loading

timothysc commented Jul 2, 2019 •

edited

Loading

lowang-bh commented Mar 28, 2023 •

edited

Loading