code-generator: rewrite hack/update-codecs.sh into reusable generate-{internal,}-groups.sh #52186

sttts · 2017-09-08T17:57:58Z

Generating everything for groups inside of an apiserver (with internal types) becomes:

generate-internal-groups.sh all "$(dirname ${BASH_SOURCE})/../../.." k8s.io/sample-apiserver/pkg/client k8s.io/sample-apiserver/pkg/apis k8s.io/sample-apiserver/pkg/apis wardle:v1alpha1

Generating everything for a CRD (versioned types) becomes:

generate-groups.sh all "$(dirname ${BASH_SOURCE})/../../.." k8s.io/sample-apiserver/pkg/client k8s.io/sample-apiserver/pkg/apis wardle:v1alpha1

This should cover the 90% percent use-case. For the other 10% this can be forked and adapted as needed.

Furthermore, we can put this into a Docker container. Then code-generator consumers can then do:

$ docker run -v $GOPATH:/go k8s.io/code-generator:1.8 generate-group.sh github.com/foo/bar example:v1

This is possibly only the first step towards a code-generator binary. For the later deeper generator changes are necessary (e.g. #53202) and hence the later is only feasible in 1.9. This PR here in contrast, we can cherry-pick to 1.8.

Add generate-groups.sh and generate-internal-groups.sh to k8s.io/code-generator to easily run generators against CRD or User API Server types.

Fixes #48714.

k8s-github-robot · 2017-09-08T17:58:10Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sttts
We suggest the following additional approver: lavalamp

Assign the PR to them by writing /assign @lavalamp in a comment when ready.

No associated issue. Update pull-request body to add a reference to an issue, or get approval with /approve no-issue

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

staging/src/k8s.io/code-generator/OWNERS
~~staging/src/k8s.io/sample-apiserver/OWNERS~~ [sttts]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

sttts · 2017-09-08T18:00:08Z

/cc @mrIncompetent

k8s-ci-robot · 2017-09-08T18:00:09Z

@sttts: GitHub didn't allow me to request PR reviews from the following users: mrIncompetent.

Note that only kubernetes members can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @mrIncompetent

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mrIncompetent · 2017-09-08T19:24:53Z

This is wonderful! Very much needed!

deads2k · 2017-09-12T16:38:33Z

@eparis can you help review the bash?

sttts · 2017-09-19T11:19:05Z

/cc @colemickens @nikhita – could you take a look at this? It is especially meant to make the life of CRD users easier.

sttts · 2017-09-19T13:55:44Z

/cc @ericchiang @scottrigby @kubernetes/sig-apps-misc – review and feedback is welcome

munnerz · 2017-09-19T17:12:10Z

staging/src/k8s.io/code-generator/hack/lib/codegen.sh

+    IFS=: read G Vs <<<"${GVs}"
+
+    # enumerate versions
+    for V in ${Vs//,/ } do


I think there's a missing ; after for V in ${Vs//,/ } here. I get the following when running on OS X:

Adding it resolves the following:

/Users/james/go/src/k8s.io/kubernetes/vendor/k8s.io/code-generator/hack/lib/codegen.sh:98: parse error near 'done'

indeed, missing ;

munnerz · 2017-09-19T17:35:07Z

It'd be great if this library could also expose the --verify flag of it's underlying generators. I've found this really useful in CI, to ensure that generated files are up to date in the repo.

hongchaodeng

@sttts
Awesome work!
A shared docker container like k8s.io/code-generator:1.8 would be very useful too!

hongchaodeng · 2017-09-19T17:28:40Z

staging/src/k8s.io/code-generator/hack/lib/codegen.sh

+
+for gen in defaulter-gen conversion-gen client-gen lister-gen informer-gen deepcopy-gen conversion-gen; do
+  echo "Building ${gen}"
+  go install ${CODEGEN_PKG}/cmd/${gen}


This incurs side effect for source-ing this lib.

true. We should do this only on-demand and only once.

hongchaodeng · 2017-09-19T17:35:53Z

staging/src/k8s.io/code-generator/hack/lib/codegen.sh

+}
+
+function generate-groups() {
+  local BASE="$1" # the base packages are resolved against (empty or the vendor/ directory ${OUTPUT_PKG} is in)


I have to confess these vars are difficult to understand. Can you give some specific examples?

simplified this. BASE is usually not needed, only in the kube case where we have staging/ directories.

hongchaodeng · 2017-09-19T17:39:17Z

@sttts
Can you assign to me? I can help test the script out.

sttts · 2017-09-28T13:26:18Z

@ericchiang @munnerz addressed your comments. Moreover, I have switch from the codegen.sh library approach to two executable scripts in k8s.io/code-generator. This is more in-line with the future code-generator binary, if we decide to rewrite the shell scripts to Go.

From my side this is complete now to go into master and the be cherry-picked into 1.8. A possible Golang rewrite has to wait for 1.9.

hongchaodeng · 2017-09-28T21:04:25Z

lgtm if tests pass

…internal,}-groups.sh scripts

…ups.sh

sttts · 2017-10-04T16:08:29Z

/retest

k8s-github-robot · 2017-10-04T16:27:09Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-10-04T17:32:48Z

Automatic merge from submit-queue (batch tested with PRs 53317, 52186). If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot · 2017-10-04T17:42:15Z

@sttts: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-federation-e2e-gce	`ab37198`	link	`/test pull-kubernetes-federation-e2e-gce`
pull-kubernetes-cross	`e2753b1`	link	`/test pull-kubernetes-cross`
pull-kubernetes-e2e-gce-bazel	`96b5961`	link	`/test pull-kubernetes-e2e-gce-bazel`
pull-kubernetes-e2e-gce-etcd3	`96b5961`	link	`/test pull-kubernetes-e2e-gce-etcd3`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

…upstream-release-1.8 Automatic merge from submit-queue. Automated cherry pick of #52186 ```release-note Add generate-groups.sh and generate-internal-groups.sh to k8s.io/code-generator to easily run generators against CRD or User API Server types. ```

@sttts

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. sample-controller: add example CRD controller **What this PR does / why we need it**: Adds a sample-controller example repository fixes #52752 **Special notes for your reviewer**: This is currently based on the sttts:sttts-codegen-scripts branch and should not be merged until that is (ref #52186) **Release note**: ``` Add sample-controller repository ``` /cc @sttts @nikhita @colemickens

sttts · 2017-10-23T09:08:02Z

@pwittrock after seeing your community meeting slides with Support for bootstrapping CRD go clients, go controllers, ref docs?, would be great to get your input on these kind of changes.

@sttts

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. sample-controller: add example CRD controller **What this PR does / why we need it**: Adds a sample-controller example repository fixes #52752 **Special notes for your reviewer**: This is currently based on the sttts:sttts-codegen-scripts branch and should not be merged until that is (ref kubernetes/kubernetes#52186) **Release note**: ``` Add sample-controller repository ``` /cc @sttts @nikhita @colemickens Kubernetes-commit: 9a7800f7d2efb88b397674672ac56f898826cf7c

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Sep 8, 2017

k8s-github-robot assigned lavalamp and smarterclayton Sep 8, 2017

k8s-github-robot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Sep 8, 2017

sttts assigned caesarxuchao and deads2k and unassigned smarterclayton Sep 8, 2017

sttts added kind/enhancement kind/documentation Categorizes issue or PR as related to documentation. labels Sep 8, 2017

sttts assigned ncdc Sep 8, 2017

sttts force-pushed the sttts-codegen-scripts branch from 86d2bc7 to ab37198 Compare September 9, 2017 10:25

mbohlool self-assigned this Sep 11, 2017

lavalamp unassigned lavalamp and caesarxuchao Sep 11, 2017

k8s-ci-robot added the sig/apps Categorizes an issue or PR as relevant to SIG Apps. label Sep 19, 2017

munnerz reviewed Sep 19, 2017

View reviewed changes

hongchaodeng reviewed Sep 19, 2017

View reviewed changes

sttts mentioned this pull request Sep 28, 2017

Remove implicit magic with flag registration in generators kubernetes/gengo#80

Merged

sttts force-pushed the sttts-codegen-scripts branch from e2753b1 to e56003d Compare September 28, 2017 13:20

sttts changed the title ~~code-generator: rewrite hack/update-codecs.sh into a reusable library~~ code-generator: rewrite hack/update-codecs.sh into reusable generate-{internal,}-groups.sh Sep 28, 2017

sttts force-pushed the sttts-codegen-scripts branch 2 times, most recently from ce13dc6 to 590d493 Compare September 28, 2017 14:11

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 28, 2017

sttts added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Sep 28, 2017

sttts force-pushed the sttts-codegen-scripts branch from 590d493 to 189dfba Compare October 4, 2017 14:50

sttts added 5 commits October 4, 2017 16:53

code-generator: rename _test to _examples

8a98063

code-generator: turn hack/update-codegen.sh into re-usable generate-{…

cba2c09

…internal,}-groups.sh scripts

sample-apiserver: port to k8s.io/code-generator/generate-internal-gro…

abfb658

…ups.sh

verify-pkg-names.sh: exclude generated informers

873a550

Update generated files

96b5961

sttts force-pushed the sttts-codegen-scripts branch from 189dfba to 96b5961 Compare October 4, 2017 14:53

sttts added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 4, 2017

k8s-github-robot merged commit 1357cef into kubernetes:master Oct 4, 2017

This was referenced Oct 19, 2017

Update Kubernetes deps to use release-1.8 branches jetstack/navigator#47

Merged

conversion-gen --extra-peer-dirs references k8s.io/kubernetes types #54301

Closed

munnerz mentioned this pull request Oct 27, 2017

Provide binary release of code generators kubernetes/code-generator#15

Closed

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

code-generator: rewrite hack/update-codecs.sh into reusable generate-{internal,}-groups.sh #52186

code-generator: rewrite hack/update-codecs.sh into reusable generate-{internal,}-groups.sh #52186

sttts commented Sep 8, 2017 •

edited

Loading

k8s-github-robot commented Sep 8, 2017

sttts commented Sep 8, 2017

k8s-ci-robot commented Sep 8, 2017

mrIncompetent commented Sep 8, 2017

deads2k commented Sep 12, 2017

sttts commented Sep 19, 2017

sttts commented Sep 19, 2017 •

edited

Loading

munnerz Sep 19, 2017

sttts Sep 20, 2017

munnerz commented Sep 19, 2017

hongchaodeng left a comment

hongchaodeng Sep 19, 2017

sttts Sep 20, 2017

hongchaodeng Sep 19, 2017

sttts Sep 20, 2017

hongchaodeng commented Sep 19, 2017

sttts commented Sep 28, 2017

hongchaodeng commented Sep 28, 2017

sttts commented Oct 4, 2017

k8s-github-robot commented Oct 4, 2017

k8s-github-robot commented Oct 4, 2017

k8s-ci-robot commented Oct 4, 2017

sttts commented Oct 23, 2017

code-generator: rewrite hack/update-codecs.sh into reusable generate-{internal,}-groups.sh #52186

code-generator: rewrite hack/update-codecs.sh into reusable generate-{internal,}-groups.sh #52186

Conversation

sttts commented Sep 8, 2017 • edited Loading

k8s-github-robot commented Sep 8, 2017

sttts commented Sep 8, 2017

k8s-ci-robot commented Sep 8, 2017

mrIncompetent commented Sep 8, 2017

deads2k commented Sep 12, 2017

sttts commented Sep 19, 2017

sttts commented Sep 19, 2017 • edited Loading

munnerz Sep 19, 2017

Choose a reason for hiding this comment

sttts Sep 20, 2017

Choose a reason for hiding this comment

munnerz commented Sep 19, 2017

hongchaodeng left a comment

Choose a reason for hiding this comment

hongchaodeng Sep 19, 2017

Choose a reason for hiding this comment

sttts Sep 20, 2017

Choose a reason for hiding this comment

hongchaodeng Sep 19, 2017

Choose a reason for hiding this comment

sttts Sep 20, 2017

Choose a reason for hiding this comment

hongchaodeng commented Sep 19, 2017

sttts commented Sep 28, 2017

hongchaodeng commented Sep 28, 2017

sttts commented Oct 4, 2017

k8s-github-robot commented Oct 4, 2017

k8s-github-robot commented Oct 4, 2017

k8s-ci-robot commented Oct 4, 2017

sttts commented Oct 23, 2017

sttts commented Sep 8, 2017 •

edited

Loading

sttts commented Sep 19, 2017 •

edited

Loading