diff --git a/site/content/en/docs/Components/_index.md b/site/content/en/docs/Components/_index.md deleted file mode 100644 index 673a539d35..0000000000 --- a/site/content/en/docs/Components/_index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: "Components" -weight: 2 ---- - -## What are Prow components? - -Prow is made up of a collection of microservices, or "components". diff --git a/site/content/en/docs/Legacy Snapshot/prow/ANNOUNCEMENTS.md b/site/content/en/docs/Legacy Snapshot/prow/ANNOUNCEMENTS.md deleted file mode 100644 index 99c34a1331..0000000000 --- a/site/content/en/docs/Legacy Snapshot/prow/ANNOUNCEMENTS.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: "prow/ANNOUNCEMENTS.md" ---- - -# Announcements - -Please see [this page](/docs/announcements) for the new home of Prow announcements. diff --git a/site/content/en/docs/Legacy Snapshot/prow/README.md b/site/content/en/docs/Legacy Snapshot/prow/README.md deleted file mode 100644 index 0da3d92370..0000000000 --- a/site/content/en/docs/Legacy Snapshot/prow/README.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -title: "prow/README.md" ---- - -# ![Prow](https://github.com/kubernetes/test-infra/tree/master/prow/logo_horizontal_solid.png) - -Prow is a Kubernetes based CI/CD system. Jobs can be triggered by various types of events and report their status to many different services. In addition to job execution, Prow provides GitHub automation in the form of policy enforcement, chat-ops via `/foo` style commands, and automatic PR merging. - -See the [GoDoc](https://godoc.org/k8s.io/test-infra/prow) for library docs. -Please note that these libraries are intended for use by prow only, and we do -not make any attempt to preserve backwards compatibility. - -For a brief overview of how Prow runs jobs take a look at ["Life of a Prow Job"](https://github.com/kubernetes/test-infra/tree/master/prow/life_of_a_prow_job.md). - -To see common Prow usage and interactions flow, see the pull request interactions [sequence diagram](https://raw.githubusercontent.com/kubernetes/test-infra/master/prow/docs/pr-interactions-sequence.svg?sanitize=true). - -#### Functions and Features -* Job execution for testing, batch processing, artifact publishing. - * GitHub events are used to trigger post-PR-merge (postsubmit) jobs and on-PR-update (presubmit) jobs. - * Support for multiple execution platforms and source code review sites. -* Pluggable GitHub bot automation that implements `/foo` style commands and enforces configured policies/processes. -* GitHub merge automation with batch testing logic. -* Front end for viewing jobs, merge queue status, dynamically generated help information, and more. -* Automatic deployment of source control based config. -* Automatic GitHub org/repo administration configured in source control. -* Designed for multi-org scale with dozens of repositories. (The Kubernetes Prow instance uses only 1 GitHub bot token!) -* High availability as benefit of running on Kubernetes. (replication, load balancing, rolling updates...) -* JSON structured logs. -* Prometheus metrics. - - -## Documentation - -### Getting started - -* With your own Prow deployment: [`getting_started_deploy.md`](https://github.com/kubernetes/test-infra/tree/master/prow/getting_started_deploy.md) -* With developing for Prow: [`getting_started_develop.md`](https://github.com/kubernetes/test-infra/tree/master/prow/getting_started_develop.md) -* As a job author: [`jobs.md`](https://github.com/kubernetes/test-infra/tree/master/prow/jobs.md) - -### More details -- [Components](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/README.md) -- [Plugins](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/README.md) -- [ProwJobs](https://github.com/kubernetes/test-infra/tree/master/prow/jobs.md) -- [Building, Testing, and Updating](https://github.com/kubernetes/test-infra/tree/master/prow/build_test_update.md) -- [General Configuration](https://github.com/kubernetes/test-infra/tree/master/prow/config/README.md) -- [Pod Utilities](https://github.com/kubernetes/test-infra/tree/master/prow/pod-utilities.md) -- [Scaling Prow](https://github.com/kubernetes/test-infra/tree/master/prow/scaling.md) -- [Tide](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide/README.md) -- [Metrics](https://github.com/kubernetes/test-infra/tree/master/prow/metrics/README.md) -- ["Life of a Prow Job"](https://github.com/kubernetes/test-infra/tree/master/prow/life_of_a_prow_job.md) -- [Getting more out of Prow](https://github.com/kubernetes/test-infra/tree/master/prow/more_prow.md) - -### Tests - -The stability of prow is heavily relying on unit tests and integration tests. - -* Unit tests are co-located with prow source code -* [Integration tests](https://github.com/kubernetes/test-infra/tree/master/prow/test/integration) utilizes [kind](https://kind.sigs.k8s.io/) with hermetic integration tests. See [instructions for adding new integration tests](https://github.com/kubernetes/test-infra/tree/master/prow/test/integration#add-new-integration-tests) for more details - -## Useful Talks - -### KubeCon 2020 EU virtual - - - [Going Beyond CI/CD with Prow](https://youtu.be/qQvoImxHydk) - -### KubeCon 2018 EU - -- [Automation and the Kubernetes Contributor Experience](https://www.youtube.com/watch?v=BsIC7gPkH5M) -- [SIG Testing Deep Dive](https://www.youtube.com/watch?v=M32NIHRKaOI) - -### KubeCon 2018 China - -- [SIG Testing Intro](https://youtu.be/WFvC_VdkDFk) - -### KubeCon 2018 Seattle - -- [Behind you PR: K8s with K8s on K8s](https://www.youtube.com/watch?v=pz0lpl6h-Gc) -- [Using Prow for Testing Outside of K8s](https://www.youtube.com/watch?v=DBrkSC6nS8A) -- [Jenkins X (featuring Tide)](https://www.youtube.com/watch?v=IDEa8seAzVc) -- [SIG Testing Intro](https://www.youtube.com/watch?v=7-_O41W3FRU) -- [SIG Testing Deep Dive](https://www.youtube.com/watch?v=1rwiKDTJILY) - -### Misc - -- [Deploy Your Own Kubernetes Prow](https://www.youtube.com/watch?v=eMNwB96A1Qc) - - -## Prow in the wild - -Prow is used by the following organizations and projects: -- [Kubernetes](https://prow.k8s.io) - - This includes [kubernetes](https://github.com/kubernetes), [kubernetes-client](https://github.com/kubernetes-client), [kubernetes-csi](https://github.com/kubernetes-csi), and [kubernetes-sigs](https://github.com/kubernetes-sigs). -- [OpenShift](https://prow.ci.openshift.org/) - - This includes [openshift](https://github.com/openshift), [openshift-s2i](https://github.com/openshift-s2i), [operator-framework](https://github.com/operator-framework), and some repos in [containers](https://github.com/containers) and [heketi](https://github.com/heketi). -- [Istio](https://prow.istio.io/) -- [Knative](https://prow.knative.dev/) -- [Jetstack](https://prow.build-infra.jetstack.net/) -- [Kyma](https://status.build.kyma-project.io/) -- [Metal³](https://prow.apps.test.metal3.io/) -- [Prometheus](http://prombench.prometheus.io/) -- [Caicloud](https://github.com/caicloud) -- [Kubeflow](https://github.com/kubeflow) -- [Azure AKS Engine](https://github.com/Azure/aks-engine/tree/master/.prowci) -- [tensorflow/minigo](https://github.com/tensorflow/minigo#automated-tests) -- [Daisy (Google Compute Image Tools)](https://github.com/GoogleCloudPlatform/compute-image-tools/tree/master/test-infra#prow-and-gubenator) -- [KubeEdge (Kubernetes Native Edge Computing Framework)](https://github.com/kubeedge/kubeedge) -- [Volcano (Kubernetes Native Batch System)](https://github.com/volcano-sh/volcano) -- [Loodse](https://public-prow.loodse.com/) -- [Feast](https://github.com/gojek/feast) -- [Falco](http://prow.falco.org) -- [TiDB](https://prow.tidb.io) -- [Amazon EKS Distro and Amazon EKS Anywhere](https://prow.eks.amazonaws.com/) -- [KubeSphere](https://prow.kubesphere.io) -- [OpenYurt](https://github.com/openyurtio/openyurt) -- [KubeVirt](https://prow.ci.kubevirt.io/) -- [AWS Controllers for Kubernetes](https://prow.ack.aws.dev/) - -[Jenkins X](https://jenkins-x.io/) uses [Prow as part of Serverless Jenkins](https://medium.com/@jdrawlings/serverless-jenkins-with-jenkins-x-9134cbfe6870). - -## Contact us - -If you need to contact the maintainers of Prow you have a few options: -1. Open an issue in the [kubernetes/test-infra](https://github.com/kubernetes/test-infra) repo. -1. Reach out to the `#prow` channel of the [Kubernetes Slack](https://github.com/kubernetes/community/tree/master/communication#social-media). -1. Contact one of the code owners in [prow/OWNERS](https://github.com/kubernetes/test-infra/tree/master/prow/OWNERS) or in a more specifically scoped OWNERS file. - -### Bots home -[@k8s-ci-robot](https://github.com/k8s-ci-robot) lives here and is the face of the Kubernetes Prow instance. Here is a [command list](https://go.k8s.io/bot-commands) for interacting with @k8s-ci-robot and other Prow bots. diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/README.md b/site/content/en/docs/Legacy Snapshot/prow/cmd/README.md deleted file mode 100644 index 8dda042882..0000000000 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/README.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -title: "prow/cmd/README.md" ---- - -# Prow Images - -This directory includes a sub directory for every Prow component and is where all binary and container images are built. You can find the `main` packages here. For details about building the binaries and images see [`build_test_update.md`](https://github.com/kubernetes/test-infra/tree/master/prow/build_test_update.md). - -## Cluster Components - -Prow has a microservice architecture implemented as a collection of container images that run as Kubernetes deployments. A brief description of each service component is provided here. - -#### Core Components - -* [`crier`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/crier) reports on ProwJob status changes. Can be configured to report to gerrit, github, pubsub, slack, etc. -* [`deck`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/deck) presents a nice view of [recent jobs](https://prow.k8s.io/), [command](https://prow.k8s.io/command-help) and [plugin](https://prow.k8s.io/plugins) help information, the [current status](https://prow.k8s.io/tide) and [history](https://prow.k8s.io/tide-history) of merge automation, and a [dashboard for PR authors](https://prow.k8s.io/pr). -* [`hook`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/hook) is the most important piece. It is a stateless server that listens for GitHub webhooks and dispatches them to the appropriate plugins. Hook's plugins are used to trigger jobs, implement 'slash' commands, post to Slack, and more. See the [`prow/plugins`](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/) directory for more information on plugins. -* [`horologium`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/horologium) triggers periodic jobs when necessary. -* [`prow-controller-manager`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/prow-controller-manager) manages the job execution and lifecycle for jobs that run in k8s pods. It currently acts as a replacement for [`plank`](https://github.com/kubernetes/test-infra/tree/master/prow/plank) -* [`sinker`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/sinker) cleans up old jobs and pods. - -#### Merge Automation - -* [`tide`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide) manages retesting and merging PRs once they meet the configured merge criteria. See [its README](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide/README.md) for more information. - -#### Optional Components - -* [`branchprotector`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/branchprotector) configures [github branch protection] according to a specified policy -* [`exporter`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/exporter) exposes metrics about ProwJobs not directly related to a specific Prow component -* [`gerrit`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/gerrit) is a Prow-gerrit adapter for handling CI on [gerrit] workflows -* [`hmac`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/hmac) updates HMAC tokens, GitHub webhooks and HMAC secrets for the orgs/repos specified in the Prow config file -* [`jenkins-operator`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/jenkins-operator) is the controller that manages jobs that run on Jenkins. We moved away from using this component in favor of running all jobs on Kubernetes. -* [`tot`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tot) vends sequential build numbers. Tot is only necessary for integration with automation that expects sequential build numbers. If Tot is not used, Prow automatically generates build numbers that are monotonically increasing, but not sequential. -* [`status-reconciler`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/status-reconciler) ensures changes to blocking presubmits in Prow configuration does not cause in-flight GitHub PRs to get stuck -* [`sub`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/sub) listen to Cloud Pub/Sub notification to trigger Prow Jobs. - -## CLI Tools - -* [`checkconfig`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/checkconfig) loads and verifies the configuration, useful as a pre-submit. -* [`config-bootstrapper`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/config-bootstrapper) bootstraps a configuration that would be incrementally updated by the [`updateconfig` Prow plugin] -* [`generic-autobumper`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/generic-autobumper) automates image version upgrades (e.g. for a Prow deployment) by opening a PR with images changed to their latest version according to a config file. -* [`invitations-accepter`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/invitations-accepter) approves all pending GitHub repository invitations -* [`mkpj`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/mkpj) creates `ProwJobs` using Prow configuration. -* [`mkpod`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/mkpod) creates `Pods` from `ProwJobs`. -* [`peribolos`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/peribolos) manages GitHub org, team and membership settings according to a config file. Used by [kubernetes/org] -* [`phaino`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/phaino) runs an approximation of a ProwJob on your local workstation -* [`phony`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/phony) sends fake webhooks for testing hook and plugins. - -## Pod Utilities - -These are small tools that are automatically added to ProwJob pods for jobs that request pod decoration. They are used to transparently provide source code cloning and upload of metadata, logs, and job artifacts to persistent storage. See [their README](https://github.com/kubernetes/test-infra/tree/master/prow/pod-utilities.md) for more information. - -* [`clonerefs`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/clonerefs) -* [`initupload`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/initupload) -* [`entrypoint`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/entrypoint) -* [`sidecar`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/sidecar) - -## Base Images - -The container images in [`images`](https://github.com/kubernetes/test-infra/tree/master/images) are used as base images for Prow components. - -## TODO: undocumented - -* [`admission`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/admission) -* [`gcsupload`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/gcsupload) -* [`grandmatriarch`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/grandmatriarch) -* [`pipeline`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/pipeline) -* [`tackle`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tackle) - -## Deprecated - -* [`cm2kc`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/cm2kc) is a CLI tool used to convert a [clustermap file][clustermap docs] to a [kubeconfig file][kubeconfig docs]. Deprecated because we have moved away from clustermaps; you should use [`gencred`] to generate a [kubeconfig file] directly. - - - -[github branch protection]: https://help.github.com/articles/about-protected-branches/ -[clustermap docs]: https://github.com/kubernetes/test-infra/blob/1c7d9a4ae0f2ae1e0c11d8357f47163d18521b84/prow/getting_started_deploy.md#run-test-pods-in-different-clusters -[kubeconfig docs]: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/ -[`gencred`]: https://github.com/kubernetes/test-infra/tree/master/gencred -[gerrit]: https://www.gerritcodereview.com/ -[`updateconfig` Prow plugin]: https://github.com/kubernetes/test-infra/tree/master/prow/plugins/updateconfig -[kubernetes/org]: https://github.com/kubernetes/org diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/sub/README.md b/site/content/en/docs/Legacy Snapshot/prow/cmd/sub/README.md deleted file mode 100644 index 44262ed64e..0000000000 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/sub/README.md +++ /dev/null @@ -1,155 +0,0 @@ ---- -title: "prow/cmd/sub/README.md" ---- - -# Sub - -Sub is a Prow Cloud Pub/Sub adapter for handling CI Pub/Sub notification requests to create Prow jobs. -Note that the prow job need to be defined in the configuration. - -## Deployment Usage - -Sub can listen to Pub/Sub subscriptions (known as "pull subscriptions"). - -When deploy the sub component, you need to specify `--config-path` to your prow config, and optionally -`--job-config-path` to your prowjob config if you have split them up. - -Options: - -- `--dry-run`: Dry run for testing. Uses API tokens but does not mutate. -- `--grace-period`: On shutdown, try to handle remaining events for the specified duration. - -### Pull Server - -All pull subscriptions need to be defined in Prow Configuration: - -``` -pubsub_subscriptions: - "gcp-project-01": - - "subscription-01" - - "subscription-02" - - "subscription-03" - "gcp-project-02": - - "subscription-01" - - "subscription-02" - - "subscription-03" -``` - -Sub must be running with `GOOGLE_APPLICATION_CREDENTIALS` environment variable pointing to the service -account credentials JSON file. The service account used must have the right permission on the -subscriptions (`Pub/Sub Subscriber`, and `Pub/Sub Editor`). - -More information at https://cloud.google.com/pubsub/docs/access-control. - -### Sending a Pub/Sub Notification - -#### Periodic Prow Jobs - -When creating your Pub/Sub message, add an attributes with key `prow.k8s.io/pubsub.EventType` -and value `prow.k8s.io/pubsub.PeriodicProwJobEvent`, and a payload like so: - -```json -{ - "name":"my-periodic-job", - "envs":{ - "GIT_BRANCH":"v.1.2", - "MY_ENV":"overwrite" - }, - "labels":{ - "myLabel":"myValue", - }, - "annotations":{ - # GCP project where prowjobs statues are published by prow. Must also provide "prow.k8s.io/pubsub.topic" to take effect. - # It's highly recommended to configure this even if prowjobs are monitorings by other means, since this is also where errors are - # reported when the job failed to be triggered - "prow.k8s.io/pubsub.project":"myProject", - "prow.k8s.io/pubsub.runID":"asdfasdfasdf", - # GCP pubsub topic where prowjobs statues are published by prow, must be a different topic from where this payload is published to - "prow.k8s.io/pubsub.topic":"myTopic" - } -} -``` - -This will find and start the periodic job `my-periodic-job`, and add / overwrite the -annotations and envs to the Prow job. The `prow.k8s.io/pubsub.*` annotations are -used to publish job status. - -_Note: periodic jobs always clone source code from ref instead of specific SHA, if it's desired to trigger a prowjob on specific SHA you can use [postsubmit job](#postsubmit-prow-jobs)_ - -#### Presubmit Prow Jobs - -Triggering presubmit job is similar to periodic jobs. Two things to change: - -- instead of an attributes with key `prow.k8s.io/pubsub.EventType` and value - `prow.k8s.io/pubsub.PeriodicProwJobEvent`, replace the value with `prow.k8s.io/pubsub.PresubmitProwJobEvent` -- requires setting `refs` instructing presubmit jobs how to clone source code: - -```json -{ - # Common fields as above - "name":"my-presubmit-job", - "envs":{...}, - "labels":{...}, - "annotations":{...}, - - "refs":{ - "org": "org-a", - "repo": "repo-b", - "base_ref": "main", - "base_sha": "abc123", - "pulls": [ - { - "sha": "def456" - } - ] - } -} -``` - -This will start presubmit job `my-presubmit-job`, clones source code like pull requests -defined under `pulls`, which merges to `base_ref` at `base_sha`. - -(There are more fields can be supplied, see [full documentation](https://github.com/kubernetes/test-infra/blob/18678b3b8f4bc7c51475f41964927ff7e635f3b9/prow/apis/prowjobs/v1/types.go#L883). For example, if you want the job to be reported on the PR, add `number` field right next to `sha`) - -#### Postsubmit Prow Jobs - -Triggering presubmit job is similar to periodic jobs. Two things to change: - -- instead of an attributes with key `prow.k8s.io/pubsub.EventType` and value - `prow.k8s.io/pubsub.PeriodicProwJobEvent`, replace the value with `prow.k8s.io/pubsub.PostsubmitProwJobEvent` -- requires setting `refs` instructing postsubmit jobs how to clone source code: - -```json -{ - # Common fields as above - "name":"my-postsubmit-job", - "envs":{...}, - "labels":{...}, - "annotations":{...}, - - "refs":{ - "org": "org-a", - "repo": "repo-b", - "base_ref": "main", - "base_sha": "abc123" - } -} -``` - -This will start postsubmit job `my-postsubmit-job`, clones source code from `base_ref` -at `base_sha`. - -(There are more fields can be supplied, see [full documentation](https://github.com/kubernetes/test-infra/blob/18678b3b8f4bc7c51475f41964927ff7e635f3b9/prow/apis/prowjobs/v1/types.go#L883)) - -#### Gerrit Presubmits and Postsubmits - -Gerrit presubmit and postsubmit jobs require some additional labels and annotations to be specified in the pubsub payload if you wish for them to report results back to the Gerrit change. Specifically the following annotations must be supplied (values are examples): - -```yaml - annotations: - prow.k8s.io/gerrit-id: my-repo~master~I79eee198f020c2ff23d49dbe4d2b2ef7cdc4091b - prow.k8s.io/gerrit-instance: https://my-project-review.googlesource.com - labels: - prow.k8s.io/gerrit-patchset: "4" - prow.k8s.io/gerrit-revision: 2b8cafaab9bd3a829a6bdaa819a18f908bc677ca -``` diff --git a/site/content/en/docs/Legacy Snapshot/prow/plugins/lgtm/README.md b/site/content/en/docs/Legacy Snapshot/prow/plugins/lgtm/README.md deleted file mode 100644 index 5bb6a0cd82..0000000000 --- a/site/content/en/docs/Legacy Snapshot/prow/plugins/lgtm/README.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: "prow/plugins/lgtm/README.md" ---- - -# LGTM plugin - -See [the documentation in the approve plugin for details on the LGTM flow](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/approve/approvers/README.md#lgtm-label). diff --git a/site/content/en/docs/Overview/_index.md b/site/content/en/docs/Overview/_index.md deleted file mode 100644 index 32d318f8ce..0000000000 --- a/site/content/en/docs/Overview/_index.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -title: "Overview" -weight: 1 -description: > - A brief guide about Prow and its ecosystem ---- - -## What is Prow? - -Prow is a Kubernetes-based CI/CD system. - -## How can I use it? - - -If you've contributed to [Kubernetes][k8s], chances are you've already used it. -If you've ever used a TestGrid dashboard, you've definitely used it. - - - -[k8s]: https://github.com/kubernetes/kubernetes diff --git a/site/content/en/docs/_index.md b/site/content/en/docs/_index.md index 5097d04724..cac708fcdb 100644 --- a/site/content/en/docs/_index.md +++ b/site/content/en/docs/_index.md @@ -1,4 +1,3 @@ - --- title: "Documentation" linkTitle: "Documentation" diff --git a/site/content/en/docs/ANNOUNCEMENTS.md b/site/content/en/docs/announcements.md similarity index 57% rename from site/content/en/docs/ANNOUNCEMENTS.md rename to site/content/en/docs/announcements.md index afdc37ad78..781a58860e 100644 --- a/site/content/en/docs/ANNOUNCEMENTS.md +++ b/site/content/en/docs/announcements.md @@ -1,6 +1,6 @@ --- title: "Announcements" -weight: 1 +weight: 10 description: > News about Prow development --- @@ -8,25 +8,27 @@ description: > ## New features New features added to each component: - - *August 4, 2022* override plugin will now override checkruns set by GitHub Actions and other CI + +- *August 4, 2022* override plugin will now override checkruns set by GitHub Actions and other CI systems on a PR. - - *June 8, 2022* `deck.rerun_auth_configs` can optionally be replaced with +- *June 8, 2022* `deck.rerun_auth_configs` can optionally be replaced with `deck.default_rerun_auth_configs` which supports a new format that is a slice of filters with associated rerun auth configs rather than a map. Currently entries can filter by repo and/or cluster. The old field is still supported and will not be deprecated. - - *April 6, 2022* Highlight and pin interesting lines. To do this, +- *April 6, 2022* Highlight and pin interesting lines. To do this, shift-click on log lines in the buildlog lens. The URL fragment causes the same lines to be highlighted next page load. Additionally, when viewing a GCS log pressing the pin button saves the highlight. The saved highlight automatically displays next page load. - - *January 24, 2022* It is possible now to define GitHub Apps bots as trusted - users to allow automatic tests trigger without relying on `/ok-to-test` - from organization member. Trigger and DCO plugins configuration now support +- *January 24, 2022* It is possible now to define GitHub Apps bots as trusted + users to allow automatic tests trigger without relying on `/ok-to-test` + from organization member. Trigger and DCO plugins configuration now support additional field `trusted_apps`, which contains list of GitHub Apps bot usernames without `[bot]` suffix. - - *January 11, 2022* Trigger plugin can now trigger failed github jobs. +- *January 11, 2022* Trigger plugin can now trigger failed github jobs. The feature needs to be enabled in the `triggers` section of the `plugin.yaml` config and can be specified per trigger as follows: + ```yaml triggers: - repos: @@ -34,7 +36,8 @@ New features added to each component: - org2 trigger_github_workflows: true ``` - - *August 24, 2021* Postsubmit Prow jobs now support the `always_run` field. + +- *August 24, 2021* Postsubmit Prow jobs now support the `always_run` field. This field interacts with the `run_if_changed` and `skip_if_only_changed` fields as follows: 1. (NEW) If the field is explicitly set to `always_run: false`, then the @@ -59,137 +62,139 @@ New features added to each component: 2. If one of `run_if_changed` or `skip_if_only_changed` is set: same as old behavior (running will depend on `run_if_changed` or `skip_if_only_changed`). - - *May 14th, 2021*: All components that interact with GitHub newly allow client-side throttling customization +- *May 14th, 2021*: All components that interact with GitHub newly allow client-side throttling customization via `--github-hourly-tokens` and `--github-allowed-burst` parameters. A notable exception to this is Tide which has custom throttling logic and does not expose these two new options. Other existing custom options in branchprotector, peribolos, status-reconciler and needs-rebase (`--tokens/--hourly-tokens` etc.) are deprecated and will be removed in August 2021. - - *April 12th, 2021* End of grace period for storage bucket validation, additional buckets have to be allowed +- *April 12th, 2021* End of grace period for storage bucket validation, additional buckets have to be allowed by adding them to the `deck.additional_allowed_buckets` list. - - *March 9th, 2021* Tide batchtesting will now continue to test a given batch even +- *March 9th, 2021* Tide batchtesting will now continue to test a given batch even when more PRs became eligible while a test failed. You can disable this by setting `tide.prioritize_existing_batches.: false` in your Prow config. - - *March 3, 2021* `plank.default_decoration_configs` can optionally be replaced with +- *March 3, 2021* `plank.default_decoration_configs` can optionally be replaced with `plank.default_decoration_config_entries` which supports a new format that is a slice of filters with associated decoration configs rather than a map. Currently entries can filter by repo and/or cluster. The old field is still supported and will not be deprecated. - - *February 23, 2021* New format introduced in `plugins.yaml`. Repos can be excluded from plugin definition +- *February 23, 2021* New format introduced in `plugins.yaml`. Repos can be excluded from plugin definition at org level using `excluded_repos` notation. The previous format will be deprecated in *July 2021*, see - https://github.com/kubernetes/test-infra/issues/20631. - - *November 2, 2020* Tide is now able to respect checkruns. - - *September 15, 2020* Added validation to Deck that will restrict artifact requests based on storage buckets. + . +- *November 2, 2020* Tide is now able to respect checkruns. +- *September 15, 2020* Added validation to Deck that will restrict artifact requests based on storage buckets. Opt-out by setting `deck.skip_storage_path_validation` in your Prow config. Buckets specified in job configs (`.gcs_configuration.bucket`) and plank configs (`plank.default_decoration_configs[*].gcs_configuration.bucket`) are automatically allowed access. Additional buckets can be allowed by adding them to the `deck.additional_allowed_buckets` list. (This feature will be enabled by default ~Jan 2021. For now, you will begin to notice violation warnings in your logs.) - - *August 31th, 2020* Added `gcs_browser_prefixes` field in spyglass configuration. `gcs_browser_prefix` will +- *August 31th, 2020* Added `gcs_browser_prefixes` field in spyglass configuration. `gcs_browser_prefix` will be deprecated in February 2021. You can now specify different values for different repositories. The format should be in org, org/repo or '\*' which is the default value. - - *July 13th, 2020* Configuring `job_url_prefix_config` with `gcs/` prefix is now deprecated. +- *July 13th, 2020* Configuring `job_url_prefix_config` with `gcs/` prefix is now deprecated. Please configure a job url prefix without the `gcs/` storage provider suffix. From now on the storage provider is appended automatically so multiple storage providers can be used for builds of the same repository. For now we still handle the old configuration format, this will be removed in *September 2020*. To be clear handling of URLs with `/view/gcs` in Deck is not deprecated. - - *June 23rd, 2020* An [hmac](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/hmac) tool was added to automatically reconcile webhooks and hmac +- *June 23rd, 2020* An [hmac](/docs/components/optional/hmac/) tool was added to automatically reconcile webhooks and hmac tokens for the orgs and repos integrated with your prow instance. - - *June 8th, 2020* A new informer-based Plank implementation was added. It can be used by deploying +- *June 8th, 2020* A new informer-based Plank implementation was added. It can be used by deploying the new [prow-controller-manager](https://github.com/kubernetes/test-infra/tree/master/config/prow/experimental/controller_manager.yaml) binary. - We plan to gradually move all our controllers into that binary, see https://github.com/kubernetes/test-infra/issues/17024 - - *May 31, 2020* '--gcs-no-auth' in Deck is deprecated and not used anymore. We always + We plan to gradually move all our controllers into that binary, see +- *May 31, 2020* '--gcs-no-auth' in Deck is deprecated and not used anymore. We always fall back to an anonymous GCS client now, if all other options fail. This flag will be removed in *July 2020*. - - *May 25, 2020* Added `--blob-storage-workers` and `--kubernetes-blob-storage-workers` +- *May 25, 2020* Added `--blob-storage-workers` and `--kubernetes-blob-storage-workers` flags to crier. The flags `--gcs-workers` and `--kubernetes-gcs-workers` are now deprecated and will be removed in *August 2020*. - - *May 13, 2020* Added a `decorate_all_jobs` option to job configuration that +- *May 13, 2020* Added a `decorate_all_jobs` option to job configuration that allows to control whether jobs are decorated by default. Individual jobs can use the `decorate` option to override this setting. - - *March 25, 2020* Added a `report_templates` option to the Plank config that allows +- *March 25, 2020* Added a `report_templates` option to the Plank config that allows to specify different report templates for each organization or a specific repository. The `report_template` option is deprecated and it will be removed on *September 2020* which is going to be replaced with the `*` value in `report_templates`. - - *January 03, 2020* Added a `pr_status_base_urls` option to the Tide config +- *January 03, 2020* Added a `pr_status_base_urls` option to the Tide config that allows to specify different tide's URL for each organization or a specific repository. The `pr_status_base_url` will be deprecated on *June 2020* and it will be replaced with the `*` value in `pr_status_base_urls`. - - *November 05, 2019* The `config-updater` plugin supports update configs on build clusters - by using [`clusters`](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/updateconfig#usage). - The fields _namespace_ and _additional_namespaces_ are deprecated. - - *October 27, 2019* The `trusted_org` functionality in trigger is being +- *November 05, 2019* The `config-updater` plugin supports update configs on build clusters + by using [`clusters`](/docs/components/plugins/updateconfig/#usage). + The fields *namespace* and *additional_namespaces* are deprecated. +- *October 27, 2019* The `trusted_org` functionality in trigger is being deprecated in favour of being more explicit in the fact that org members or repo collaborators are the trusted users. This option will be removed completely in January 2020. - - *October 07, 2019* Added a `default_decoration_configs` option to the Plank config +- *October 07, 2019* Added a `default_decoration_configs` option to the Plank config that allows to specify different plank's default configuration for each organization or a specific repository. `default_decoration_config` will be deprecated in April 2020 and it will be replaced with the `*` value in `default_decoration_configs`. - - *August 29, 2019* Added a `batch_size_limit` option to the Tide config that +- *August 29, 2019* Added a `batch_size_limit` option to the Tide config that allows the batch size limit to be specified globally, per org, or per repo. Values default to 0 indicating no size limit. A value of -1 disables batches. - - *July 30, 2019* `authorized_users` in `rerun_auth_config` for deck will become `github_users`. - - *July 19, 2019* deck will soon remove its default value for `--cookie-secret-file`. +- *July 30, 2019* `authorized_users` in `rerun_auth_config` for deck will become `github_users`. +- *July 19, 2019* deck will soon remove its default value for `--cookie-secret-file`. If you set `--oauth-url` but not `--cookie-secret-file`, add `--cookie-secret-file=/etc/cookie-secret` to your deck instance. The default value will be removed at the end of October 2019. - - *July 2, 2019* prow defaults to report status for both presubmit and postsubmit +- *July 2, 2019* prow defaults to report status for both presubmit and postsubmit jobs on GitHub now. - - *June 17, 2019* It is now possible to configure the channel for the Slack reporter +- *June 17, 2019* It is now possible to configure the channel for the Slack reporter directly on jobs via the `.reporter_config.slack.channel` config option - - *May 13, 2019* New `plank` config `pod_running_timeout` is added and +- *May 13, 2019* New `plank` config `pod_running_timeout` is added and defaulted to two days to allow plank abort pods stuck in running state. - - *April 25, 2019* `--job-config` in `peribolos` has never been used; it is +- *April 25, 2019* `--job-config` in `peribolos` has never been used; it is deprecated and will be removed in July 2019. Remove the flag from any calls to the tool. - - *April 24, 2019* `file_weight_count` in blunderbuss is being deprecated in +- *April 24, 2019* `file_weight_count` in blunderbuss is being deprecated in favour of the more current `max_request_count` functionality. Please ensure your configuration is up to date before the end of May 2019. - - *March 12, 2019* tide now records a history of its actions and exposes a +- *March 12, 2019* tide now records a history of its actions and exposes a filterable view of these actions at the `/tide-history` deck path. - - *March 9, 2019* prow components now support reading gzipped config files - - *February 13, 2019* prow (both plank and crier) can set status on the commit +- *March 9, 2019* prow components now support reading gzipped config files +- *February 13, 2019* prow (both plank and crier) can set status on the commit for postsubmit jobs on github now! Type of jobs can be reported to github is gated by a config field like + ```yaml github_reporter: job_types_to_report: - presubmit - postsubmit ``` + now and default to report for presubmit only. - *** The default will change in April to include postsubmit jobs as well *** + ***The default will change in April to include postsubmit jobs as well*** You can also add `skip_report: true` to your post-submit jobs to skip reporting if you enable postsubmit reporting on. - - *January 15, 2019* `approve` now considers self-approval and github review +- *January 15, 2019* `approve` now considers self-approval and github review state by default. Configure with `require_self_approval` and `ignore_review_state`. Temporarily revert to old defaults with `use_deprecated_2018_implicit_self_approve_default_migrate_before_july_2019` and `use_deprecated_2018_review_acts_as_approve_default_migrate_before_july_2019`. - - *January 12, 2019* `blunderbluss` plugin now provides a new command, `/auto-cc`, +- *January 12, 2019* `blunderbluss` plugin now provides a new command, `/auto-cc`, that triggers automatic review requests. - - *January 7, 2019* `implicit_self_approve` will become `require_self_approval` in +- *January 7, 2019* `implicit_self_approve` will become `require_self_approval` in the second half of this year. - - *January 7, 2019* `review_acts_as_approve` will become `ignore_review_state` in +- *January 7, 2019* `review_acts_as_approve` will become `ignore_review_state` in the second half of this year. - - *October 10, 2018* `tide` now supports the `-repo:foo/bar` tag in queries via +- *October 10, 2018* `tide` now supports the `-repo:foo/bar` tag in queries via the `excludedRepos` YAML field. - - *October 3, 2018* `welcome` now supports a configurable message on a per-org, +- *October 3, 2018* `welcome` now supports a configurable message on a per-org, or per-repo basis. Please note that this includes a config schema change that will break previous deployments of this plugin. - - *August 22, 2018* `spyglass` is a pluggable viewing framework for artifacts +- *August 22, 2018* `spyglass` is a pluggable viewing framework for artifacts produced by Prowjobs. See a demo [here](https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-performance/121)! - - *July 13, 2018* `blunderbluss` plugin will now support `required_reviewers` in +- *July 13, 2018* `blunderbluss` plugin will now support `required_reviewers` in OWNERS file to specify a person or github team to be cc'd on every PR that touches the corresponding part of the code. - - *June 25, 2018* `updateconfig` plugin will now support update/remove keys +- *June 25, 2018* `updateconfig` plugin will now support update/remove keys from a glob match. - - *June 05, 2018* `blunderbuss` plugin may now suggest approvers in addition +- *June 05, 2018* `blunderbuss` plugin may now suggest approvers in addition to reviewers. Use `exclude_approvers: true` to revert to previous behavior. - - *April 10, 2018* `cla` plugin now supports `/check-cla` command +- *April 10, 2018* `cla` plugin now supports `/check-cla` command to force rechecking of the CLA status. - - *February 1, 2018* `updateconfig` will now update any configmap on merge - - *November 14, 2017* `jenkins-operator:0.58` exposes prometheus metrics. - - *November 8, 2017* `horologium:0.14` prow periodic job now support cron - triggers. See https://godoc.org/gopkg.in/robfig/cron.v2 for doc to the +- *February 1, 2018* `updateconfig` will now update any configmap on merge +- *November 14, 2017* `jenkins-operator:0.58` exposes prometheus metrics. +- *November 8, 2017* `horologium:0.14` prow periodic job now support cron + triggers. See for doc to the cron library we are using. ## Breaking changes @@ -200,17 +205,18 @@ state and no claims of backwards compatibility are made for any external API. Note: versions specified in these announcements may not include bug fixes made in more recent versions so it is recommended that the most recent versions are used when updating deployments. - - *August 24th, 2022* Deck by default validating storage buckets, can still opt + +- *August 24th, 2022* Deck by default validating storage buckets, can still opt out by setting `deck.skip_storage_path_validation: true` in your Prow config. Buckets specified in job configs (`.gcs_configuration.bucket`) and plank configs (`plank.default_decoration_configs[*].gcs_configuration.bucket`) are automatically allowed access. Additional buckets can be allowed by adding them to the `deck.additional_allowed_buckets` list. - - *May 27th, 2022* Crier flags `--gcs-workers` and `--kubernetes-gcs-workers` are +- *May 27th, 2022* Crier flags `--gcs-workers` and `--kubernetes-gcs-workers` are removed in favor of `--blob-storage-workers` and `--kubernetes-blob-storage-workers`. - - *May 27th, 2022* The `owners_dir_blacklist` field in prow config is removed +- *May 27th, 2022* The `owners_dir_blacklist` field in prow config is removed in favor of `owners_dir_denylist`. - - *February 22nd, 2022* Since prow version `v20220222-acb5731b85`, the +- *February 22nd, 2022* Since prow version `v20220222-acb5731b85`, the entrypoint container in a prow job will run as `--copy-mode-only`, instead of `/bin/cp /entrypoint /tools/entrypoint`. Entrypoint images before the mentioned version will not work with `--copy-mode-only`, and entrypoint image since the @@ -218,8 +224,8 @@ used when updating deployments. In another word, prow versions newer than or equal to `v20220222-acb5731b85` will stop working with pod utilities versions older than `v20220222-acb5731b85`. If your prow instance is bumped by - `prow/cmd/generic-autobumper` then you should not be affected. - - *February 22nd, 2022* Since prow version `v20220222-acb5731b85`, prow images + `prow/cmd/generic-autobumper` then you should not be affected. +- *February 22nd, 2022* Since prow version `v20220222-acb5731b85`, prow images pushed to gcr.io/k8s-prow will be built with ko, and the binaries will be placed under `/ko-app/`, for example [/robots/commenter](https://github.com/kubernetes/test-infra/tree/master/robots/commenter) is pushed to gcr.io/k8s-prow/commenter, the commenter binary is located at @@ -227,200 +233,212 @@ used when updating deployments. to `command: - /ko-app/commenter` to make it work, alternatively, the command could also be `command: - commenter` as `/ko-app` is added to `$PATH` env var in the image. - - *February 22nd, 2022* Since prow version `v20220222-acb5731b85`, static files +- *February 22nd, 2022* Since prow version `v20220222-acb5731b85`, static files in `deck` image will be stored under `/var/run/ko/` directory. - - *October 27th, 2021* The checkconfig flag `--prow-yaml-repo-path` no longer defaults to `/home/prow/go/src/github.com/<< prow-yaml-repo-name >>/.prow.yaml` when `--prow-yaml-repo-name` is set. The defaulting has instead been replaced with the assumption that the Prow YAML file/directory can be found in the current working directory if `--prow-yaml-repo-path` is not specified. If you are running checkconfig from a decorated ProwJobs as is typical, then this is already the case. - - *September 16th, 2021* The ProwJob [CRD manifest](https://github.com/kubernetes/test-infra/tree/master/config/prow/cluster/prowjob-crd/prowjob_customresourcedefinition.yaml) +- *October 27th, 2021* The checkconfig flag `--prow-yaml-repo-path` no longer defaults to `/home/prow/go/src/github.com/<< prow-yaml-repo-name >>/.prow.yaml` when `--prow-yaml-repo-name` is set. The defaulting has instead been replaced with the assumption that the Prow YAML file/directory can be found in the current working directory if `--prow-yaml-repo-path` is not specified. If you are running checkconfig from a decorated ProwJobs as is typical, then this is already the case. +- *September 16th, 2021* The ProwJob [CRD manifest](https://github.com/kubernetes/test-infra/tree/master/config/prow/cluster/prowjob-crd/prowjob_customresourcedefinition.yaml) has been extended to specify a schema. Unfortunately, this results in a huge manifest which in turn makes the standard `kubectl apply` fail, as the last-applied annotation it generates exceeds the maximum annotation size. If you are using Kubernetes 1.18 or newer, you can add the `--server-side=true` argument to work around this. If not, you can use a [schemaless manifest](https://github.com/kubernetes/test-infra/tree/master/config/prow/cluster/prowjob-crd/legacy/prowjob-schemaless_customresourcedefinition.yaml) - - *September 15th, 2021* `autobump` removed, please use `generic-autobumper` instead, see [example config](https://github.com/kubernetes/test-infra/tree/master/config/prow/autobump-config/prow-component-autobump-config.yaml) - - *April 16th, 2021* Flagutil remove default value for `--github-token-path`. - - *April 15th, 2021* Sinker requires --dry-run=false (default is true) to function correctly in production. - - *April 14th, 2021* Deck remove default value for `--cookie-secret-file`. - - *April 12th, 2021* Horologium now uses a cached client, which requires it to have watch permissions for Prowjobs on top of the already-required list and create. - - *April 11th, 2021* The plank binary has been removed. Please use the [Prow Controller Manager](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/prow-controller-manager) instead, which provides a more modern implementation +- *September 15th, 2021* `autobump` removed, please use `generic-autobumper` instead, see [example config](https://github.com/kubernetes/test-infra/tree/master/config/prow/autobump-config/prow-component-autobump-config.yaml) +- *April 16th, 2021* Flagutil remove default value for `--github-token-path`. +- *April 15th, 2021* Sinker requires --dry-run=false (default is true) to function correctly in production. +- *April 14th, 2021* Deck remove default value for `--cookie-secret-file`. +- *April 12th, 2021* Horologium now uses a cached client, which requires it to have watch permissions for Prowjobs on top of the already-required list and create. +- *April 11th, 2021* The plank binary has been removed. Please use the [Prow Controller Manager](/docs/components/core/prow-controller-manager/) instead, which provides a more modern implementation of the same functionality. - - *April 1st, 2021* The `owners_dir_blacklist` field in prow config has been deprecated in favor of `owners_dir_denylist`. The support of `owners_dir_blacklist` will be stopped in October 2021. - - *April 1st, 2021* The `labels_blacklist` field in verify-owners plugin config +- *April 1st, 2021* The `owners_dir_blacklist` field in prow config has been deprecated in favor of `owners_dir_denylist`. The support of `owners_dir_blacklist` will be stopped in October 2021. +- *April 1st, 2021* The `labels_blacklist` field in verify-owners plugin config is deprecated in favor of `labels_denylist`. The support for `labels_blacklist` shall be stopped in *October 2021*. - - *January 24th, 2021* Planks Pod pending and Pod scheduling timeout defaults where changed from 24h each to the more reasonable 10 minutes/5 minutes, respectively. - - *January 1, 2021* Support for `whitelist` and `branch_whitelist` fields in Slack merge warning configuration is discontinued. You can use `exempt_users` and `exempt_branches` fields instead. - - *November 24, 2020* The `requiresig` plugin has been removed in favor of the `require-matching-label` plugin +- *January 24th, 2021* Planks Pod pending and Pod scheduling timeout defaults where changed from 24h each to the more reasonable 10 minutes/5 minutes, respectively. +- *January 1, 2021* Support for `whitelist` and `branch_whitelist` fields in Slack merge warning configuration is discontinued. You can use `exempt_users` and `exempt_branches` fields instead. +- *November 24, 2020* The `requiresig` plugin has been removed in favor of the `require-matching-label` plugin which provides equivalent functionality ([example plugin config](https://github.com/kubernetes/test-infra/blob/e42b0745404017bc71c668da0342ef6857d87fa9/config/prow/plugins.yaml#L494-L498)) - - *November 14, 2020* The `whitelist` and `branch_whitelist` fields in Slack merge warning were deprecated on *August 22, 2020* in favor of the new `exempt_users` and `exempt_branches` fields. The support for these fields shall be stopped in *January 2021*. - - *November 11th, 2020* The prow-controller-manager and sinker now require RBAC to be set up to manage their leader lock in the `coordination.k8s.io` group. See [here](https://github.com/kubernetes/test-infra/pull/19906/files?diff=split&w=1) - - *November, 2020* The deprecated `namespace` and `additional_namespaces` properties have been removed from the config updater plugin +- *November 14, 2020* The `whitelist` and `branch_whitelist` fields in Slack merge warning were deprecated on *August 22, 2020* in favor of the new `exempt_users` and `exempt_branches` fields. The support for these fields shall be stopped in *January 2021*. +- *November 11th, 2020* The prow-controller-manager and sinker now require RBAC to be set up to manage their leader lock in the `coordination.k8s.io` group. See [here](https://github.com/kubernetes/test-infra/pull/19906/files?diff=split&w=1) +- *November, 2020* The deprecated `namespace` and `additional_namespaces` properties have been removed from the config updater plugin for more details. - - *November, 2020* The `blacklist` flag in status reconciler has been deprecated in favor of `denylist`. The support of `blacklist` will be stopped in February 2021. - - *October, 2020* The `plank` binary has been deprecated in favor of the more modern implementation in the prow-controller-manager that provides the same functionality. Check out - its [README](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/prow-controller-manager/README.md) or check out its [deployment](https://github.com/kubernetes/test-infra/tree/master/config/prow/cluster/prow_controller_manager_deployment.yaml) and +- *November, 2020* The `blacklist` flag in status reconciler has been deprecated in favor of `denylist`. The support of `blacklist` will be stopped in February 2021. +- *October, 2020* The `plank` binary has been deprecated in favor of the more modern implementation in the prow-controller-manager that provides the same functionality. Check out + its [README](/docs/components/core/prow-controller-manager/) or check out its [deployment](https://github.com/kubernetes/test-infra/tree/master/config/prow/cluster/prow_controller_manager_deployment.yaml) and [rbac](https://github.com/kubernetes/test-infra/tree/master/config/prow/cluster/prow_controller_manager_rbac.yaml) manifest. The plank binary will be removed in February, 2021. - - *September 14th, 2020* Sinker now requires `LIST` and `WATCH` permissions for pods - - *September 2, 2020* The already deprecated `namespace` and `additional_namespaces` settings in the config updater will be removed in October, 2020 - - *August 28, 2020* `tide` now ignores archived repositories in queries. - - *August 28, 2020* The `Clusters` format and associated `--build-cluster` flag has been removed. - - *August 24, 2020* The deprecated reporting functionality has been removed from Plank, use crier with `--github-workers=1` instead +- *September 14th, 2020* Sinker now requires `LIST` and `WATCH` permissions for pods +- *September 2, 2020* The already deprecated `namespace` and `additional_namespaces` settings in the config updater will be removed in October, 2020 +- *August 28, 2020* `tide` now ignores archived repositories in queries. +- *August 28, 2020* The `Clusters` format and associated `--build-cluster` flag has been removed. +- *August 24, 2020* The deprecated reporting functionality has been removed from Plank, use crier with `--github-workers=1` instead Use a `.kube/config` with the `--kubeconfig` flag to specify credentials for external build clusters. - - *August 22, 2020* The `whitelist` and `branch_whitelist` fields in Slack merge warning are deprecated in favor of the new `exempt_users` and `exempt_branches` fields. - - *July 17, 2020* Slack reporter will no longer report all states of a Prow job if it has `Channel` +- *August 22, 2020* The `whitelist` and `branch_whitelist` fields in Slack merge warning are deprecated in favor of the new `exempt_users` and `exempt_branches` fields. +- *July 17, 2020* Slack reporter will no longer report all states of a Prow job if it has `Channel` specified on the Prow job config. Instead, it will report the `job_states_to_report` configured in the Prow job or in the Prow core config if the former does not exist. - - *May 18, 2020* `expiry` field has been replaced with `created_at` in the HMAC secret. - - *April 24, 2020* Horologium now defaults to `--dry-run=true` - - *April 23, 2020* Explicitly setting `--config-path` is now required. - - *April 23, 2020* Update the `autobump` image to at least `v20200422-8c8546d74` before June 2020. - - *April 23, 2020* Deleted deprecated `default_decoration_config`. - - *April 22, 2020* Deleted the `file_weight_count` blunderbuss option. - - *April 16, 2020* The `docs-no-retest` prow plugin has been deleted. +- *May 18, 2020* `expiry` field has been replaced with `created_at` in the HMAC secret. +- *April 24, 2020* Horologium now defaults to `--dry-run=true` +- *April 23, 2020* Explicitly setting `--config-path` is now required. +- *April 23, 2020* Update the `autobump` image to at least `v20200422-8c8546d74` before June 2020. +- *April 23, 2020* Deleted deprecated `default_decoration_config`. +- *April 22, 2020* Deleted the `file_weight_count` blunderbuss option. +- *April 16, 2020* The `docs-no-retest` prow plugin has been deleted. The plugin was deprecated in January 2020. - - *April 14, 2020* GitHub reporting via plank is deprecated, set --github-workers=1 on crier before July 2020. - - *March 27, 2020* The deprecated `allow_cancellations` option has been removed from +- *April 14, 2020* GitHub reporting via plank is deprecated, set --github-workers=1 on crier before July 2020. +- *March 27, 2020* The deprecated `allow_cancellations` option has been removed from Plank and the Jenkins operator. - - *March 19, 2020* The `rerun_auth_config` config field has been deprecated in +- *March 19, 2020* The `rerun_auth_config` config field has been deprecated in favor of the new `rerun_auth_configs` field which allows configuration on a global, organization or repo level. `rerun_auth_config` will be removed in July 2020. - - *November 21, 2019* The boskos metrics component replaced the existing prometheus +- *November 21, 2019* The boskos metrics component replaced the existing prometheus metrics with a single, label-qualified metric. Metrics are now served at `/metrics` on port 9090. This actually happened August 5th, but is being documented now. - Details: https://github.com/kubernetes/test-infra/pull/13767 - - *November 18, 2019* The `mkbuild-cluster` command-line utility and `build-cluster` + Details: +- *November 18, 2019* The `mkbuild-cluster` command-line utility and `build-cluster` format is deprecated and will be removed in May 2020. Use `gencred` and the `kubeconfig` format as an alternative. - - *November 14, 2019* The `slack_reporter` config field has been deprecated in +- *November 14, 2019* The `slack_reporter` config field has been deprecated in favor of the new `slack_reporter_configs` field which allows configuration on a global, organization or repo level. `slack_reporter` will be removed in May 2020. - - *November 7, 2019* The `plank.allow_cancellations` and `jenkins_operators.allow_cancellations` +- *November 7, 2019* The `plank.allow_cancellations` and `jenkins_operators.allow_cancellations` settings are deprecated and will be removed and set to always `true` in March 2020. - - *October 7, 2019* Prow will drop support for the deprecated knative-builds in +- *October 7, 2019* Prow will drop support for the deprecated knative-builds in November 2019. - - *September 24, 2019* Sending an http `GET` request to the `/hook` endpoint now returns a `405` +- *September 24, 2019* Sending an http `GET` request to the `/hook` endpoint now returns a `405` (Method Not Allowed) instead of a `200` (OK). - - *September 8, 2019* The deprecated `job_url_prefix` option has been removed from Plank. - - *May 2, 2019* All components exposing Prometheus metrics will now either push them +- *September 8, 2019* The deprecated `job_url_prefix` option has been removed from Plank. +- *May 2, 2019* All components exposing Prometheus metrics will now either push them to the Prometheus PushGateway, if configured, or serve them locally on port 9090 at `/metrics`, if not configured (the default). - - *April 26, 2019* `blunderbuss`, `approve`, and other plugins that read OWNERS +- *April 26, 2019* `blunderbuss`, `approve`, and other plugins that read OWNERS now treat `owners_dir_blacklist` as a list of regular expressions matched against the entire (repository-relative) directory path of the OWNERS file rather than as a list of strings matched exactly against the basename only of the directory containing the OWNERS file. - - *April 2, 2019* `hook`, `deck`, `horologium`, `tide`, `plank` and `sinker` will no +- *April 2, 2019* `hook`, `deck`, `horologium`, `tide`, `plank` and `sinker` will no longer provide a default value for the `--config-path` flag. It is required to explicitly provide `--config-path` when upgrading to a new version of these components that were previously relying on the default `--config-path=/etc/config/config.yaml`. - - *March 29, 2019* Custom logos should be provided as full paths in the configuration +- *March 29, 2019* Custom logos should be provided as full paths in the configuration under `deck.branding.logos` and will not implicitly be assumed to be under the static assets directory. - - *February 26, 2019* The `job_url_prefix` option from `plank` has been deprecated in +- *February 26, 2019* The `job_url_prefix` option from `plank` has been deprecated in favor of the new `job_url_prefix_config` option which allows configuration on a global, organization or repo level. `job_url_prefix` will be removed in September 2019. - - *February 13, 2019* `horologium` and `sinker` deployments will soon require `--dry-run=false` in production, please set this before March 15. At that time flag will default to --dry-run=true instead of --dry-run=false. - - *February 1, 2019* Now that `hook` and `tide` will no longer post "Skipped" statuses +- *February 13, 2019* `horologium` and `sinker` deployments will soon require `--dry-run=false` in production, please set this before March 15. At that time flag will default to --dry-run=true instead of --dry-run=false. +- *February 1, 2019* Now that `hook` and `tide` will no longer post "Skipped" statuses for jobs that do not need to run, it is not possible to require those statuses with branch protection. Therefore, it is necessary to run the `branchprotector` from at least version `510db59` before upgrading `tide` to that version. - - *February 1, 2019* `horologium` and `sinker` now support the `--dry-run` flag, +- *February 1, 2019* `horologium` and `sinker` now support the `--dry-run` flag, so you must pass `--dry-run=false` to keep the previous behavior (see Feb 13 update). - - *January 31, 2019* `sub` no longer supports the `--masterurl` flag for connecting +- *January 31, 2019* `sub` no longer supports the `--masterurl` flag for connecting to the infrastructure cluster. Use `--kubeconfig` with `--context` for this. - - *January 31, 2019* `crier` no longer supports the `--masterurl` flag for connecting +- *January 31, 2019* `crier` no longer supports the `--masterurl` flag for connecting to the infrastructure cluster. Use `--kubeconfig` with `--context` for this. - - *January 27, 2019* Jobs that do not run will no longer post "Skipped" statuses. - - *January 27, 2019* Jobs that do not run always will no longer be required by +- *January 27, 2019* Jobs that do not run will no longer post "Skipped" statuses. +- *January 27, 2019* Jobs that do not run always will no longer be required by branch protection as they will not always produce a status. They will continue to be required for merge by `tide` if they are configured as required. - - *January 27, 2019* All support for `run_after_success` jobs has been removed. +- *January 27, 2019* All support for `run_after_success` jobs has been removed. Configuration of these jobs will continue to parse but will ignore the field. - - *January 27, 2019* `hook` will now correctly honor the `run_always` field on Gerrit +- *January 27, 2019* `hook` will now correctly honor the `run_always` field on Gerrit presubmits. Previously, if this field was unset it would have defaulted to `true`; now, it will correctly default to `false`. - - *January 22, 2019* `sinker` prefers `.kube/config` instead of the custom `Clusters` +- *January 22, 2019* `sinker` prefers `.kube/config` instead of the custom `Clusters` file to specify credentials for external build clusters. The flag name has changed from `--build-cluster` to `--kubeconfig`. Migrate before June 2019. - - *November 29, 2018* `plank` will no longer default jobs with `decorate: true` +- *November 29, 2018* `plank` will no longer default jobs with `decorate: true` to have `automountServiceAccountToken: false` in their PodSpec if unset, if the job explicitly sets `serviceAccountName` - - *November 26, 2018* job names must now match `^[A-Za-z0-9-._]+$`. Jobs that did not +- *November 26, 2018* job names must now match `^[A-Za-z0-9-._]+$`. Jobs that did not match this before were allowed but did not provide a good user experience. - - *November 15, 2018* the `hook` service account now requires RBAC privileges +- *November 15, 2018* the `hook` service account now requires RBAC privileges to create `ConfigMaps` to support new functionality in the `updateconfig` plugin. - - *November 9, 2018* Prow gerrit client label/annotations now have a `prow.k8s.io/` namespace +- *November 9, 2018* Prow gerrit client label/annotations now have a `prow.k8s.io/` namespace prefix, if you have a gerrit deployment, please bump both cmd/gerrit and cmd/crier. - - *November 8, 2018* `plank` now defaults jobs with `decorate: true` to have +- *November 8, 2018* `plank` now defaults jobs with `decorate: true` to have `automountServiceAccountToken: false` in their PodSpec if unset. Jobs that used the default service account should explicitly set this field to maintain functionality. - - *October 16, 2018* Prow tls-cert management has been migrated from kube-lego to cert-manager. - - *October 12, 2018* Removed deprecated `buildId` environment variable from prow jobs. Use `BUILD_ID.` - - *October 3, 2018* `-github-token-file` replaced with +- *October 16, 2018* Prow tls-cert management has been migrated from kube-lego to cert-manager. +- *October 12, 2018* Removed deprecated `buildId` environment variable from prow jobs. Use `BUILD_ID.` +- *October 3, 2018* `-github-token-file` replaced with `-github-token-path` for consistency with `branchprotector` and `peribolos` which were already using `-github-token-path`. `-github-token-file` will continue to work through the remainder of 2018, but it will be removed in early 2019. The following commands are affected: `cherrypicker`, `hook`, `jenkins-operator`, `needs-rebase`, `phony`, `plank`, `refresh`, and `tide`. - - *October 1, 2018* bazel is the one official way to build container images. +- *October 1, 2018* bazel is the one official way to build container images. Please use prow/bump.sh and/or bazel run //prow:release-push - - *Sep 27, 2018* If you are setting explicit decorate configs, the format has changed from +- *Sep 27, 2018* If you are setting explicit decorate configs, the format has changed from + ```yaml - name: job-foo decorate: true timeout: 1 ``` + to + ```yaml - name: job-foo decorate: true decoration_config: timeout: 1 ``` - - *September 24, 2018* the `splice` component has been deleted following the + +- *September 24, 2018* the `splice` component has been deleted following the deletion of mungegithub. - - *July 9, 2018* `milestone` format has changed from +- *July 9, 2018* `milestone` format has changed from + ```yaml milestone: maintainers_id: maintainers_team: ``` + to `repo_milestone` + ```yaml repo_milestone: : maintainers_id: maintainers_team: ``` - - *July 2, 2018* the `trigger` plugin will now trust PRs from repo + +- *July 2, 2018* the `trigger` plugin will now trust PRs from repo collaborators. Use `only_org_members: true` in the trigger config to temporarily disable this behavior. - - *June 14, 2018* the `updateconfig` plugin will only add data to your `ConfigMaps` +- *June 14, 2018* the `updateconfig` plugin will only add data to your `ConfigMaps` using the basename of the updated file, instead of using that and also duplicating the data using the name of the `ConfigMap` as a key - - *June 1, 2018* all unquoted `boolean` fields in config.yaml that were unmarshall +- *June 1, 2018* all unquoted `boolean` fields in config.yaml that were unmarshall into type `string` now need to be quoted to avoid unmarshalling error. - - *May 9, 2018* `deck` logs for jobs run as `Pods` will now return logs for the +- *May 9, 2018* `deck` logs for jobs run as `Pods` will now return logs for the `"test"` container only. - - *April 2, 2018* `updateconfig` format has been changed from +- *April 2, 2018* `updateconfig` format has been changed from + ```yaml path/to/some/other/thing: configName ``` + to + ```yaml path/to/some/other/thing: Name: configName # If unspecified, Namespace default to the value of ProwJobNamespace. Namespace: myNamespace ``` - - *March 15, 2018* `jenkins_operator` is removed from the config in favor of + +- *March 15, 2018* `jenkins_operator` is removed from the config in favor of `jenkins_operators`. - - *March 1, 2018* `MilestoneStatus` has been removed from the plugins Configuration +- *March 1, 2018* `MilestoneStatus` has been removed from the plugins Configuration in favor of the `Milestone` which is shared between two plugins: 1) `milestonestatus` and 2) `milestone`. The milestonestatus plugin now uses the `Milestone` object to get the maintainers team ID - - *February 27, 2018* `jenkins-operator` does not use `$BUILD_ID` as a fallback +- *February 27, 2018* `jenkins-operator` does not use `$BUILD_ID` as a fallback to `$PROW_JOB_ID` anymore. - - *February 15, 2018* `jenkins-operator` can now accept the `--tot-url` flag +- *February 15, 2018* `jenkins-operator` can now accept the `--tot-url` flag and will use the connection to `tot` to vend build identifiers as `plank` does, giving control over where in GCS artifacts land to Prow and away from Jenkins. Furthermore, the `$BUILD_ID` variable in Jenkins jobs will now @@ -428,61 +446,63 @@ used when updating deployments. `$PROW_JOB_ID`, points to the identifier used to link ProwJobs to Jenkins builds. `$PROW_JOB_ID` fallbacks to `$BUILD_ID` for backwards-compatibility, ie. to not break in-flight jobs during the time of the jenkins-operator update. - - *February 1, 2018* The `config_updater` section in `plugins.yaml` +- *February 1, 2018* The `config_updater` section in `plugins.yaml` now uses a `maps` object instead of `config_file`, `plugin_file` strings. Please switch over before July. - - *November 30, 2017* If you use tide, you'll need to switch your query format +- *November 30, 2017* If you use tide, you'll need to switch your query format and bump all prow component versions to reflect the changes in #5754. - - *November 14, 2017* `horologium:0.17` fixes cron job not being scheduled. - - *November 10, 2017* If you want to use cron feature in prow, you need to bump to: +- *November 14, 2017* `horologium:0.17` fixes cron job not being scheduled. +- *November 10, 2017* If you want to use cron feature in prow, you need to bump to: `hook:0.181`, `sinker:0.23`, `deck:0.62`, `splice:0.32`, `horologium:0.15` `plank:0.60`, `jenkins-operator:0.57` and `tide:0.12` to avoid error spamming from the config parser. - - *November 7, 2017* `plank:0.56` fixes bug introduced in `plank:0.53` that +- *November 7, 2017* `plank:0.56` fixes bug introduced in `plank:0.53` that affects controllers using an empty kubernetes selector. - - *November 7, 2017* `jenkins-operator:0.51` provides jobs with the `$BUILD_ID` +- *November 7, 2017* `jenkins-operator:0.51` provides jobs with the `$BUILD_ID` variable as well as the `$buildId` variable. The latter is deprecated and will be removed in a future version. - - *November 6, 2017* `plank:0.55` provides `Pods` with the `$BUILD_ID` variable +- *November 6, 2017* `plank:0.55` provides `Pods` with the `$BUILD_ID` variable as well as the `$BUILD_NUMBER` variable. The latter is deprecated and will be removed in a future version. - - *November 3, 2017* Added `EmptyDir` volume type. To update to `hook:0.176+` +- *November 3, 2017* Added `EmptyDir` volume type. To update to `hook:0.176+` or `horologium:0.11+` the following components must have the associated minimum versions: `deck:0.58+`, `plank:0.54+`, `jenkins-operator:0.50+`. - - *November 2, 2017* `plank:0.53` changes the `type` label key to `prow.k8s.io/type` +- *November 2, 2017* `plank:0.53` changes the `type` label key to `prow.k8s.io/type` and the `job` annotation key to `prow.k8s.io/job` added in pods. - - *October 14, 2017* `deck:0:53+` needs to be updated in conjunction with +- *October 14, 2017* `deck:0:53+` needs to be updated in conjunction with `jenkins-operator:0:48+` since Jenkins logs are now exposed from the operator and `deck` needs to use the `external_agent_logs` option in order to redirect requests to the location `jenkins-operator` exposes logs. - - *October 13, 2017* `hook:0.174`, `plank:0.50`, and `jenkins-operator:0.47` +- *October 13, 2017* `hook:0.174`, `plank:0.50`, and `jenkins-operator:0.47` drop the deprecated `github-bot-name` flag. - - *October 2, 2017* `hook:0.171`: The label plugin was split into three +- *October 2, 2017* `hook:0.171`: The label plugin was split into three plugins (label, sigmention, milestonestatus). Breaking changes: - - The configuration key for the milestone maintainer team's ID has been + - The configuration key for the milestone maintainer team's ID has been changed. Previously the team ID was stored in the plugins config at key `label`>>`milestone_maintainers_id`. Now that the milestone status labels are handled in the `milestonestatus` plugin instead of the `label` plugin, the team ID is stored at key `milestonestatus`>>`maintainers_id`. - - The sigmention and milestonestatus plugins must be enabled on any repos + - The sigmention and milestonestatus plugins must be enabled on any repos that require them since their functionality is no longer included in the label plugin. - - *September 3, 2017* `sinker:0.17` now deletes pods labeled by `plank:0.42` in +- *September 3, 2017* `sinker:0.17` now deletes pods labeled by `plank:0.42` in order to avoid cleaning up unrelated pods that happen to be found in the same namespace prow runs pods. If you run other pods in the same namespace, you will have to manually delete or label the prow-owned pods, otherwise you can bulk-label all of them with the following command and let sinker collect them normally: - ``` + + ```shell kubectl label pods --all -n pod_namespace created-by-prow=true ``` - - *September 1, 2017* `deck:0.44` and `jenkins-operator:0.41` controllers + +- *September 1, 2017* `deck:0.44` and `jenkins-operator:0.41` controllers no longer provide a default value for the `--jenkins-token-file` flag. Cluster administrators should provide `--jenkins-token-file=/etc/jenkins/jenkins` explicitly when upgrading to a new version of these components if they were previously relying on the default. For more context, please see [this pull request.](https://github.com/kubernetes/test-infra/pull/4210) - - *August 29, 2017* Configuration specific to plugins is now held in the +- *August 29, 2017* Configuration specific to plugins is now held in the `plugins` `ConfigMap` and serialized in this repo in the `plugins.yaml` file. Cluster administrators upgrading to `hook:0.148` or newer should move plugin configuration from the main `ConfigMap`. For more context, please see @@ -490,10 +510,10 @@ used when updating deployments. ## Project updates - - *October 28, 2022* Documentation migration: existing Markdown files in +- *October 28, 2022* Documentation migration: existing Markdown files in [k/t-i][] have been migrated over to - https://docs.prow.k8s.io/docs/legacy-snapshot. The old locations now have + . The old locations now have "tombstones" to redirect to the new location. See - https://github.com/kubernetes/test-infra/pull/27818 for details. + for details. [k/t-i]: https://github.com/kubernetes/test-infra diff --git a/site/content/en/docs/build-clusters.md b/site/content/en/docs/build-clusters.md index 5dbf24d248..eab6581656 100644 --- a/site/content/en/docs/build-clusters.md +++ b/site/content/en/docs/build-clusters.md @@ -2,7 +2,7 @@ title: "GKE Build Clusters" description: > Sometimes you might want to run all of your jobs in a GKE cluster that is separate from Prow (perhaps running in your own GCP project). What you need is a dedicated "build cluster". -weight: 3 +weight: 40 --- *Note: This page discusses build clusters that use GKE. Technically speaking, a diff --git a/site/content/en/docs/Legacy Snapshot/prow/build_test_update.md b/site/content/en/docs/build-test-update.md similarity index 90% rename from site/content/en/docs/Legacy Snapshot/prow/build_test_update.md rename to site/content/en/docs/build-test-update.md index 5bb0d2a0a6..6f1052137e 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/build_test_update.md +++ b/site/content/en/docs/build-test-update.md @@ -1,10 +1,11 @@ --- -title: "prow/build_test_update.md" +title: "Building, Testing, and Updating Prow" +weight: 70 +description: > + --- -# Building, Testing, and Updating Prow - -This guide is directed at Prow developers and maintainers who want to build/test individual components or deploy changes to an existing Prow cluster. [`getting_started_deploy.md`](https://github.com/kubernetes/test-infra/tree/master/prow/getting_started_deploy.md) is a better reference for deploying a new Prow cluster. +This guide is directed at Prow developers and maintainers who want to build/test individual components or deploy changes to an existing Prow cluster. ["Deploying Prow"](/docs/getting-started-deploy/) is a better reference for deploying a new Prow cluster. ## How to build and test Prow @@ -22,7 +23,7 @@ Unit test with: ```shell make test ``` -Integration test with([more details](https://github.com/kubernetes/test-infra/tree/master/prow/test/integration)): +Integration test with([more details](/docs/test/integration/)): ```shell ./prow/test/integration/integration-test.sh ``` @@ -36,7 +37,7 @@ tests, and for e2e test use [`runlocal`](https://github.com/kubernetes/test-infr ### How to test a plugin -If you are making changes to a Prow plugin you can test the new behavior by sending fake webhooks to [`hook`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/hook) with [`phony`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/phony#phony). +If you are making changes to a Prow plugin you can test the new behavior by sending fake webhooks to [`hook`](/docs/components/core/hook/) with [`phony`](/docs/components/cli-tools/phony/). ## How to update the cluster @@ -142,9 +143,9 @@ export KUBECONFIG='//.kube/kind-config-mkpod' After pointing to the correct master you will be able to drop into the container using `kubectl exec -it `. **This pod will only last the lifecycle of the job, if you need more time to debug you might add a `sleep` within the job execution. #### Using Phaino -[Phaino](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/phaino) lets you interactively mock and run the job locally on +[Phaino](/docs/components/cli-tools/phaino/) lets you interactively mock and run the job locally on your workstation in a docker container. Detailed instructions can be found in -Phaino's [Readme](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/phaino/README.md). +Phaino's [Readme](/docs/components/cli-tools/phaino/). Note: Test containers designed for decorated jobs (configured with `decorate: true`) may behave incorrectly or fail entirely without the environment the pod utilities @@ -184,6 +185,6 @@ to generate ProwJob YAML. [Docker]: https://docs.docker.com/install/ [kubectl]: https://kubernetes.io/docs/tasks/tools/install-kubectl/ [Kind]: https://sigs.k8s.io/kind -[mkpj]: https://github.com/kubernetes/test-infra/tree/master/prow/cmd/mkpj -[mkpod]: https://github.com/kubernetes/test-infra/tree/master/prow/cmd/mkpod +[mkpj]: /docs/components/cli-tools/mkpj/ +[mkpod]: /docs/components/cli-tools/mkpod/ [pj-on-kind.sh]: https://github.com/kubernetes/test-infra/tree/master/prow/pj-on-kind.sh diff --git a/site/content/en/docs/components/_index.md b/site/content/en/docs/components/_index.md new file mode 100644 index 0000000000..694bee20ba --- /dev/null +++ b/site/content/en/docs/components/_index.md @@ -0,0 +1,73 @@ +--- +title: "Components" +weight: 30 +--- + +## Prow Images + +This directory includes a sub directory for every Prow component and is where all binary and container images are built. You can find the `main` packages here. For details about building the binaries and images see ["Building, Testing, and Updating Prow"](/docs/build-test-update/). + +## Cluster Components + +Prow has a microservice architecture implemented as a collection of container images that run as Kubernetes deployments. A brief description of each service component is provided here. + +#### Core Components + +* `crier` ([doc](/docs/components/core/crier/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/crier)) reports on ProwJob status changes. Can be configured to report to gerrit, github, pubsub, slack, etc. +* `deck` ([doc](/docs/components/core/deck/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/deck)) presents a nice view of [recent jobs](https://prow.k8s.io/), [command](https://prow.k8s.io/command-help) and [plugin](https://prow.k8s.io/plugins) help information, the [current status](https://prow.k8s.io/tide) and [history](https://prow.k8s.io/tide-history) of merge automation, and a [dashboard for PR authors](https://prow.k8s.io/pr). +* `hook` ([doc](/docs/components/core/hook/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/hook)) is the most important piece. It is a stateless server that listens for GitHub webhooks and dispatches them to the appropriate plugins. Hook's plugins are used to trigger jobs, implement 'slash' commands, post to Slack, and more. See the plugins [doc](/docs/components/plugins/) and [code directory](https://github.com/kubernetes/test-infra/tree/master/prow/plugins) for more information on plugins. +* `horologium` ([doc](/docs/components/core/horologium/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/horologium)) triggers periodic jobs when necessary. +* `prow-controller-manager` ([doc](/docs/components/core/prow-controller-manager/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/prow-controller-manager)) manages the job execution and lifecycle for jobs that run in k8s pods. It currently acts as a replacement for [`plank`](/docs/components/deprecated/plank/) +* `sinker` ([doc](/docs/components/core/sinker/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/sinker)) cleans up old jobs and pods. + +#### Merge Automation + +* `tide` ([doc](/docs/components/core/tide/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide)) manages retesting and merging PRs once they meet the configured merge criteria. See [its README](/docs/components/core/tide/) for more information. + +#### Optional Components + +* `branchprotector` ([doc](/docs/components/optional/branchprotector/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/branchprotector)) configures [github branch protection](https://help.github.com/articles/about-protected-branches/) according to a specified policy +* `exporter` ([doc](/docs/components/optional/exporter/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/exporter)) exposes metrics about ProwJobs not directly related to a specific Prow component +* `gcsupload` ([doc](/docs/components/optional/gcsupload/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/gcsupload)) +* `gerrit` ([doc](/docs/components/optional/gerrit/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/gerrit)) is a Prow-gerrit adapter for handling CI on [gerrit](https://www.gerritcodereview.com/) workflows +* `hmac` ([doc](/docs/components/optional/hmac/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/hmac)) updates HMAC tokens, GitHub webhooks and HMAC secrets for the orgs/repos specified in the Prow config file +* `jenkins-operator` ([doc](/docs/components/optional/jenkins-operator/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/jenkins-operator)) is the controller that manages jobs that run on Jenkins. We moved away from using this component in favor of running all jobs on Kubernetes. +* `tot` ([doc](/docs/components/optional/tot/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tot)) vends sequential build numbers. Tot is only necessary for integration with automation that expects sequential build numbers. If Tot is not used, Prow automatically generates build numbers that are monotonically increasing, but not sequential. +* `status-reconciler` ([doc](/docs/components/optional/status-reconciler/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/status-reconciler)) ensures changes to blocking presubmits in Prow configuration does not cause in-flight GitHub PRs to get stuck +* `sub` ([doc](/docs/components/optional/sub/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/sub)) listen to Cloud Pub/Sub notification to trigger Prow Jobs. + +## CLI Tools + +* `checkconfig` ([doc](/docs/components/cli-tools/checkconfig/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/checkconfig)) loads and verifies the configuration, useful as a pre-submit. +* `config-bootstrapper` ([doc](/docs/components/cli-tools/config-bootstrapper/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/config-bootstrapper)) bootstraps a configuration that would be incrementally updated by the [`updateconfig` Prow plugin](/docs/components/plugins/updateconfig/) +* `generic-autobumper` ([doc](/docs/components/cli-tools/generic-autobumper/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/generic-autobumper)) automates image version upgrades (e.g. for a Prow deployment) by opening a PR with images changed to their latest version according to a config file. +* `invitations-accepter` ([doc](/docs/components/cli-tools/invitations-accepter/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/invitations-accepter)) approves all pending GitHub repository invitations +* `mkpj` ([doc](/docs/components/cli-tools/mkpj/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/mkpj)) creates `ProwJobs` using Prow configuration. +* `mkpod` ([doc](/docs/components/cli-tools/mkpod/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/mkpod)) creates `Pods` from `ProwJobs`. +* `peribolos` ([doc](/docs/components/cli-tools/peribolos/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/peribolos)) manages GitHub org, team and membership settings according to a config file. Used by [kubernetes/org](https://github.com/kubernetes/org) +* `phaino` ([doc](/docs/components/cli-tools/phaino/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/phaino)) runs an approximation of a ProwJob on your local workstation +* `phony` ([doc](/docs/components/cli-tools/phony/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/phony)) sends fake webhooks for testing hook and plugins. + +## Pod Utilities + +These are small tools that are automatically added to ProwJob pods for jobs that request pod decoration. They are used to transparently provide source code cloning and upload of metadata, logs, and job artifacts to persistent storage. See [their README](/docs/components/pod-utilities/) for more information. + +* `clonerefs` ([doc](/docs/components/pod-utilities/clonerefs/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/clonerefs)) +* `initupload` ([doc](/docs/components/pod-utilities/initupload/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/initupload)) +* `entrypoint` ([doc](/docs/components/pod-utilities/entrypoint/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/entrypoint)) +* `sidecar` ([doc](/docs/components/pod-utilities/sidecar/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/sidecar)) + +## Base Images + +The container images in [`images`](https://github.com/kubernetes/test-infra/tree/master/images) are used as base images for Prow components. + +## TODO: undocumented + +* `admission` ([doc](/docs/components/undocumented/admission/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/admission)) +* `grandmatriarch` ([doc](/docs/components/undocumented/grandmatriarch/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/grandmatriarch)) +* `pipeline` ([doc](/docs/components/undocumented/pipeline/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/pipeline)) +* `tackle` ([doc](/docs/components/undocumented/tackle/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tackle)) + +## Deprecated + +* `cm2kc` ([doc](/docs/components/deprecated/cm2kc/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/cm2kc)) is a CLI tool used to convert a [clustermap file](/docs/getting-started-deploy/#run-test-pods-in-different-clusters) to a [kubeconfig file](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/). Deprecated because we have moved away from clustermaps; you should use [`gencred`](https://github.com/kubernetes/test-infra/tree/master/gencred) to generate a [kubeconfig file](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) directly. diff --git a/site/content/en/docs/components/cli-tools/_index.md b/site/content/en/docs/components/cli-tools/_index.md new file mode 100644 index 0000000000..e788827189 --- /dev/null +++ b/site/content/en/docs/components/cli-tools/_index.md @@ -0,0 +1,6 @@ +--- +title: "CLI Tools" +weight: 30 +description: > + +--- diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/checkconfig/README.md b/site/content/en/docs/components/cli-tools/checkconfig.md similarity index 82% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/checkconfig/README.md rename to site/content/en/docs/components/cli-tools/checkconfig.md index 4f953b00f9..6d6efad3ff 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/checkconfig/README.md +++ b/site/content/en/docs/components/cli-tools/checkconfig.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/checkconfig/README.md" +title: "checkconfig" +weight: 10 +description: > + --- -# `checkconfig` - `checkconfig` loads the Prow configuration given with `--config-path`, `--job-config-path` and `--plugin-config` in order to validate it. Use `checkconfig` as a pre-submit for any repository holding Prow diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/config-bootstrapper/README.md b/site/content/en/docs/components/cli-tools/config-bootstrapper.md similarity index 92% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/config-bootstrapper/README.md rename to site/content/en/docs/components/cli-tools/config-bootstrapper.md index 5a49dfe194..29d354c444 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/config-bootstrapper/README.md +++ b/site/content/en/docs/components/cli-tools/config-bootstrapper.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/config-bootstrapper/README.md" +title: "config-bootstrapper" +weight: 10 +description: > + --- -# `config-bootstrapper` - `config-bootstrapper` is used to bootstrap a configuration that would be incrementally updated by the config-updater Prow plugin. @@ -15,7 +16,8 @@ base state and hand off ownership to the plugin for updates. Provide the config-bootstrapper with the latest state of the Prow configuration (plugins.yaml, config.yaml, any job configuration files) to boot-strap with the latest configuration. Sample usage: -``` + +```shell ./config-bootstrapper \ --dry-run=false \ --source-path=. \ diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/generic-autobumper/README.md b/site/content/en/docs/components/cli-tools/generic-autobumper.md similarity index 98% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/generic-autobumper/README.md rename to site/content/en/docs/components/cli-tools/generic-autobumper.md index 2ab24d3867..799faeae1e 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/generic-autobumper/README.md +++ b/site/content/en/docs/components/cli-tools/generic-autobumper.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/generic-autobumper/README.md" +title: "generic-autobumper" +weight: 10 +description: > + --- -# Autobumper - This tool automates the version upgrading of images such as the [prow.k8s.io](https://prow.k8s.io) Prow deployment. Its workflow is: @@ -25,6 +26,7 @@ a postsubmit job, e.g., [`post-test-infra-deploy-prow`](https://github.com/kuber for [prow.k8s.io](https://prow.k8s.io/) is defined for deploying the yaml files. ### Requirement + We need to fulfil those requirements to use this tool: * a "committable" local repo, i.e., `git-commit` command can be executed successfully, e.g., `git-config` is set up correctly. @@ -44,6 +46,7 @@ We need to fulfil those requirements to use this tool: * For info about what should go in the config look at [the documentation for the Options here](https://pkg.go.dev/k8s.io/test-infra/prow/cmd/generic-autobumper/bumper#Options) and look at the example below. e.g., + ```yaml gitHubLogin: "k8s-ci-robot" gitHubToken: "/etc/github-token/oauth" @@ -86,4 +89,3 @@ prefixes: summarise: false consistentImages: false ``` - diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/invitations-accepter/README.md b/site/content/en/docs/components/cli-tools/invitations-accepter.md similarity index 78% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/invitations-accepter/README.md rename to site/content/en/docs/components/cli-tools/invitations-accepter.md index 6c96903e3e..c9f9d5e534 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/invitations-accepter/README.md +++ b/site/content/en/docs/components/cli-tools/invitations-accepter.md @@ -1,14 +1,16 @@ --- -title: "prow/cmd/invitations-accepter/README.md" +title: "invitations-accepter" +weight: 10 +description: > + --- -# Invitations Accepter - -The `invitations-accepter` tool approves all pending repository invitations. +The `invitations-accepter` tool approves all pending repository invitations. ## Usage *example*: + ```sh invitations-accepter --dry-run=false --github-token-path=/etc/github/oauth ``` @@ -18,4 +20,4 @@ using with GitHub Apps ```sh invitations-accepter --dry-run=false --github-app-id=12345 --github-app-private-key-path=/etc/github/cert -``` \ No newline at end of file +``` diff --git a/site/content/en/docs/components/cli-tools/mkpj.md b/site/content/en/docs/components/cli-tools/mkpj.md new file mode 100644 index 0000000000..24949a2caf --- /dev/null +++ b/site/content/en/docs/components/cli-tools/mkpj.md @@ -0,0 +1,8 @@ +--- +title: "mkpj" +weight: 10 +description: > + +--- + +This is a placeholder page. Some contents needs to be filled. diff --git a/site/content/en/docs/components/cli-tools/mkpod.md b/site/content/en/docs/components/cli-tools/mkpod.md new file mode 100644 index 0000000000..d367b78910 --- /dev/null +++ b/site/content/en/docs/components/cli-tools/mkpod.md @@ -0,0 +1,8 @@ +--- +title: "mkpod" +weight: 10 +description: > + +--- + +This is a placeholder page. Some contents needs to be filled. diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/peribolos/README.md b/site/content/en/docs/components/cli-tools/peribolos.md similarity index 92% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/peribolos/README.md rename to site/content/en/docs/components/cli-tools/peribolos.md index faf1794976..6c22c52d57 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/peribolos/README.md +++ b/site/content/en/docs/components/cli-tools/peribolos.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/peribolos/README.md" +title: "Peribolos" +weight: 10 +description: > + --- -# Peribolos Documentation - Peribolos allows the org settings, teams and memberships to be declared in a yaml file. GitHub is then updated to match the declared configuration. See the [kubernetes/org] repo, in particular the [merge] and [`update.sh`] parts of that repo for this tool in action. @@ -63,18 +64,19 @@ orgs: ``` This config will: + * Ensure the org settings match the following: - - Set the company, email, name and descriptions fields for the org to foo - - Allow projects to be created at the org and repo levels - - Give everyone read access to repos by default - - Disallow members from creating repositories + * Set the company, email, name and descriptions fields for the org to foo + * Allow projects to be created at the org and repo levels + * Give everyone read access to repos by default + * Disallow members from creating repositories * Ensure the following memberships exist: - - anne and bob are members, carl is an admin + * anne and bob are members, carl is an admin * Configure the node and another-team in the following manner: - - Set node's description and privacy setting. - - Rename the backend team to node - - Add anne as a member and jane as a maintainer to node - - Similar things for another-team (details elided) + * Set node's description and privacy setting. + * Rename the backend team to node + * Add anne as a member and jane as a maintainer to node + * Similar things for another-team (details elided) * Ensure that the team has admin rights to `some-repo`, read access to `other-repo` and no other privileges Note that any fields missing from the config will not be managed by peribolos. So if description is missing from the org setting, the current value will remain. @@ -146,8 +148,6 @@ $ go run ./prow/cmd/peribolos --config-path ~/current.yaml --github-token-path ~ ... ``` - - ## Settings In order to mitigate the chance of applying erroneous configs, the peribolos binary includes a few safety checks: @@ -164,11 +164,8 @@ This flag is designed to protect against typos in the configuration which might * `--confirm=false` - no github mutations will be made until this flag is true. It is safe to run the binary without this flag. It will print what it would do, without actually making any changes. - See `go run ./prow/cmd/peribolos --help` for the full and current list of settings that can be configured with flags. - - [`config.yaml`]: https://github.com/kubernetes/test-infra/tree/master/config/prow/config.yaml [edit team]: https://developer.github.com/v3/teams/#edit-team [edit org]: https://developer.github.com/v3/orgs/#edit-an-organization diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/phaino/README.md b/site/content/en/docs/components/cli-tools/phaino.md similarity index 89% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/phaino/README.md rename to site/content/en/docs/components/cli-tools/phaino.md index 883e7398b6..721667d607 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/phaino/README.md +++ b/site/content/en/docs/components/cli-tools/phaino.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/phaino/README.md" +title: "Phaino" +weight: 10 +description: > + --- -# Phaino - Run prowjobs on your local workstation with `phaino`. Plato believed that [ideas and forms] are the ultimate truth, @@ -20,6 +21,7 @@ Phaino shares a prefix with [Pharos], meaning lighthouse and in particular the a ## Usage Usage: + ```console # Use a job from deck go run ./prow/cmd/phaino $URL # or /path/to/prowjob.yaml @@ -50,54 +52,65 @@ volumes that the Prow Job may require. Phaino is smart at prompting for where repo is located, volume mounts etc., if it's desired to save the prompts, use the following tricks instead: -- If the repo needs to be cloned under GOPATH, use: +* If the repo needs to be cloned under GOPATH, use: + ``` --code-mount-path==/whatever/go/src # Controls where source code is mounted in container --extra-volume-mounts=/whatever/go/src/k8s.io/test-infra=/Users/xyz/k8s-test-infra ``` + - If job requires mounting kubeconfig, assume the mount is named `kubeconfig`,use: + ``` --use-local-kubeconfig --skip-volume-mounts=kubeconfig ``` + - If job requires mounting gcloud default credentials, assume the mount is named `service-account`,use: + ``` --use-local-gcloud-credentials --skip-volume-mounts=service-account ``` + - If job requires mounting something else like `name:foo; mountPath: /bar`,use: + ``` --extra-volume-mounts=/bar=/Users/xyz/local/bar --skip-volume-mounts=foo ``` + - If job requires env vars,use: + ``` --extra-envs=env1=val1,env2=val2 ``` - See `go run ./prow/cmd/phaino --help` for full option list. ### Usage examples + #### URL example * Go to your [deck deployment](https://prow.k8s.io) * Pick a job and click the rerun icon on the left * Copy the URL (something like `https://prow.k8s.io/rerun?prowjob=d08f1ca5-5d63-11e9-ab62-0a580a6c1281`) * Paste it as a phaino arg - - `go run ./prow/cmd/phaino https://prow.k8s.io/rerun?prowjob=d08f1ca5-5d63-11e9-ab62-0a580a6c1281` - - Alternatively `go run ./prow/cmd/phaino <(curl $URL)` + * `go run ./prow/cmd/phaino https://prow.k8s.io/rerun?prowjob=d08f1ca5-5d63-11e9-ab62-0a580a6c1281` + * Alternatively `go run ./prow/cmd/phaino <(curl $URL)` +#### Configuration example -#### Configuration example: +* Use [`mkpj`](/docs/components/cli-tools/mkpj/) to create the job and pipe this to `phaino` + * For prow.k8s.io jobs use `//config:mkpj` -* Use [`mkpj`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/mkpj) to create the job and pipe this to `phaino` - - For prow.k8s.io jobs use `//config:mkpj` ``` go run ./config:mkpj --job=pull-test-infra-bazel > /tmp/foo go run ./prow/cmd/phaino /tmp/foo ``` - - Other deployments will need to clone that rule and/or pass in extra flags: + + * Other deployments will need to clone that rule and/or pass in extra flags: + ``` go run ./prow/cmd/mkpj --config-path=/my/config.yaml --job=my-job go run ./prow/cmd/phaino /tmp/foo diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/phony/README.md b/site/content/en/docs/components/cli-tools/phony.md similarity index 56% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/phony/README.md rename to site/content/en/docs/components/cli-tools/phony.md index 1122cbc00c..501bc97be2 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/phony/README.md +++ b/site/content/en/docs/components/cli-tools/phony.md @@ -1,45 +1,51 @@ --- -title: "prow/cmd/phony/README.md" +title: "Phony" +weight: 10 +description: > + --- -# Phony - `phony` sends fake GitHub webhooks. ## Running a GitHub event manager -`phony` is most commonly used for testing [`hook`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/hook) and its [plugins](https://github.com/kubernetes/test-infra/tree/master/prow/plugins), but can be used for testing any externally exposed service configured to receive GitHub events (external plugins). + +`phony` is most commonly used for testing [`hook`](/docs/components/core/hook/) and its [plugins](/docs/components/plugins/), but can be used for testing any externally exposed service configured to receive GitHub events (external plugins). To get an idea of `phony`'s behavior, start a local instance of `hook` with this: + ``` go run prow/cmd/hook/main.go \ - --config-path=config/prow/config.yaml \ - --plugin-config=config/prow/plugins.yaml \ - --hmac-secret-file=path/to/hmac \ - --github-token-path=path/to/github-token + --config-path=config/prow/config.yaml \ + --plugin-config=config/prow/plugins.yaml \ + --hmac-secret-file=path/to/hmac \ + --github-token-path=path/to/github-token # Note: # --hmac-secret-file is required for running locally, use the same hmac token for phony below ``` ## Usage + Once you have a running server that manages github webhook events, generate an -`hmac` token (same process as in [prow](https://github.com/kubernetes/test-infra/tree/master/prow)), and point a `phony` pull +`hmac` token (same process as in [prow](/docs/overview/)), and point a `phony` pull request event at it with the following: + ``` phony --help Usage of ./phony: -address string - Where to send the fake hook. (default "http://localhost:8888/hook") + Where to send the fake hook. (default "http://localhost:8888/hook") -event string - Type of event to send, such as pull_request. (default "ping") + Type of event to send, such as pull_request. (default "ping") -hmac string - HMAC token to sign payload with. (default "abcde12345") + HMAC token to sign payload with. (default "abcde12345") -payload string - File to send as payload. If unspecified, sends "{}". + File to send as payload. If unspecified, sends "{}". ``` If you are testing `hook` and successfully sent the webhook from `phony`, you should see a log from `hook` resembling the following: + ``` {"author":"","component":"hook","event-GUID":"GUID","event-type":"pull_request","level":"info","msg":"Pull request .","org":"","pr":0,"repo":"","time":"2018-05-29T11:38:57-07:00","url":""} ``` diff --git a/site/content/en/docs/components/core/_index.md b/site/content/en/docs/components/core/_index.md new file mode 100644 index 0000000000..a8db4bf553 --- /dev/null +++ b/site/content/en/docs/components/core/_index.md @@ -0,0 +1,6 @@ +--- +title: "Core Components" +weight: 10 +description: > + +--- diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/crier/README.md b/site/content/en/docs/components/core/crier.md similarity index 96% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/crier/README.md rename to site/content/en/docs/components/core/crier.md index b1b0819c5f..f19c2155bb 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/crier/README.md +++ b/site/content/en/docs/components/core/crier.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/crier/README.md" +title: "Crier" +weight: 10 +description: > + --- -# Crier - Crier reports your prowjobs on their status changes. ## Usage / How to enable existing available reporters @@ -15,10 +16,10 @@ flag as most of other prow controllers do. You can enable gerrit reporter in crier by specifying `--gerrit-workers=n` flag. -Similar to the [gerrit adapter](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/gerrit), you'll need to specify `--gerrit-projects` for +Similar to the [gerrit adapter](/docs/components/optional/gerrit/), you'll need to specify `--gerrit-projects` for your gerrit projects, and also `--cookiefile` for the gerrit auth token (leave it unset for anonymous). -Gerrit reporter will send an aggregated summary message, when all [gerrit adapter](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/gerrit) +Gerrit reporter will send an aggregated summary message, when all [gerrit adapter](/docs/components/optional/gerrit/) scheduled prowjobs with the same report label finish on a revision. It will also attach a report url so people can find logs of the job. @@ -63,7 +64,8 @@ You can enable the Slack reporter in crier by specifying the `--slack-workers=n` The `--slack-token-file` flag takes a path to a file containing a Slack [**OAuth Access Token**](https://api.slack.com/docs/oauth). The **OAuth Access Token** can be obtained as follows: -1. Navigate to: https://api.slack.com/apps. + +1. Navigate to: . 1. Click **Create New App**. 1. Provide an **App Name** (e.g. Prow Slack Reporter) and **Development Slack Workspace** (e.g. Kubernetes). 1. Click **Permissions**. @@ -160,6 +162,7 @@ slack_reporter_configs: ``` The `channel`, `job_states_to_report` and `report_template` can be overridden at the ProwJob level via the `reporter_config.slack` field: + ```yaml postsubmits: some-org/some-repo: @@ -177,8 +180,10 @@ postsubmits: command: - echo ``` + To silence notifications at the ProwJob level you can pass an empty slice to `reporter_config.slack.job_states_to_report`: postsubmits: + ```yaml some-org/some-repo: - name: example-job @@ -201,17 +206,17 @@ will get prowjob change notifications from a [shared informer](https://github.co If you are interested in how client-go works under the hood, the details are explained [in this doc](https://github.com/kubernetes/sample-controller/blob/master/docs/controller-client-go.md) - ## Adding a new reporter Each crier controller takes in a reporter. Each reporter will implement the following interface: + ```go type reportClient interface { - Report(pj *v1.ProwJob) error - GetName() string - ShouldReport(pj *v1.ProwJob) bool + Report(pj *v1.ProwJob) error + GetName() string + ShouldReport(pj *v1.ProwJob) bool } ``` @@ -237,6 +242,7 @@ Before migrating, be sure plank is setting the [PrevReportStates field](https:// by describing a finished presubmit prowjob. Plank started to set this field after commit [2118178](https://github.com/kubernetes/test-infra/pull/10975/commits/211817826fc3c4f3315a02e46f3d6aa35573d22f), if not, you want to upgrade your plank to a version includes this commit before moving forward. you can check this entry by: + ```sh $ kubectl get prowjobs -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.prev_report_states.github-reporter}{"\n"}' ... @@ -253,12 +259,14 @@ fb09e7d8-3abb-11e9-816a-0a580a6c0f7f success You want to add a crier deployment, similar to ours [config/prow/cluster/crier_deployment.yaml](https://github.com/kubernetes/test-infra/blob/de3775a7480fe0a724baacf24a87cbf058cd9fd5/prow/cluster/crier_deployment.yaml), flags need to be specified: + - point `config-path` and `--job-config-path` to your prow config and job configs accordingly. - Set `--github-worker` to be number of parallel github reporting threads you need - Point `--github-endpoint` to ghproxy, if you have set that for plank - Bind github oauth token as a secret and set `--github-token-path` if you've have that set for plank. In your plank deployment, you can + - Remove the `--github-endpoint` flags - Remove the github oauth secret, and `--github-token-path` flag if set - Flip on `--skip-report`, so plank will skip the reporting logic diff --git a/site/content/en/docs/Components/Deck/_index.md b/site/content/en/docs/components/core/deck/_index.md similarity index 97% rename from site/content/en/docs/Components/Deck/_index.md rename to site/content/en/docs/components/core/deck/_index.md index fa7c6b1268..6c99230ace 100644 --- a/site/content/en/docs/Components/Deck/_index.md +++ b/site/content/en/docs/components/core/deck/_index.md @@ -1,6 +1,6 @@ --- title: "Deck" -weight: 10 +weight: 20 description: > Shows what jobs are running or have recently run in prow --- diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/deck/csrf.md b/site/content/en/docs/components/core/deck/csrf.md similarity index 97% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/deck/csrf.md rename to site/content/en/docs/components/core/deck/csrf.md index 4021782cc8..4371dc571e 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/deck/csrf.md +++ b/site/content/en/docs/components/core/deck/csrf.md @@ -1,14 +1,17 @@ --- -title: "prow/cmd/deck/csrf.md" +title: "CSRF attacks" +weight: 20 +description: > + --- -# CSRF attacks In Deck, we make a number of `POST` requests that require user authentication. These requests are susceptible to [cross site request forgery (CSRF) attacks](https://en.wikipedia.org/wiki/Cross-site_request_forgery), in which a malicious actor tricks an already authenticated user into submitting a form to one of these endpoints and performing one of these protected actions on their behalf. -# Protection +## Protection + If `--cookie-secret` is 32 or more bytes long, CSRF protection is automatically enabled. If `--rerun-creates-job` is specified, CSRF protection is required, and accordingly, `--cookie-secret` must be 32 bytes long. diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/deck/github_oauth_setup.md b/site/content/en/docs/components/core/deck/github-oauth-setup.md similarity index 98% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/deck/github_oauth_setup.md rename to site/content/en/docs/components/core/deck/github-oauth-setup.md index 4867e97740..b5000f5e78 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/deck/github_oauth_setup.md +++ b/site/content/en/docs/components/core/deck/github-oauth-setup.md @@ -1,8 +1,10 @@ --- -title: "prow/cmd/deck/github_oauth_setup.md" +title: "How to setup GitHub Oauth" +weight: 10 +description: > + --- -# How to setup GitHub Oauth This document helps configure GitHub Oauth, which is required for [PR Status](https://prow.k8s.io/pr) and for the rerun button on [Prow Status](https://prow.k8s.io). If OAuth is configured, Prow will perform GitHub actions on behalf of the authenticated users. @@ -10,6 +12,7 @@ This is necessary to fetch information about pull requests for the PR Status pag authenticate users when checking if they have permission to rerun jobs via the rerun button on Prow Status. ## Set up secrets + The following steps will show you how to set up an OAuth app. 1. Create your GitHub Oauth application @@ -81,6 +84,7 @@ The following steps will show you how to set up an OAuth app. * To prevent `deck` from making mutating GitHub API calls, pass in the `--dry-run` flag. ## Using A GitHub bot + The rerun button can be configured so that certain GitHub teams are allowed to trigger certain jobs from the frontend. In order to make API calls to determine whether a user is on a given team, `deck` needs to use the access token of an org member. @@ -93,6 +97,7 @@ Then create the access token secret: `kubectl create secret generic oauth-token --from-file=secret=` Add the following to `volumes` and `volumeMounts`: + ```yaml volumeMounts: - name: oauth-token @@ -111,6 +116,7 @@ Pass the file path to `deck` as a flag: You can optionally use [ghproxy](https://github.com/kubernetes/test-infra/blob/master/ghproxy/README.md) to reduce token usage. ## Run PR Status endpoint locally + Firstly, you will need a GitHub OAuth app. Please visit step 1 - 3 above. When testing locally, pass the path to your secrets to `deck` using the `--github-oauth-config-file` and `--cookie-secret` flags. @@ -120,4 +126,5 @@ Run the command: `go build . && ./deck --config-path=../../../config/prow/config.yaml --github-oauth-config-file= --cookie-secret= --oauth-url=/pr` ## Using a test cluster + If hosting your test instance on http instead of https, you will need to use the `--allow-insecure` flag in `deck`. diff --git a/site/content/en/docs/Components/Deck/prow_abort.png b/site/content/en/docs/components/core/deck/prow_abort.png similarity index 100% rename from site/content/en/docs/Components/Deck/prow_abort.png rename to site/content/en/docs/components/core/deck/prow_abort.png diff --git a/site/content/en/docs/Components/Deck/prow_rerun.png b/site/content/en/docs/components/core/deck/prow_rerun.png similarity index 100% rename from site/content/en/docs/Components/Deck/prow_rerun.png rename to site/content/en/docs/components/core/deck/prow_rerun.png diff --git a/site/content/en/docs/Components/Deck/spyglass_abort.png b/site/content/en/docs/components/core/deck/spyglass_abort.png similarity index 100% rename from site/content/en/docs/Components/Deck/spyglass_abort.png rename to site/content/en/docs/components/core/deck/spyglass_abort.png diff --git a/site/content/en/docs/Components/Deck/spyglass_rerun.png b/site/content/en/docs/components/core/deck/spyglass_rerun.png similarity index 100% rename from site/content/en/docs/Components/Deck/spyglass_rerun.png rename to site/content/en/docs/components/core/deck/spyglass_rerun.png diff --git a/site/content/en/docs/components/core/hook.md b/site/content/en/docs/components/core/hook.md new file mode 100644 index 0000000000..b60131b7da --- /dev/null +++ b/site/content/en/docs/components/core/hook.md @@ -0,0 +1,8 @@ +--- +title: "Hook" +weight: 30 +description: > + +--- + +This is a placeholder page. Some contents needs to be filled. diff --git a/site/content/en/docs/components/core/horologium.md b/site/content/en/docs/components/core/horologium.md new file mode 100644 index 0000000000..9f85199cd8 --- /dev/null +++ b/site/content/en/docs/components/core/horologium.md @@ -0,0 +1,8 @@ +--- +title: "Horologium" +weight: 40 +description: > + +--- + +This is a placeholder page. Some contents needs to be filled. diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/prow-controller-manager/README.md b/site/content/en/docs/components/core/prow-controller-manager.md similarity index 79% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/prow-controller-manager/README.md rename to site/content/en/docs/components/core/prow-controller-manager.md index 11a8b30166..74de75d011 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/prow-controller-manager/README.md +++ b/site/content/en/docs/components/core/prow-controller-manager.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/prow-controller-manager/README.md" +title: "Prow-Controller-Manager" +weight: 50 +description: > + --- -# Prow-Controller-Manager - `prow-controller-manager` manages the job execution and lifecycle for jobs running in k8s. It currently acts as a replacement for [Plank]. @@ -32,6 +33,6 @@ $ go run ./prow/cmd/prow-controller-manager --help * [Deployment manifest](https://github.com/kubernetes/test-infra/tree/master/config/prow/cluster/prow_controller_manager_deployment.yaml) * [RBAC manifest](https://github.com/kubernetes/test-infra/tree/master/config/prow/cluster/prow_controller_manager_rbac.yaml) -[Plank]: https://github.com/kubernetes/test-infra/tree/master/prow/plank -[Sinker]: https://github.com/kubernetes/test-infra/tree/master/prow/cmd/sinker -[Crier]: https://github.com/kubernetes/test-infra/tree/master/prow/cmd/crier +[Plank]: /docs/components/deprecated/plank/ +[Sinker]: /docs/components/core/sinker/ +[Crier]: /docs/components/core/crier/ diff --git a/site/content/en/docs/components/core/sinker.md b/site/content/en/docs/components/core/sinker.md new file mode 100644 index 0000000000..80f70fe371 --- /dev/null +++ b/site/content/en/docs/components/core/sinker.md @@ -0,0 +1,8 @@ +--- +title: "Sinker" +weight: 60 +description: > + +--- + +This is a placeholder page. Some contents needs to be filled. diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/tide/README.md b/site/content/en/docs/components/core/tide/_index.md similarity index 90% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/tide/README.md rename to site/content/en/docs/components/core/tide/_index.md index 3f483d3edf..1667766c6a 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/tide/README.md +++ b/site/content/en/docs/components/core/tide/_index.md @@ -1,10 +1,11 @@ --- -title: "prow/cmd/tide/README.md" +title: "Tide" +weight: 70 +description: > + --- -# Tide - -Tide is a [Prow](https://github.com/kubernetes/test-infra/blob/master/prow/README.md) +Tide is a [Prow](/docs/overview/) component for managing a pool of GitHub PRs that match a given set of criteria. It will automatically retest PRs that meet the criteria ("tide comes in") and automatically merge them when they have up-to-date passing test results ("tide goes out"). @@ -12,19 +13,20 @@ them when they have up-to-date passing test results ("tide goes out"). [Open Issues](https://github.com/kubernetes/test-infra/issues?utf8=%E2%9C%93&q=is%3Aopen+is%3Aissue+label%3Aarea%2Fprow%2Ftide) ## Documentation -- [I JUST WANT MY PR TO MERGE!](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide/pr-authors.md) -- [Configuring Tide](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide/config.md) -- [Maintainer's Guide](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide/maintainers.md) +- [I JUST WANT MY PR TO MERGE!](/docs/components/core/tide/pr-authors/) +- [Configuring Tide](/docs/components/core/tide/config/) +- [Maintainer's Guide](/docs/components/core/tide/maintainers/) ## Features + - Automatically runs batch tests and merges multiple PRs together whenever possible. - Ensures that PRs are tested against the most recent base branch commit before they are allowed to merge. - Maintains a GitHub status context that indicates if each PR is in a pool or what requirements are missing. - Supports blocking merge to individual branches or whole repos using specifically labelled GitHub issues. - Exposes Prometheus metrics. - Supports repos that have 'optional' status contexts that shouldn't be required for merge. -- Serves live data about current pools and a history of actions which can be consumed by [Deck](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/deck) to populate the [Tide dashboard](https://prow.k8s.io/tide), the [PR dashboard](https://prow.k8s.io/pr), and the [Tide history page](https://prow.k8s.io/tide-history). +- Serves live data about current pools and a history of actions which can be consumed by [Deck](/docs/components/core/deck/) to populate the [Tide dashboard](https://prow.k8s.io/tide), the [PR dashboard](https://prow.k8s.io/pr), and the [Tide history page](https://prow.k8s.io/tide-history). - Scales efficiently so that a single instance with a single bot token can provide merge automation to dozens of orgs and repos with unique merge criteria. Every distinct 'org/repo:branch' combination defines a disjoint merge pool so that merges only affect other PRs in the same branch. - Provides configurable merge modes ('merge', 'squash', or 'rebase'). diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/tide/config.md b/site/content/en/docs/components/core/tide/config.md similarity index 97% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/tide/config.md rename to site/content/en/docs/components/core/tide/config.md index f6e7a4286c..becbcf1bf8 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/tide/config.md +++ b/site/content/en/docs/components/core/tide/config.md @@ -1,14 +1,15 @@ --- -title: "prow/cmd/tide/config.md" +title: "Configuring Tide" +weight: 10 +description: > + --- -# Configuring Tide - Configuration of Tide is located under the [config/prow/config.yaml](https://github.com/kubernetes/test-infra/tree/master/config/prow/config.yaml) file. All configuration for merge behavior and criteria belongs in the `tide` yaml struct, but it may be necessary to also configure presubmits for Tide to run against PRs (see ['Configuring Presubmit Jobs'](#configuring-presubmit-jobs) below). This document will describe the fields of the `tide` configuration and how to populate them, but you can also check out the [GoDocs](https://godoc.org/github.com/kubernetes/test-infra/prow/config#Tide) for the most up to date configuration specification. -To deploy Tide for your organization or repository, please see [how to get started with prow](https://github.com/kubernetes/test-infra/tree/master/prow/getting_started_deploy.md). +To deploy Tide for your organization or repository, please see [how to get started with prow](/docs/getting-started-deploy/). ### General configuration @@ -176,4 +177,4 @@ not be publicly readable if any repos are sensitive and must be a GCS URI like ` Before a PR is merged, Tide ensures that all jobs configured as required in the `presubmits` part of the `config.yaml` file are passing against the latest base branch commit, rerunning the jobs if necessary. **No job is required to be configured** in which case it's enough if a PR meets all GitHub search criteria. -Semantic of individual fields of the `presubmits` is described in [prow/jobs.md](https://github.com/kubernetes/test-infra/tree/master/prow/jobs.md). +Semantic of individual fields of the `presubmits` is described in [ProwJobs](/docs/jobs/). diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/tide/maintainers.md b/site/content/en/docs/components/core/tide/maintainers.md similarity index 94% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/tide/maintainers.md rename to site/content/en/docs/components/core/tide/maintainers.md index 0631e1270c..dfc4e055e5 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/tide/maintainers.md +++ b/site/content/en/docs/components/core/tide/maintainers.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/tide/maintainers.md" +title: "Maintainer's Guide to Tide" +weight: 10 +description: > + --- -# Maintainer's Guide to Tide - ## Best practices 1. Don't let humans (or other bots) merge especially if tests have a long duration. Every merge invalidates currently running tests for that pool. @@ -22,4 +23,4 @@ title: "prow/cmd/tide/maintainers.md" ## Other resources -- [Configuring Tide](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide/config.md) +- [Configuring Tide](/docs/components/core/tide/config/) diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/tide/pr-authors.md b/site/content/en/docs/components/core/tide/pr-authors.md similarity index 94% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/tide/pr-authors.md rename to site/content/en/docs/components/core/tide/pr-authors.md index ce83615961..b0e5683be2 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/tide/pr-authors.md +++ b/site/content/en/docs/components/core/tide/pr-authors.md @@ -1,16 +1,17 @@ --- -title: "prow/cmd/tide/pr-authors.md" +title: "PR Author's Guide to Tide" +weight: 10 +description: > + --- -# PR Author's Guide to Tide - If you just want to figure out how to get your PR to merge this is the document for you! ## Sources of Information 1. The `tide` status context at the bottom of your PR. The status either indicates that your PR is in the merge pool or explains why it is not in the merge pool. The 'Details' link will take you to either the Tide or PR dashboard. -![Tide Status Context](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide/status-context.png) +![Tide Status Context](/docs/components/core/tide/status-context.png) 1. The PR dashboard at "``/pr" where `` is something like "https://prow.k8s.io". This dashboard shows a card for each of your PRs. Each card shows the current test results for the PR and the difference between the PR state and the merge criteria. [K8s PR dashboard](https://prow.k8s.io/pr) 1. The Tide dashboard at "``/tide". diff --git a/site/content/en/docs/components/core/tide/status-context.png b/site/content/en/docs/components/core/tide/status-context.png new file mode 100644 index 0000000000..3e874ee5ad Binary files /dev/null and b/site/content/en/docs/components/core/tide/status-context.png differ diff --git a/site/content/en/docs/components/deprecated/_index.md b/site/content/en/docs/components/deprecated/_index.md new file mode 100644 index 0000000000..d3634280dc --- /dev/null +++ b/site/content/en/docs/components/deprecated/_index.md @@ -0,0 +1,6 @@ +--- +title: "Deprecated Components" +weight: 90 +description: > + +--- diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/cm2kc/README.md b/site/content/en/docs/components/deprecated/cm2kc.md similarity index 88% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/cm2kc/README.md rename to site/content/en/docs/components/deprecated/cm2kc.md index b558af1904..d10a590326 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/cm2kc/README.md +++ b/site/content/en/docs/components/deprecated/cm2kc.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/cm2kc/README.md" +title: "cm2kc (clustermap to kubeconfig)" +weight: 10 +description: > + --- -# cm2kc (clustermap to kubeconfig) - ## Description `cm2kc` is a CLI tool used to convert a [clustermap file][clustermap docs] to a [kubeconfig file][kubeconfig docs]. @@ -23,9 +24,10 @@ The following is a list of supported options for `cm2kc`: ## Examples -#### Add a kubeconfig file in a secret: `kubeconfig` from a clustermap file in another secret: `build-cluster` for context: `my-context`. +#### Add a kubeconfig file in a secret: `kubeconfig` from a clustermap file in another secret: `build-cluster` for context: `my-context` The following command will: + 1. Get a *clustermap* formatted secret: `build-cluster` in key: `cluster` for context: `my-context`. 1. Base64 decode the secret. 1. Convert the *clustermap* data to a *kubeconfig* format. @@ -40,10 +42,10 @@ kubectl --context=my-context get secrets build-cluster -o jsonpath='{.data.clust Lastly, to begin using this in Prow, update the volume mount and replace `--build-cluster` with `--kubeconfig` in the [deployment](https://github.com/istio/test-infra/pull/1713) of each relevant Prow component (e.g. crier, deck, plank, and sinker). -#### Create a kubeconfig file at path `/path/to/kubeconfig.yaml` from a clustermap file at path `/path/to/clustermap.yaml`. +#### Create a kubeconfig file at path `/path/to/kubeconfig.yaml` from a clustermap file at path `/path/to/clustermap.yaml` Ensure the *clustermap* file exists at the specified `--input` path: - + ```yaml # /path/to/clustermap.yaml @@ -59,7 +61,7 @@ build: endpoint: https://5.6.7.8 ``` -Execute `cm2kc` specifying an `--input` path to the *clustermap* file and an `--output` path to the desired location of the generated *kubeconfig* file: +Execute `cm2kc` specifying an `--input` path to the *clustermap* file and an `--output` path to the desired location of the generated *kubeconfig* file: ```shell go run ./prow/cmd/cm2kc --input=/path/to/clustermap.yaml --output=/path/to/kubeconfig.yaml @@ -103,5 +105,5 @@ users: client-key-data: fake-build-ca-cert ``` -[clustermap docs]: https://github.com/kubernetes/test-infra/blob/1c7d9a4ae0f2ae1e0c11d8357f47163d18521b84/prow/getting_started_deploy.md#run-test-pods-in-different-clusters -[kubeconfig docs]: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/ \ No newline at end of file +[clustermap docs]: /docs/getting-started-deploy/#run-test-pods-in-different-clusters +[kubeconfig docs]: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/ diff --git a/site/content/en/docs/Legacy Snapshot/prow/plank/README.md b/site/content/en/docs/components/deprecated/plank.md similarity index 96% rename from site/content/en/docs/Legacy Snapshot/prow/plank/README.md rename to site/content/en/docs/components/deprecated/plank.md index feca7d327c..24ab27b887 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/plank/README.md +++ b/site/content/en/docs/components/deprecated/plank.md @@ -1,17 +1,20 @@ --- -title: "prow/plank/README.md" +title: "Plank" +weight: 10 +description: > + --- -# Plank - Plank is the controller that manages the job execution and lifecycle for jobs running in k8s. ### Usage + ```bash -$ go run ./prow/cmd/prow-controller-manager --help +go run ./prow/cmd/prow-controller-manager --help ``` ### Configuration + GCS and S3 are supported as the job log storage. ```yaml diff --git a/site/content/en/docs/components/external-plugins/_index.md b/site/content/en/docs/components/external-plugins/_index.md new file mode 100644 index 0000000000..12a61fc0e3 --- /dev/null +++ b/site/content/en/docs/components/external-plugins/_index.md @@ -0,0 +1,8 @@ +--- +title: "External Plugins" +weight: 80 +description: > + +--- + +This is a placeholder page. Some contents needs to be filled. diff --git a/site/content/en/docs/Legacy Snapshot/prow/external-plugins/cherrypicker/README.md b/site/content/en/docs/components/external-plugins/cherrypicker.md similarity index 89% rename from site/content/en/docs/Legacy Snapshot/prow/external-plugins/cherrypicker/README.md rename to site/content/en/docs/components/external-plugins/cherrypicker.md index a68ffa2e10..8ccfb381e0 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/external-plugins/cherrypicker/README.md +++ b/site/content/en/docs/components/external-plugins/cherrypicker.md @@ -1,9 +1,10 @@ --- -title: "prow/external-plugins/cherrypicker/README.md" +title: "cherrypicker" +weight: 10 +description: > + --- -# Cherrypicker - Cherrypicker is an external prow plugin that can also run as a standalone bot. It automates cherry-picking merged PRs into different branches. Cherrypicks are triggered from either comments or labels in GitHub PRs that need to be cherrypicked. @@ -27,6 +28,6 @@ where XXX is the name of the branch. The bot uses its own fork to push patches that need to be cherry-picked and opens PRs out of those patches. The fork is created automatically by the bot so there is -no need to set it up manually. +no need to set it up manually. Required scopes for the oauth token that need to be used are `read:org` and `repo`. diff --git a/site/content/en/docs/components/optional/_index.md b/site/content/en/docs/components/optional/_index.md new file mode 100644 index 0000000000..19228d5405 --- /dev/null +++ b/site/content/en/docs/components/optional/_index.md @@ -0,0 +1,6 @@ +--- +title: "Optional Components" +weight: 20 +description: > + +--- diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/branchprotector/README.md b/site/content/en/docs/components/optional/branchprotector.md similarity index 80% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/branchprotector/README.md rename to site/content/en/docs/components/optional/branchprotector.md index 6715fddb8f..6c402ea9e3 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/branchprotector/README.md +++ b/site/content/en/docs/components/optional/branchprotector.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/branchprotector/README.md" +title: "Branchprotector" +weight: 10 +description: > + --- -# Branch Protection Documentation - branchprotector configures [github branch protection] according to a specified policy. @@ -38,13 +39,14 @@ presubmits: ``` This config will: - * Enable protection for every branch in the `kubernetes/test-infra` + +* Enable protection for every branch in the `kubernetes/test-infra` repo. - * Require `extra-process-followed` and `fancy-job-name` [status contexts] to pass +* Require `extra-process-followed` and `fancy-job-name` [status contexts] to pass before allowing a merge - - Although it will always allow `oncall-team` to merge, even if required + * Although it will always allow `oncall-team` to merge, even if required contexts fail. - - Note that `fancy-job-name` is pulled in automatically from the + * Note that `fancy-job-name` is pulled in automatically from the `presubmits` config for the repo, if one exists. ### Updating @@ -60,7 +62,6 @@ editor and then send out a PR. When the PR merges prow pushes the updated config ### Advanced configuration - #### Fields See [`branch_protection.go`] and GitHub's [protection api] for a complete list of fields allowed @@ -104,11 +105,8 @@ branch-protection: - those ``` - - #### Scope - It is possible to define a policy at the `branch-protection`, `org`, `repo` or `branch` level. For example: @@ -142,23 +140,25 @@ branch-protection: ``` The general rule for how to compute child values is: - * If the child value is `null` or missing, inherit the parent value. - * Otherwise: - - List values (like `contexts`), create a union of the parent and child lists. - - For bool/int values (like `protect`), the child value replaces the parent value. + +* If the child value is `null` or missing, inherit the parent value. +* Otherwise: + * List values (like `contexts`), create a union of the parent and child lists. + * For bool/int values (like `protect`), the child value replaces the parent value. So in the example above: - * The `secure` branch in `unprotected-org/protected-repo` - - enables protection (set a branch level) - - requires `foo` `tested` `cla` [status contexts] + +* The `secure` branch in `unprotected-org/protected-repo` + * enables protection (set a branch level) + * requires `foo` `tested` `cla` [status contexts] (the latter two are appended by ancestors) - * All other branches in `unprotected-org/protected-repo` - - disable protection (inherited from org level) - * All branches in all other repos in `unprotected-org` - - disable protection (set at org level) - * All branches in all repos in `different-org` - - Enable protection (inherited from branch-protection level) - - Require the `cla` context to be green to merge (appended by parent) +* All other branches in `unprotected-org/protected-repo` + * disable protection (inherited from org level) +* All branches in all other repos in `unprotected-org` + * disable protection (set at org level) +* All branches in all repos in `different-org` + * Enable protection (inherited from branch-protection level) + * Require the `cla` context to be green to merge (appended by parent) ## Developer docs @@ -186,6 +186,7 @@ This will say how the binary will actually change github if you add a ### Deploy local changes to dev cluster Run things like the following: + ```sh # Build image locally make -C prow push-single-image PROW_IMAGE=prow/cmd/branchprotector REGISTRY= @@ -197,7 +198,7 @@ This will build an image with your local changes, push it to `` [branchprotector image](https://gcr.io/k8s-prow/branchprotector) is automatically built as part of prow, see -[build_test_update.md#how-to-update-the-cluster](https://github.com/kubernetes/test-infra/tree/master/prow/build_test_update.md#how-to-update-the-cluster) +["How to update the cluster"](/docs/build-test-update/#how-to-update-the-cluster) for more details. Branchprotector runs as a prow periodic job, for example diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/exporter/README.md b/site/content/en/docs/components/optional/exporter.md similarity index 92% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/exporter/README.md rename to site/content/en/docs/components/optional/exporter.md index fad1328ddb..1bbcc19979 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/exporter/README.md +++ b/site/content/en/docs/components/optional/exporter.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/exporter/README.md" +title: "Exporter" +weight: 10 +description: > + --- -# Exporter - The prow-exporter exposes metrics about prow jobs while the metrics are not directly related to a specific prow-component. @@ -22,5 +23,5 @@ it with other metrics using a [Prometheus matching operator](https://prometheus. Note that `job_name` is [`.spec.job`](https://github.com/kubernetes/test-infra/blob/98fac12af0e0b98970606dd7a5c48028a72e7f1d/prow/apis/prowjobs/v1/types.go#L117) instead of `.metadata.name` as taken in `kube_pod_labels`. -The gauge value is always `1` because we have another metric [`prowjobs`](https://github.com/kubernetes/test-infra/tree/master/prow/metrics) -for the number jobs by name. The metric here shows only the existence of such a job with the label set in the cluster. \ No newline at end of file +The gauge value is always `1` because we have another metric [`prowjobs`](/docs/metrics/) +for the number jobs by name. The metric here shows only the existence of such a job with the label set in the cluster. diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/gcsupload/README.md b/site/content/en/docs/components/optional/gcsupload.md similarity index 97% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/gcsupload/README.md rename to site/content/en/docs/components/optional/gcsupload.md index 0771cc890d..0b7955290a 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/gcsupload/README.md +++ b/site/content/en/docs/components/optional/gcsupload.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/gcsupload/README.md" +title: "gcsupload" +weight: 10 +description: > + --- -# `gcsupload` - `gcsupload` uploads artifacts to cloud storage at a path resolved from the job configuration. `gcsupload` can be configured by either passing in flags or by specifying a full set of options diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/gerrit/README.md b/site/content/en/docs/components/optional/gerrit.md similarity index 76% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/gerrit/README.md rename to site/content/en/docs/components/optional/gerrit.md index fb88635b18..1643198b61 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/gerrit/README.md +++ b/site/content/en/docs/components/optional/gerrit.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/gerrit/README.md" +title: "Gerrit" +weight: 10 +description: > + --- -# Gerrit - Gerrit is a Prow-gerrit adapter for handling CI on gerrit workflows. It can poll gerrit changes from multiple gerrit instances, and trigger presubmits on Prow upon new patchsets on Gerrit changes, and postsubmits when Gerrit changes are merged. @@ -31,6 +32,6 @@ it empty for anonymous access to gerrit API. ## Underlying infra -Also take a look at [gerrit related packages](https://github.com/kubernetes/test-infra/tree/master/prow/gerrit/README.md) for implementation details. +Also take a look at [gerrit related packages](/docs/gerrit/) for implementation details. -You might also want to deploy [Crier](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/crier) which reports job results back to gerrit. +You might also want to deploy [Crier](/docs/components/core/crier/) which reports job results back to gerrit. diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/hmac/README.md b/site/content/en/docs/components/optional/hmac.md similarity index 96% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/hmac/README.md rename to site/content/en/docs/components/optional/hmac.md index c09b6f1be6..a972db3298 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/hmac/README.md +++ b/site/content/en/docs/components/optional/hmac.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/hmac/README.md" +title: "HMAC" +weight: 10 +description: > + --- -# HMAC - `hmac` is a tool to update the HMAC token, GitHub webhooks and HMAC secret for the orgs/repos as per the `managed_webhooks` configuration changes in the Prow config file. @@ -13,8 +14,8 @@ To run this tool, you'll need: 1. A github account that has admin permission to the orgs/repos. -1. A [personal access token](https://github.com/settings/tokens) for the github account. -Note the token must be granted `admin:repo_hook` and ` admin:org_hook` scopes. +1. A [personal access token](https://github.com/settings/tokens) for the github account. +Note the token must be granted `admin:repo_hook` and `admin:org_hook` scopes. 1. Permissions to read&write the hmac secret in the Prow cluster. @@ -74,6 +75,7 @@ managed_webhooks: ### Workflow example Suppose the current `org_repo_config` in the `managed_webhooks` configuration is + ```yaml qux: token_created_after: 2017-10-02T15:00:00Z @@ -81,13 +83,14 @@ foo/bar: token_created_after: 2018-10-02T15:00:00Z foo/baz: token_created_after: 2019-10-02T15:00:00Z -``` +``` There can be 3 scenarios to modify the configuration, as explained below: #### Rotate an existing HMAC token User updates the `token_created_after` for `foo/baz` to a later time, as shown below: + ```yaml qux: token_created_after: 2017-10-02T15:00:00Z @@ -95,7 +98,7 @@ foo/bar: token_created_after: 2018-10-02T15:00:00Z foo/baz: token_created_after: 2020-03-02T15:00:00Z -``` +``` The `hmac` tool will generate a new HMAC token for the `foo/baz` repo, add the new token to the secret, and update the webhook for the repo. @@ -104,6 +107,7 @@ And after the update finishes, it will delete the old token. #### Onboard a new repo User adds a new repo `foo/bax` in the `managed_webhooks` configuration, as shown below: + ```yaml qux: token_created_after: 2017-10-02T15:00:00Z @@ -113,7 +117,7 @@ foo/baz: token_created_after: 2019-10-02T15:00:00Z foo/bax: token_created_after: 2020-03-02T15:00:00Z -``` +``` The `hmac` tool will generate an HMAC token for the `foo/bax` repo, add the token to the secret, and add the webhook for the repo. @@ -121,12 +125,13 @@ add the token to the secret, and add the webhook for the repo. #### Remove an existing repo User deletes the repo `foo/baz` from the `managed_webhooks` configuration, as shown below: + ```yaml qux: token_created_after: 2017-10-02T15:00:00Z foo/bar: token_created_after: 2018-10-02T15:00:00Z -``` +``` The `hmac` tool will delete the HMAC token for the `foo/baz` repo from the secret, and delete the corresponding webhook for this repo. diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/jenkins-operator/README.md b/site/content/en/docs/components/optional/jenkins-operator.md similarity index 93% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/jenkins-operator/README.md rename to site/content/en/docs/components/optional/jenkins-operator.md index 6ef82186a2..4881b2ba6e 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/jenkins-operator/README.md +++ b/site/content/en/docs/components/optional/jenkins-operator.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/jenkins-operator/README.md" +title: "jenkins-operator" +weight: 10 +description: > + --- -# Jenkins operator - `jenkins-operator` is a controller that enables Prow to use Jenkins as a backend for running jobs. @@ -40,6 +41,7 @@ case of a job failure. A ProwJob is provided as input to the template. ### Security Various flavors of authentication are supported: + * basic auth, using `--jenkins-user` and `--jenkins-token-file`. * OpenShift bearer token auth, using `--jenkins-bearer-token-file`. * certificate-based auth, using `--cert-file`, `--key-file`, and @@ -56,6 +58,7 @@ needs to be used on the operator's side to allow Prow to work correctly. Apart from a controller, the Jenkins operator also runs a http server to serve Jenkins logs. You can configure the Prow frontend to show Jenkins logs with the following Prow config: + ```yaml deck: external_agent_logs: @@ -75,6 +78,7 @@ side configuration is consistent when rendering jobs on the main page. ## Job configuration Below follows the Prow configuration for a Jenkins job: + ```yaml presubmits: org/repo: @@ -86,12 +90,13 @@ presubmits: trigger: "((?m)^/test( all| unit),?(\\s+|$))" ``` -You can read more about the different types of Prow jobs [elsewhere](https://github.com/kubernetes/test-infra/tree/master/prow#how-to-add-new-jobs). +You can read more about the different types of Prow jobs [elsewhere](/docs/jobs/#how-to-configure-new-jobs). What is interesting for us here is the `agent` field which needs to be set to `jenkins` in order for jobs to be dispatched to Jenkins and `name` which is the name of the job inside Jenkins. The following parameters must be added within each Jenkins job: + * `BUILD_ID` * `PROW_JOB_ID` @@ -100,16 +105,19 @@ The following parameters must be added within each Jenkins job: Sharding of Jenkins jobs is supported via Kubernetes labels and label selectors. This enables Prow to work with multiple Jenkins masters. Three places need to be configured in order to use sharding: + * `--label-selector` in the Jenkins operator. * `label_selector` in `jenkins_operators` in the Prow config. * `labels` in the job config. For example, one would set the following options: + * `--label-selector=master=jenkins-master` in a Jenkins operator. This option forces the operator to list all ProwJobs with `master=jenkins-master`. * `label_selector: master=jenkins-master` in the Prow config. + ```yaml jenkins_operators: - label_selector: master=jenkins-master @@ -151,21 +159,21 @@ adding comments about failed tests. Note that this functionality may potentially move into its own service, then the Jenkins operator will not need to contact the GitHub API. The required options are already defaulted: + * `github-token-path` set to `/etc/github/oauth`. This is the GitHub bot oauth token that is used for updating job statuses and adding comments in GitHub. * `github-endpoint` set to `https://api.github.com`. - ## Prometheus support The following Prometheus metrics are exposed by the operator: * `jenkins_requests` is the number of Jenkins requests made. - - `verb` is the type of request (`GET`, `POST`) - - `handler` is the path of the request, usually containing a + * `verb` is the type of request (`GET`, `POST`) + * `handler` is the path of the request, usually containing a job name (eg. `job/test-pull-request-unit`). - - `code` is the status code of the request (`200`, `404`, etc.). + * `code` is the status code of the request (`200`, `404`, etc.). * `jenkins_request_retries` is the number of Jenkins request retries made. * `jenkins_request_latency` is the time for a request to roundtrip @@ -173,11 +181,12 @@ between the operator and Jenkins. * `resync_period_seconds` is the time the operator takes to complete one reconciliation loop. * `prowjobs` is the number of Jenkins prowjobs in the system. - - `job_name` is the name of the job. - - `type` is the type of the prowjob: presubmit, postsubmit, periodic, batch - - `state` is the state of the prowjob: triggered, pending, success, failure, aborted, error + * `job_name` is the name of the job. + * `type` is the type of the prowjob: presubmit, postsubmit, periodic, batch + * `state` is the state of the prowjob: triggered, pending, success, failure, aborted, error If a push gateway needs to be used it can be configured in the Prow config: + ```yaml push_gateway: endpoint: http://prometheus-push-gateway diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/status-reconciler/README.md b/site/content/en/docs/components/optional/status-reconciler.md similarity index 74% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/status-reconciler/README.md rename to site/content/en/docs/components/optional/status-reconciler.md index 7621d840f8..960b0ae9c6 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/status-reconciler/README.md +++ b/site/content/en/docs/components/optional/status-reconciler.md @@ -1,17 +1,18 @@ --- -title: "prow/cmd/status-reconciler/README.md" +title: "status-reconciler" +weight: 10 +description: > + --- -# `status-reconciler` - `status-reconciler` ensures that changes to blocking presubmits in Prow configuration while PRs are in flight do not cause those PRs to get stuck. When the set of blocking presubmits changes for a repository, one of three cases occurs: - - a new blocking presubmit exists and should be triggered for every trusted pull request in flight - - an existing blocking presubmit is removed and should have its' status retired - - an existing blocking presubmit is renamed and should have its' status migrated +- a new blocking presubmit exists and should be triggered for every trusted pull request in flight +- an existing blocking presubmit is removed and should have its' status retired +- an existing blocking presubmit is renamed and should have its' status migrated The `status-reconciler` watches the job configuration for Prow and ensures that the above actions are taken as necessary. diff --git a/site/content/en/docs/Components/sub.md b/site/content/en/docs/components/optional/sub.md similarity index 99% rename from site/content/en/docs/Components/sub.md rename to site/content/en/docs/components/optional/sub.md index 64fecc743f..8ad03dbcda 100644 --- a/site/content/en/docs/Components/sub.md +++ b/site/content/en/docs/components/optional/sub.md @@ -1,5 +1,6 @@ --- title: "Sub" +weight: 150 description: > Triggers Prow jobs from Pub/Sub. --- diff --git a/site/content/en/docs/components/optional/tot/_index.md b/site/content/en/docs/components/optional/tot/_index.md new file mode 100644 index 0000000000..93c16532b5 --- /dev/null +++ b/site/content/en/docs/components/optional/tot/_index.md @@ -0,0 +1,8 @@ +--- +title: "tot" +weight: 10 +description: > + +--- + +This is a placeholder page. Some contents needs to be filled. diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/tot/fallbackcheck/README.md b/site/content/en/docs/components/optional/tot/fallbackcheck.md similarity index 89% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/tot/fallbackcheck/README.md rename to site/content/en/docs/components/optional/tot/fallbackcheck.md index b1cdc7a761..cb726a6178 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/tot/fallbackcheck/README.md +++ b/site/content/en/docs/components/optional/tot/fallbackcheck.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/tot/fallbackcheck/README.md" +title: "fallbackcheck" +weight: 10 +description: > + --- -# fallbackcheck - Ensure your GCS bucket layout is what `tot` expects to use. Useful when you want to transition from versioning your GCS buckets away from Jenkins build numbers to build numbers vended by prow. @@ -24,6 +25,7 @@ fallbackcheck -bucket GCS_BUCKET -prow-url LIVE_DECK_DEPLOYMENT ``` For example: + ```shell fallbackcheck -bucket https://gcsweb-ci.svc.ci.openshift.org/gcs/origin-ci-test/ -prow-url https://deck-ci.svc.ci.openshift.org/ ``` diff --git a/site/content/en/docs/Legacy Snapshot/prow/plugins/README.md b/site/content/en/docs/components/plugins/_index.md similarity index 77% rename from site/content/en/docs/Legacy Snapshot/prow/plugins/README.md rename to site/content/en/docs/components/plugins/_index.md index 8b397aea7b..eaeee44775 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/plugins/README.md +++ b/site/content/en/docs/components/plugins/_index.md @@ -1,10 +1,11 @@ --- -title: "prow/plugins/README.md" +title: "Plugins" +weight: 70 +description: > + --- -# Plugins - -Plugins are sub-components of [`hook`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/hook) that consume [GitHub webhooks](https://developer.github.com/webhooks/) related to their function and can be individually enabled per repo or org. +Plugins are sub-components of [`hook`](/docs/components/core/hook/) that consume [GitHub webhooks](https://developer.github.com/webhooks/) related to their function and can be individually enabled per repo or org. All plugin specific configuration is stored in [`plugins.yaml`](https://github.com/kubernetes/test-infra/tree/master/config/prow/plugins.yaml). The `Configuration` golang struct holds all the config fields organized into substructures by plugin. See its [GoDoc](https://godoc.org/k8s.io/test-infra/prow/plugins#Configuration) for up-to-date descriptions of every config option. @@ -13,13 +14,13 @@ The `Configuration` golang struct holds all the config fields organized into sub Most plugins lack README's but instead generate `PluginHelp` structs on demand that include general explanations and help information in addition to details about the current configuration. -Please see https://prow.k8s.io/plugins for a list of all plugins deployed on the Kubernetes Prow instance, what they do, and what commands they offer. -For an alternate view, please see https://prow.k8s.io/command-help to see all of the commands offered by the deployed plugins. +Please see for a list of all plugins deployed on the Kubernetes Prow instance, what they do, and what commands they offer. +For an alternate view, please see to see all of the commands offered by the deployed plugins. ## How to enable a plugin on a repo Add an entry to [plugins.yaml](https://github.com/kubernetes/test-infra/tree/master/config/prow/plugins.yaml). If you misspell the name then a -unit test will fail. If you have [update-config](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/updateconfig) plugin +unit test will fail. If you have [updateconfig](/docs/components/plugins/updateconfig/) plugin deployed then the config will be automatically updated once the PR is merged, else you will need to run `make update-plugins`. This does not require redeploying the binaries, and will take effect within a minute. @@ -28,12 +29,14 @@ redeploying the binaries, and will take effect within a minute. External plugins offer an alternative to compiling a plugin into the `hook` binary. Any web endpoint that can properly handle GitHub webhooks can be configured as an external plugin that `hook` will forward webhooks to. External plugin endpoints are specified per org or org/repo in [`plugins.yaml`](https://github.com/kubernetes/test-infra/tree/master/config/prow/plugins.yaml) under the `external_plugins` field. Specific event types may be optionally specified to filter which events are forwarded to the endpoint. External plugins are well suited for: + - Slow operations that would impact the performance of other plugins if run as part of `hook`. - Components that need to be triggered or notified of events beside GitHub webhooks. - Isolating a more or less privileged plugin or a plugin that executes PR code. - Integrating existing GitHub services with Prow. Examples of external plugins can be found in the [`prow/external-plugins`](https://github.com/kubernetes/test-infra/tree/master/prow/external-plugins) directory. The following is an example external plugin configuration that would live in [`plugins.yaml`](https://github.com/kubernetes/test-infra/tree/master/config/prow/plugins.yaml). + ```yaml external_plugins: org-foo/repo-bar: @@ -54,4 +57,4 @@ external_plugins: ## How to test a plugin -See [`build_test_update.md`](https://github.com/kubernetes/test-infra/tree/master/prow/build_test_update.md#How-to-test-a-plugin). +See ["Building, Testing, and Updating Prow"](/docs/build-test-update/#how-to-test-a-plugin). diff --git a/site/content/en/docs/components/plugins/approve/_index.md b/site/content/en/docs/components/plugins/approve/_index.md new file mode 100644 index 0000000000..a690bb3a7b --- /dev/null +++ b/site/content/en/docs/components/plugins/approve/_index.md @@ -0,0 +1,8 @@ +--- +title: "approve" +weight: 10 +description: > + +--- + +This is a placeholder page. Some contents needs to be filled. diff --git a/site/content/en/docs/Legacy Snapshot/prow/plugins/approve/approvers/README.md b/site/content/en/docs/components/plugins/approve/approvers/_index.md similarity index 93% rename from site/content/en/docs/Legacy Snapshot/prow/plugins/approve/approvers/README.md rename to site/content/en/docs/components/plugins/approve/approvers/_index.md index 76530f565a..41a671debc 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/plugins/approve/approvers/README.md +++ b/site/content/en/docs/components/plugins/approve/approvers/_index.md @@ -1,9 +1,10 @@ --- -title: "prow/plugins/approve/approvers/README.md" +title: "Reviewers and Approvers" +weight: 10 +description: > + --- -# Reviewers and Approvers - ## Questions this Doc Seeks To Answer 1. What are reviewers, approvers, and the OWNERS files? @@ -82,7 +83,7 @@ The exact algorithm for selecting approvers is somewhat complex; it is an set co ## Example -![Directory Structure](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/approve/approvers/images/directory_structure.png) +![Directory Structure](./directory_structure.png) Suppose files in directories E and G are changed in a PR created by PRAuthor. Any combination of approver(s) listed below can approve the PR in order to get it merged: @@ -198,9 +199,9 @@ K8s-bot updates comment: You can indicate your approval by writing `/approve` in a comment You can cancel your approval by writing `/approve cancel` in a comment -The PR is now unblocked from merging. If [Tide](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide) is configured, the K8s-bot merges the PR, because it has both the **lgtm** and **approved**. It K8s-bot still needs to wait its turn in submit queue and pass tests. +The PR is now unblocked from merging. If [Tide](/docs/components/core/tide/) is configured, the K8s-bot merges the PR, because it has both the **lgtm** and **approved**. It K8s-bot still needs to wait its turn in submit queue and pass tests. -![Bot Notification for Approval Mechanism](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/approve/approvers/images/bot_notification_for_approval_selection_mechanism.png) +![Bot Notification for Approval Mechanism](./bot_notification_for_approval_selection_mechanism.png) ## Configuration options @@ -236,4 +237,3 @@ Approve: [prow/plugins/approve/approve.go](https://git.k8s.io/test-infra/prow/plugins/approve/approve.go) [prow/plugins/approve/approvers/owners.go](https://git.k8s.io/test-infra/prow/plugins/approve/approvers/owners.go) - diff --git a/site/content/en/docs/components/plugins/approve/approvers/bot_notification_for_approval_selection_mechanism.png b/site/content/en/docs/components/plugins/approve/approvers/bot_notification_for_approval_selection_mechanism.png new file mode 100644 index 0000000000..354c195353 Binary files /dev/null and b/site/content/en/docs/components/plugins/approve/approvers/bot_notification_for_approval_selection_mechanism.png differ diff --git a/site/content/en/docs/components/plugins/approve/approvers/directory_structure.png b/site/content/en/docs/components/plugins/approve/approvers/directory_structure.png new file mode 100644 index 0000000000..3543d48829 Binary files /dev/null and b/site/content/en/docs/components/plugins/approve/approvers/directory_structure.png differ diff --git a/site/content/en/docs/Legacy Snapshot/prow/plugins/branchcleaner/README.md b/site/content/en/docs/components/plugins/branchcleaner.md similarity index 82% rename from site/content/en/docs/Legacy Snapshot/prow/plugins/branchcleaner/README.md rename to site/content/en/docs/components/plugins/branchcleaner.md index 606b81f295..6ca028490d 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/plugins/branchcleaner/README.md +++ b/site/content/en/docs/components/plugins/branchcleaner.md @@ -1,9 +1,10 @@ --- -title: "prow/plugins/branchcleaner/README.md" +title: "branchcleaner" +weight: 10 +description: > + --- -# Branchcleaner - The `branchcleaner` plugin automatically deletes source branches for merged PRs between two branches on the same repository. This is helpful to keep repos that don't allow forking clean. @@ -11,7 +12,7 @@ on the same repository. This is helpful to keep repos that don't allow forking c Enable the `branchcleaner` in the desired repos via the `plugins.yaml`: -``` +```yaml plugins: org/repo: - branchcleaner diff --git a/site/content/en/docs/components/plugins/lgtm.md b/site/content/en/docs/components/plugins/lgtm.md new file mode 100644 index 0000000000..320b5e16bb --- /dev/null +++ b/site/content/en/docs/components/plugins/lgtm.md @@ -0,0 +1,8 @@ +--- +title: "lgtm" +weight: 10 +description: > + +--- + +See [the documentation in the approve plugin for details on the LGTM flow](/docs/components/plugins/approve/approvers/#lgtm-label). diff --git a/site/content/en/docs/Legacy Snapshot/prow/plugins/updateconfig/README.md b/site/content/en/docs/components/plugins/updateconfig.md similarity index 89% rename from site/content/en/docs/Legacy Snapshot/prow/plugins/updateconfig/README.md rename to site/content/en/docs/components/plugins/updateconfig.md index 8c9ba06954..a4e6926cf2 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/plugins/updateconfig/README.md +++ b/site/content/en/docs/components/plugins/updateconfig.md @@ -1,9 +1,10 @@ --- -title: "prow/plugins/updateconfig/README.md" +title: "updateconfig" +weight: 10 +description: > + --- - - `updateconfig` allows prow to update configmaps when files in a repo change. `updateconfig` also supports glob match, or multi-key updates. @@ -11,6 +12,7 @@ title: "prow/plugins/updateconfig/README.md" ## Usage Update your `plugins.yaml` file to something along the following lines: + ```yaml plugins: my-github/repo: @@ -27,7 +29,7 @@ config_updater: name: thing2-config # Specify the clusters and namespaces that the configmap targets # which requires that the --kubeconfig arg is enabled for Hook - # https://github.com/kubernetes/test-infra/blob/master/prow/getting_started_deploy.md#run-test-pods-in-different-clusters + # https://docs.prow.k8s.io/docs/getting-started-deploy/#run-test-pods-in-different-clusters # if not set or empty, it uses the cluster where prow components are running # and the specified namespace(s) clusters: diff --git a/site/content/en/docs/Legacy Snapshot/prow/pod-utilities.md b/site/content/en/docs/components/pod-utilities/_index.md similarity index 86% rename from site/content/en/docs/Legacy Snapshot/prow/pod-utilities.md rename to site/content/en/docs/components/pod-utilities/_index.md index e76b354b1c..951c9b204b 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/pod-utilities.md +++ b/site/content/en/docs/components/pod-utilities/_index.md @@ -1,9 +1,10 @@ --- -title: "prow/pod-utilities.md" +title: "Pod Utilities" +weight: 40 +description: > + --- -# Pod Utilities - Pod utilities are small, focused Go programs used by `plank` to decorate user-provided `PodSpec`s in order to increase the ease of integration for new jobs into the entire CI infrastructure. The utilities today wrap the execution of the test code to ensure that the tests run against correct @@ -14,12 +15,12 @@ These utilities are integrated into a test run by adding `InitContainer`s and si to the user-provided `PodSpec`, as well as by overwriting the `Container` entrypoint for the test `Container` provided by the user. The following utilities exist today: - - [`clonerefs`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/clonerefs/README.md): clones source code under test - - [`initupload`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/initupload/README.md): records the beginning of a test in cloud storage +- [`clonerefs`](/docs/components/pod-utilities/clonerefs/): clones source code under test +- [`initupload`](/docs/components/pod-utilities/initupload/): records the beginning of a test in cloud storage and reports the status of the clone operations - - [`entrypoint`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/entrypoint/README.md): is injected into the test `Container`, wraps the +- [`entrypoint`](/docs/components/pod-utilities/entrypoint/): is injected into the test `Container`, wraps the test code to capture logs and exit status - - [`sidecar`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/sidecar/README.md): runs alongside the test `Container`, uploads status, logs +- [`sidecar`](/docs/components/pod-utilities/sidecar/): runs alongside the test `Container`, uploads status, logs and test artifacts to cloud storage once the test is finished ### Why use Pod Utilities? @@ -40,6 +41,7 @@ and less coupled to Prow. ### What the test container can expect Example test container script: + ```bash pwd # my repo root ls path/to/file/in/my/repo.txt # access repo file @@ -49,12 +51,13 @@ echo success > ${ARTIFACTS}/results.txt # result info that will be uploaded to G ``` More specifically, a ProwJob using the Pod Utilities can expect the following: + - **Source Code** - Jobs can expect to begin execution with their working directory set as the root of the checked out repo. The commit that is checked out depends on the type of job: - - `presubmit` jobs will have the relevant PR checked out and merged with the base branch. - - `postsubmit` jobs will have the upstream commit that triggered the job checked out. - - `periodic` jobs will have the working directory set to the root of the repo specified by the first ref in `extra_refs` (if specified). + - `presubmit` jobs will have the relevant PR checked out and merged with the base branch. + - `postsubmit` jobs will have the upstream commit that triggered the job checked out. + - `periodic` jobs will have the working directory set to the root of the repo specified by the first ref in `extra_refs` (if specified). See the `extra_refs` field if you need to clone more than one repo. - **Metadata and Logs** - Jobs can expect metadata about the job to be uploaded before the job starts, and additional metadata and logs to be uploaded when the @@ -66,10 +69,11 @@ dumped for automatic upload to GCS upon job completion. ### How to configure In order to use the pod utilities, you will need to configure plank with some settings first. -See plank's [README](https://github.com/kubernetes/test-infra/tree/master/prow/plank) for reference. +See plank's [README](/docs/components/deprecated/plank/) for reference. ProwJobs may request Pod Utility decoration by setting `decorate: true` in their config. Example ProwJob configuration: + ```yaml - name: pull-job @@ -90,10 +94,11 @@ the Dockerfile's ENTRYPOINT directive. Note that the `command` field is a string array not just a string. It should point to the test binary location in the container. Additional fields may be required for some use cases: + - Private repos need to do two things: - - Add an ssh secret that gives the bot access to the repo to the build cluster - and specify the secret name in the `ssh_key_secrets` field of the job decoration config. - - Set the `clone_uri` field of the job spec to `git@github.com:{{.Org}}/{{.Repo}}.git`. + - Add an ssh secret that gives the bot access to the repo to the build cluster + and specify the secret name in the `ssh_key_secrets` field of the job decoration config. + - Set the `clone_uri` field of the job spec to `git@github.com:{{.Org}}/{{.Repo}}.git`. - Repos requiring a non-standard clone path can use the `path_alias` field to clone the repo to different go import path than the default of `/home/prow/go/src/github.com/{{.Org}}/{{.Repo}}/` (e.g. `path_alias: k8s.io/test-infra` -> `/home/prow/go/src/k8s.io/test-infra`). - Jobs that require additional repos to be checked out can arrange for that with diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/clonerefs/README.md b/site/content/en/docs/components/pod-utilities/clonerefs.md similarity index 90% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/clonerefs/README.md rename to site/content/en/docs/components/pod-utilities/clonerefs.md index b804ac9e2c..dbf01567ad 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/clonerefs/README.md +++ b/site/content/en/docs/components/pod-utilities/clonerefs.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/clonerefs/README.md" +title: "clonerefs" +weight: 10 +description: > + --- -# `clonerefs` - `clonerefs` clones code under test at the specified locations. Regardless of the success or failure of clone operations, this utility will have an exit code of `0` and will record the clone operation status to the specified log file. Clone records have the form: @@ -39,7 +40,7 @@ status to the specified log file. Clone records have the form: Note: the utility _will_ exit with a non-zero status if a fatal error is detected and no clone operations can even begin to run. -This utility is intended to be used with [`initupload`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/initupload/README.md), which will +This utility is intended to be used with [`initupload`](/docs/components/pod-utilities/initupload/), which will decode the JSON output by `clonerefs` and can format it for human consumption. `clonerefs` can be configured by either passing in flags or by specifying a full set of options diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/entrypoint/README.md b/site/content/en/docs/components/pod-utilities/entrypoint.md similarity index 80% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/entrypoint/README.md rename to site/content/en/docs/components/pod-utilities/entrypoint.md index 4671444c67..8357583c40 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/entrypoint/README.md +++ b/site/content/en/docs/components/pod-utilities/entrypoint.md @@ -1,14 +1,15 @@ --- -title: "prow/cmd/entrypoint/README.md" +title: "entrypoint" +weight: 10 +description: > + --- -# `entrypoint` - `entrypoint` wraps a process and records its output to `stdout` and `stderr` as well as its exit code, recording both to disk. The utility will exit with a non-zero exit code if the wrapped process fails or if the utility has a fatal error. -This utility is intended to be used with [`sidecar`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/sidecar/README.md), which will +This utility is intended to be used with [`sidecar`](/docs/components/pod-utilities/sidecar/), which will watch the files written by this utility and report on the status of the wrapped process. `entrypoint` can be configured by either passing in flags or by specifying a full set of options diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/initupload/README.md b/site/content/en/docs/components/pod-utilities/initupload.md similarity index 85% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/initupload/README.md rename to site/content/en/docs/components/pod-utilities/initupload.md index b2372d69a5..1d80ec7444 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/initupload/README.md +++ b/site/content/en/docs/components/pod-utilities/initupload.md @@ -1,9 +1,10 @@ --- -title: "prow/cmd/initupload/README.md" +title: "initupload" +weight: 10 +description: > + --- -# `initupload` - `initupload` reads clone records placed by `clonerefs` in order to determine job status. The status and logs from the clone operations are uploaded to cloud storage at a path resolved from the job configuration. This utility will exit with a non-zero exit code if the clone records indicate that @@ -11,7 +12,7 @@ any clone operations failed, as well as if any fatal errors are encountered in t `initupload` can be configured by either passing in flags or by specifying a full set of options as JSON in the `$INITUPLOAD_OPTIONS` environment variable, which has the same form as that for -`gcsupload`, plus the `"log"` field. See [that documentation](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/gcsupload/README.md) for +`gcsupload`, plus the `"log"` field. See [that documentation](/docs/components/optional/gcsupload/) for an explanation. ```json diff --git a/site/content/en/docs/Legacy Snapshot/prow/cmd/sidecar/README.md b/site/content/en/docs/components/pod-utilities/sidecar.md similarity index 81% rename from site/content/en/docs/Legacy Snapshot/prow/cmd/sidecar/README.md rename to site/content/en/docs/components/pod-utilities/sidecar.md index 887a1ca7ad..490c4242c1 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/cmd/sidecar/README.md +++ b/site/content/en/docs/components/pod-utilities/sidecar.md @@ -1,21 +1,22 @@ --- -title: "prow/cmd/sidecar/README.md" +title: "sidecar" +weight: 10 +description: > + --- -# `sidecar` - `sidecar` watches disk for files containing a the `std{out,err}` output from a process as well as its exit code; when the exit code has been written, this utility uploads a status object, the logs from the process and any other specified artifacts to cloud storage. The utility will exit with the exit code of the wrapped process or otherwise non-zero if the utility has a fatal error. -This utility is intended to be used with [`entrypoint`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/entrypoint/README.md), which will +This utility is intended to be used with [`entrypoint`](/docs/components/pod-utilities/entrypoint/), which will write the files watched by this utility. `sidecar` can be configured by either passing in flags or by specifying a full set of options as JSON in the `$SIDECAR_OPTIONS` environment variable, which has the same form as that for `gcsupload`, plus the `"process_log"` and `"marker_file"` fields. See -[that documentation](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/gcsupload/README.md) for an explanation. +[that documentation](/docs/components/optional/gcsupload/) for an explanation. ```json { diff --git a/site/content/en/docs/components/undocumented/_index.md b/site/content/en/docs/components/undocumented/_index.md new file mode 100644 index 0000000000..208cb91f7e --- /dev/null +++ b/site/content/en/docs/components/undocumented/_index.md @@ -0,0 +1,6 @@ +--- +title: "Undocumented Components" +weight: 100 +description: > + +--- diff --git a/site/content/en/docs/components/undocumented/admission.md b/site/content/en/docs/components/undocumented/admission.md new file mode 100644 index 0000000000..9d18105e59 --- /dev/null +++ b/site/content/en/docs/components/undocumented/admission.md @@ -0,0 +1,8 @@ +--- +title: "admission" +weight: 10 +description: > + +--- + +This is a placeholder page. Some contents needs to be filled. diff --git a/site/content/en/docs/components/undocumented/grandmatriarch.md b/site/content/en/docs/components/undocumented/grandmatriarch.md new file mode 100644 index 0000000000..500f1cc810 --- /dev/null +++ b/site/content/en/docs/components/undocumented/grandmatriarch.md @@ -0,0 +1,8 @@ +--- +title: "grandmatriarch" +weight: 10 +description: > + +--- + +This is a placeholder page. Some contents needs to be filled. diff --git a/site/content/en/docs/components/undocumented/pipeline.md b/site/content/en/docs/components/undocumented/pipeline.md new file mode 100644 index 0000000000..ac8fd1532e --- /dev/null +++ b/site/content/en/docs/components/undocumented/pipeline.md @@ -0,0 +1,8 @@ +--- +title: "pipeline" +weight: 10 +description: > + +--- + +This is a placeholder page. Some contents needs to be filled. diff --git a/site/content/en/docs/components/undocumented/tackle.md b/site/content/en/docs/components/undocumented/tackle.md new file mode 100644 index 0000000000..8151f2ec57 --- /dev/null +++ b/site/content/en/docs/components/undocumented/tackle.md @@ -0,0 +1,8 @@ +--- +title: "tackle" +weight: 10 +description: > + +--- + +This is a placeholder page. Some contents needs to be filled. diff --git a/site/content/en/docs/Legacy Snapshot/prow/config/README.md b/site/content/en/docs/config.md similarity index 52% rename from site/content/en/docs/Legacy Snapshot/prow/config/README.md rename to site/content/en/docs/config.md index 4a49f16cd9..aa2b4f011a 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/config/README.md +++ b/site/content/en/docs/config.md @@ -1,10 +1,11 @@ --- -title: "prow/config/README.md" +title: "Prow Configuration" +weight: 130 +description: > + --- -# Prow Configuration - -Core Prow component configuration is managed by the `config` package and stored in the [`Config` struct](https://godoc.org/k8s.io/test-infra/prow/config#Config). If a configuration guide is available for a component it can be found in the [`/prow/cmd`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd) directory. See [`jobs.md`](https://github.com/kubernetes/test-infra/tree/master/prow/jobs.md) for a guide to configuring ProwJobs. -Configuration for plugins is handled and stored separately. See the [`plugins`](https://github.com/kubernetes/test-infra/tree/master/prow/plugins) package for details. +Core Prow component configuration is managed by the `config` package and stored in the [`Config` struct](https://godoc.org/k8s.io/test-infra/prow/config#Config). If a configuration guide is available for a component it can be found in the ["Components"](/docs/components/) directory. See [`jobs.md`](/docs/jobs/) for a guide to configuring ProwJobs. +Configuration for plugins is handled and stored separately. See the [`plugins`](/docs/components/plugins/) package for details. You can find a sample config with all possible options and a documentation of them [here.](https://github.com/kubernetes/test-infra/tree/master/prow/config/prow-config-documented.yaml) diff --git a/site/content/en/docs/Contribution Guidelines/_index.md b/site/content/en/docs/contribution-guidelines/_index.md similarity index 97% rename from site/content/en/docs/Contribution Guidelines/_index.md rename to site/content/en/docs/contribution-guidelines/_index.md index 56c35f8a17..d956d3efaf 100644 --- a/site/content/en/docs/Contribution Guidelines/_index.md +++ b/site/content/en/docs/contribution-guidelines/_index.md @@ -1,6 +1,6 @@ --- title: "Contribution Guidelines" -weight: 10 +weight: 50 description: > How to contribute to the docs --- @@ -39,5 +39,5 @@ engine. You write the pages in Markdown (or HTML if you want), and Hugo wraps th [k/t-i/prow]:https://github.com/kubernetes/test-infra/tree/master/prow [k-s/prow]:https://github.com/kubernetes-sigs/prow -[l-s]:{{< relref "../Legacy Snapshot" >}} +[l-s]:{{< relref "../legacy-snapshot" >}} [orig-freeze]:https://github.com/kubernetes/test-infra/pull/26458 diff --git a/site/content/en/docs/Legacy Snapshot/prow/gerrit/README.md b/site/content/en/docs/gerrit.md similarity index 62% rename from site/content/en/docs/Legacy Snapshot/prow/gerrit/README.md rename to site/content/en/docs/gerrit.md index bb520cbaaf..54ddb971b7 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/gerrit/README.md +++ b/site/content/en/docs/gerrit.md @@ -1,16 +1,16 @@ --- -title: "prow/gerrit/README.md" +title: "Gerrit" +weight: 150 +description: > + --- -# Gerrit - -[Gerrit] is a free, web-based team code collaboration tool. +[Gerrit](https://www.gerritcodereview.com/) is a free, web-based team code collaboration tool. ## Related Deployments -[Adapter](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/gerrit) - -[Reporter](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/crier) +- Prow-gerrit adapter ([doc](/docs/components/optional/gerrit/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/gerrit)) +- Crier (the reporter) ([doc](/docs/components/core/crier/), [code](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/crier)) ## Related packages @@ -25,27 +25,26 @@ start the client, like: ```go projects := map[string][]string{ - "foo.googlesource.com": { - "project-bar", - "project-baz", - }, + "foo.googlesource.com": { + "project-bar", + "project-baz", + }, } c, err := gerrit.NewClient(projects) if err != nil { - // handle error + // handle error } c.Start(cookiefilePath) ``` The client will try to refetch token from the path every 10 minutes. -You should also utilize [grandmatriarch] to generate a token from a +You should also utilize [`grandmatriarch`](/docs/components/undocumented/grandmatriarch/) to generate a token from a passed-in service account credential. If you need extra features, feel free to introduce new gerrit API functions to the client package. - #### Adapter The adapter package implements a controller that is periodically polling gerrit, and triggering @@ -53,10 +52,12 @@ presubmit and postsubmit jobs based on your prow config. #### Gerrit Labels -Prow adds the following [Labels] to Gerrit Presubmits that can be accessed in the container by leveraging the [Downward Api]. +Prow adds the following [Labels](https://github.com/kubernetes/test-infra/tree/master/prow/gerrit/client/client.go) to Gerrit Presubmits that can be accessed in the container by leveraging the [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/). + - "prow.k8s.io/gerrit-revision": SHA of current patchset from a gerrit change - "prow.k8s.io/gerrit-patchset": Numeric ID of the current patchset - "prow.k8s.io/gerrit-report-label": Gerrit label prow will cast vote on, fallback to CodeReview label if unset + ```yaml - name: PATHCSET_NUMBER valueFrom: @@ -68,11 +69,3 @@ Prow adds the following [Labels] to Gerrit Presubmits that can be accessed in th The gerrit adapter currently does not support [gerrit hooks](https://gerrit-review.googlesource.com/Documentation/config-hooks.html), If you need them, please send us a PR to support them :-) - - -[Gerrit]: https://www.gerritcodereview.com/ -[Prow]: https://github.com/kubernetes/test-infra/tree/master/prow/README.md -[grandmatriarch]: https://github.com/kubernetes/test-infra/tree/master/prow/cmd/grandmatriarch -[crier]: https://github.com/kubernetes/test-infra/tree/master/prow/crier -[Labels]: https://github.com/kubernetes/test-infra/tree/master/prow/gerrit/client/client.go -[Downward Api]: https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/ diff --git a/site/content/en/docs/Legacy Snapshot/prow/getting_started_deploy.md b/site/content/en/docs/getting-started-deploy.md similarity index 95% rename from site/content/en/docs/Legacy Snapshot/prow/getting_started_deploy.md rename to site/content/en/docs/getting-started-deploy.md index 0efe272049..38decc27ce 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/getting_started_deploy.md +++ b/site/content/en/docs/getting-started-deploy.md @@ -1,9 +1,10 @@ --- -title: "prow/getting_started_deploy.md" +title: "Deploying Prow" +weight: 80 +description: > + --- -# Deploying Prow - This document will walk you through deploying your own Prow instance to a new Kubernetes cluster. If you encounter difficulties, please open an issue so that we can make this process easier. Prow runs in any kubernetes cluster. Our `tackle` utility helps deploy it correctly, or you can perform each of the steps manually. @@ -347,7 +348,7 @@ In order to configure the Azure storage, follow the following steps: > If you want to persist logs and output in GCS, you need to follow the steps below. -When configuring Prow jobs to use the [Pod utilities](https://github.com/kubernetes/test-infra/tree/master/prow/pod-utilities.md) +When configuring Prow jobs to use the [Pod utilities](/docs/components/pod-utilities/) with `decorate: true`, job metadata, logs, and artifacts will be uploaded to a GCS bucket in order to persist results from tests and allow for the job overview page to load those results at a later point. In order to run @@ -396,7 +397,7 @@ v20191108-08fbf64ac ``` Then, we can use that tag to retrieve the corresponding utility images in `default_decoration_config_entries[]` in `config.yaml`: -For more information on how the pod utility images for prow are versioned see [generic-autobumper](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/generic-autobumper/README.md) and the [autobump config used for prow.k8s.io](https://github.com/kubernetes/test-infra/tree/master/config/prow/autobump-config/prow-component-autobump-config.yaml) +For more information on how the pod utility images for prow are versioned see [generic-autobumper](/docs/components/cli-tools/generic-autobumper/) and the [autobump config used for prow.k8s.io](https://github.com/kubernetes/test-infra/tree/master/config/prow/autobump-config/prow-component-autobump-config.yaml) ```yaml plank: @@ -416,7 +417,7 @@ plank: ### Adding more jobs There are two ways to configure jobs: -* Using the [inrepoconfig](https://github.com/kubernetes/test-infra/tree/master/prow/inrepoconfig.md) feature to configure jobs inside the repo under test +* Using the [inrepoconfig](/docs/inrepoconfig/) feature to configure jobs inside the repo under test * Using the static config by editing the `config` configmap, some samples below: Add the following to `config.yaml`: @@ -482,7 +483,7 @@ the change within a few minutes. When you push or merge a new change to the git repo, the postsubmit job will run. -For more information on the job environment, see [`jobs.md`](https://github.com/kubernetes/test-infra/tree/master/prow/jobs.md) +For more information on the job environment, see [`jobs.md`](/docs/jobs/) ### Run test pods in different clusters @@ -651,14 +652,14 @@ a separate namespace. ## Further reading -* [Developing for Prow](https://github.com/kubernetes/test-infra/tree/master/prow/getting_started_develop.md) -* [Getting more out of Prow](https://github.com/kubernetes/test-infra/tree/master/prow/more_prow.md) +* [Developing for Prow](/docs/getting-started-develop/) +* [Getting more out of Prow](/docs/more-prow/) [1]: https://github.com/settings/tokens -[2]: https://github.com/kubernetes/test-infra/tree/master/prow/jobs.md#How-to-configure-new-jobs +[2]: /docs/jobs/#how-to-configure-new-jobs [3]: https://github.com/jetstack/cert-manager [4]: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls [5]: https://github.com/kubernetes/test-infra/tree/master/gencred/ -[6]: https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide/README.md -[7]: https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide/config.md -[8]: https://github.com/kubernetes/test-infra/blob/master/prow/scaling.md#working-around-githubs-limited-acls +[6]: /docs/components/core/tide/ +[7]: /docs/components/core/tide/config/ +[8]: /docs/scaling/#working-around-githubs-limited-acls diff --git a/site/content/en/docs/Legacy Snapshot/prow/getting_started_develop.md b/site/content/en/docs/getting-started-develop.md similarity index 61% rename from site/content/en/docs/Legacy Snapshot/prow/getting_started_develop.md rename to site/content/en/docs/getting-started-develop.md index 3400121bd8..b8265388b8 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/getting_started_develop.md +++ b/site/content/en/docs/getting-started-develop.md @@ -1,13 +1,14 @@ --- -title: "prow/getting_started_develop.md" +title: "Developing and Contributing to Prow" +weight: 90 +description: > + --- -# Developing and Contributing to Prow - ## Contributing Please consider upstreaming any changes or additions you make! Contributions in any form (issues, pull requests, even constructive comments in discussions) are more than welcome! -You can develop in-tree for more help and review, or out-of-tree if you need to for whatever reason. If you upstream a new feature or a change that impacts the default behavior of Prow, consider adding an [announcement](https://github.com/kubernetes/test-infra/tree/master/prow/ANNOUNCEMENTS.md) about it and dropping an email at the [sig-testing](https://groups.google.com/forum/#!forum/kubernetes-sig-testing) mailing list. +You can develop in-tree for more help and review, or out-of-tree if you need to for whatever reason. If you upstream a new feature or a change that impacts the default behavior of Prow, consider adding an [announcement](/docs/announcements/) about it and dropping an email at the [sig-testing](https://groups.google.com/forum/#!forum/kubernetes-sig-testing) mailing list. **New Contributors** should search for issues in kubernetes/test-infra with the `help-wanted` and/or `good first issue` labels. [(Query link)](https://github.com/kubernetes/test-infra/issues?utf8=%E2%9C%93&q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22). Before starting work please ensure that the issue is still active and then provide a short design overview of your planned solution. Also reach out on the Kubernetes slack in the `sig-testing` channel. @@ -18,7 +19,7 @@ There are a number of ways that you can write code for Prow or integrate existin #### Plugins -[Prow plugins](https://github.com/kubernetes/test-infra/tree/master/prow/plugins) are sub-components of the [`hook`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/hook) binary that register event handlers for various types of GitHub events. +[Prow plugins](/docs/components/plugins/) are sub-components of the [`hook`](/docs/components/core/hook/) binary that register event handlers for various types of GitHub events. Plugin event handlers are provided a [`PluginClient`](https://godoc.org/k8s.io/test-infra/prow/plugins#PluginClient) that provides access to a suite of clients and agents for configuration, ProwJobs, GitHub, git, OWNERS file, Slack, and more. ##### How to add new plugins @@ -31,12 +32,12 @@ unit test will fail when you try to add it to `plugins.yaml`. Don't add a brand new plugin to the main `kubernetes/kubernetes` repo right away, start with somewhere smaller and make sure it is well-behaved. -The [`lgtm` plugin](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/lgtm) is a good place to start if you're looking for an example +The [`lgtm` plugin](/docs/components/plugins/lgtm/) is a good place to start if you're looking for an example plugin to mimic. ##### External plugins -For even more flexibility, *anything* that receives GitHub webhooks can be configured to be forwarded webhooks as an [external plugin](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/README.md#external-plugins). This allows in-cluster or out of cluster plugins and forwarding to other bots or infrastructure. +For even more flexibility, *anything* that receives GitHub webhooks can be configured to be forwarded webhooks as an [external plugin](/docs/components/plugins/#external-plugins). This allows in-cluster or out of cluster plugins and forwarding to other bots or infrastructure. #### Cluster Deployments @@ -46,18 +47,16 @@ The informer framework can also be used to react to job completion or update in #### Artifact Viewers -[Spyglass](https://github.com/kubernetes/test-infra/tree/master/prow/spyglass) artifact viewers allow for custom display of ProwJob artifacts that match a certain file regexp. Existing viewers display logs, metadata, and structured junit results. +[Spyglass](/docs/spyglass/) artifact viewers allow for custom display of ProwJob artifacts that match a certain file regexp. Existing viewers display logs, metadata, and structured junit results. #### ProwJobs -[ProwJobs](https://github.com/kubernetes/test-infra/tree/master/prow/jobs.md) themselves are often a sufficient integration point if you just need to execute a task on a schedule or in reaction to code changes. +[ProwJobs](/docs/jobs/) themselves are often a sufficient integration point if you just need to execute a task on a schedule or in reaction to code changes. #### Exposed Data If you just need some data from Prow you may be able to get it from the JSON exposed by Prow's front end `deck`, or from Prometheus metrics. +## Building, Testing, and Deploying -## Building, Testing, and Deploying: - -You can build, test, and deploy Prow’s binaries, container images, and cluster resources. See [`getting_started_deploy.md`](https://github.com/kubernetes/test-infra/tree/master/prow/getting_started_deploy.md) for initially deploying Prow and [`build_test_update.md`](https://github.com/kubernetes/test-infra/tree/master/prow/build_test_update.md) for iterating on an existing deployment. - +You can build, test, and deploy Prow’s binaries, container images, and cluster resources. See ["Deploying Prow"](/docs/getting-started-deploy/) for initially deploying Prow and ["Building, Testing, and Updating Prow"](/docs/build-test-update/) for iterating on an existing deployment. diff --git a/site/content/en/docs/Legacy Snapshot/prow/github/README.md b/site/content/en/docs/github.md similarity index 96% rename from site/content/en/docs/Legacy Snapshot/prow/github/README.md rename to site/content/en/docs/github.md index 4f1f8af497..fa140a670d 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/github/README.md +++ b/site/content/en/docs/github.md @@ -1,9 +1,10 @@ --- -title: "prow/github/README.md" +title: "GitHub API Library" +weight: 105 +description: > + --- -# GitHub API Library - This GitHub API library is used by multiple parts of Prow. It uses both [v3](https://developer.github.com/v3/) and [v4](https://developer.github.com/v4/) of GitHub's API. It is subject to change as needed without notice, but you can reuse and extend it diff --git a/site/content/en/docs/Legacy Snapshot/prow/inrepoconfig.md b/site/content/en/docs/inrepoconfig.md similarity index 96% rename from site/content/en/docs/Legacy Snapshot/prow/inrepoconfig.md rename to site/content/en/docs/inrepoconfig.md index 2c549567da..9ad75a06f1 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/inrepoconfig.md +++ b/site/content/en/docs/inrepoconfig.md @@ -1,9 +1,10 @@ --- -title: "prow/inrepoconfig.md" +title: "Inrepoconfig" +weight: 110 +description: > + --- -# Inrepoconfig - Inrepoconfig is a Prow feature that allows versioning Presubmit and Postsubmit jobs in the same repository that also holds the code (with a `.prow` directory or `.prow.yaml` file, akin to a `.travis.yaml` file). So instead of having all @@ -121,4 +122,4 @@ a large number of jobs and allows fine-grained OWNERS control on them. The `.prow` directory and `.prow.yaml` file are mutually exclusive; when both are present the `.prow` directory takes precedence. -For more detailed documentation of possible configuration parameters for jobs, please check the [job documentation](https://github.com/kubernetes/test-infra/tree/master/prow/jobs.md) +For more detailed documentation of possible configuration parameters for jobs, please check the [job documentation](/docs/jobs/) diff --git a/site/content/en/docs/Legacy Snapshot/prow/jobs.md b/site/content/en/docs/jobs.md similarity index 87% rename from site/content/en/docs/Legacy Snapshot/prow/jobs.md rename to site/content/en/docs/jobs.md index 33a4edb079..562bdc8041 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/jobs.md +++ b/site/content/en/docs/jobs.md @@ -1,10 +1,11 @@ --- -title: "prow/jobs.md" +title: "ProwJobs" +weight: 160 +description: > + --- -# ProwJobs - -For a brief overview of how Prow runs jobs take a look at ["Life of a Prow Job"](https://github.com/kubernetes/test-infra/tree/master/prow/life_of_a_prow_job.md). +For a brief overview of how Prow runs jobs take a look at ["Life of a Prow Job"](/docs/life-of-a-prow-job/). For a brief cookbook for jobs intended for [prow.k8s.io](https://prow.k8s.io/), please refer to [`config/jobs/README.md`](https://github.com/kubernetes/test-infra/tree/master/config/jobs/README.md) @@ -13,22 +14,22 @@ Make sure Prow has been [deployed] correctly: * The `horologium` component schedules periodic jobs. * The `hook` component schedules presubmit and postsubmit jobs, ensuring the repo: - - enabled `trigger` in [`plugins.yaml`] - - sends GitHub webhooks to prow. + * enabled `trigger` in [`plugins.yaml`] + * sends GitHub webhooks to prow. * The `plank` component schedules the pod requested by a prowjob. * The `crier` component reports status back to github. ## How to configure new jobs To configure a new job you'll need to add an entry into [config.yaml](https://github.com/kubernetes/test-infra/tree/master/config/prow/config.yaml). -If you have [update-config](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/updateconfig) plugin deployed then the +If you have [update-config](/docs/components/plugins/updateconfig/) plugin deployed then the config will be automatically updated once the PR is merged, else you will need to run `make update-config`. This does not require redeploying any binaries, and will take effect within a few minutes. -Alternatively, the [inrepoconfig](https://github.com/kubernetes/test-infra/tree/master/prow/inrepoconfig.md) feature can be used to version Presubmit jobs +Alternatively, the [inrepoconfig](/docs/inrepoconfig/) feature can be used to version Presubmit jobs in the same repository that also contains the code and have Prow load them dynamically. -See [its documentation](https://github.com/kubernetes/test-infra/tree/master/prow/inrepoconfig.md) for more details. +See [its documentation](/docs/inrepoconfig/) for more details. Prow requires you to have a basic understanding of kubernetes, such that you can define pods in yaml. Please see kubernetes documentation @@ -157,6 +158,7 @@ for triggering jobs, the GitHub status contexts that those jobs provide, and the rules for protecting those contexts on branches. ### Triggering Jobs + #### Trigger Types `prow` will consider three different types of jobs that run on pull requests @@ -216,22 +218,23 @@ Both of the above examples would trigger on a pull request containing `foo/bar.c` and `SECURITY.md`, but not one containing only `SECURITY.md`. Note: -- `run_if_changed` and `skip_if_only_changed` are mutually exclusive. -- Jobs which would otherwise be skipped based on this configuration can still + +* `run_if_changed` and `skip_if_only_changed` are mutually exclusive. +* Jobs which would otherwise be skipped based on this configuration can still be triggered explicitly with comments (see below). -- Only presubmit and postsubmit jobs are inherently associated with git refs and can use these fields. +* Only presubmit and postsubmit jobs are inherently associated with git refs and can use these fields. #### Triggering Jobs With Comments A developer may trigger presubmits by posting a comment to a pull request that contains one or more of the following phrases: - - `/test job-name` : When posting `/test job-name`, any jobs with matching triggers +* `/test job-name` : When posting `/test job-name`, any jobs with matching triggers will be triggered unconditionally. - - `/retest` : When posting `/retest`, two types of jobs will be triggered: - - all jobs that have run and failed will run unconditionally - - any not-yet-executed automatically run jobs will run conditionally - - `/test all` : When posting `/test all`, all automatically run jobs will run +* `/retest` : When posting `/retest`, two types of jobs will be triggered: + * all jobs that have run and failed will run unconditionally + * any not-yet-executed automatically run jobs will run conditionally +* `/test all` : When posting `/test all`, all automatically run jobs will run conditionally. Note: It is possible to configure a job's `trigger` to match any of the above keywords @@ -257,15 +260,16 @@ Repo administrators can also `/override job-name` in case of emergency (depends on the `override` plugin). ### Requiring Job Statuses + #### Requiring Jobs for Auto-Merge Through Tide Tide will treat jobs in the following manner for merging: - - unconditionally run jobs with required status contexts are always required to have +* unconditionally run jobs with required status contexts are always required to have passed on a pull request to merge - - conditionally run jobs with required status contexts are required to have passed on +* conditionally run jobs with required status contexts are required to have passed on a pull request to merge if the job currently matches the pull request. - - jobs with optional status contexts are ignored when merging +* jobs with optional status contexts are ignored when merging In order to set a job's context to be optional, set `optional: true` on the job. If it is required to not post the results of the job to GitHub whatsoever, the job may be set @@ -299,11 +303,11 @@ postsubmits: ... ``` -You can learn more about creating and using build clusters in [`scaling.md`](https://github.com/kubernetes/test-infra/tree/master/prow/scaling.md#separate-build-clusters) and [`getting_started_deploy.md`](https://github.com/kubernetes/test-infra/tree/master/prow/getting_started_deploy.md#Run-test-pods-in-different-clusters). +You can learn more about creating and using build clusters in ["Using Prow at Scale"](/docs/scaling/#separate-build-clusters) and ["Deploying Prow"](/docs/getting-started-deploy/#run-test-pods-in-different-clusters). ## Pod Utilities -If you are adding a new job that will execute on a Kubernetes cluster (`agent: kubernetes`, the default value) you should consider using the [Pod Utilities](https://github.com/kubernetes/test-infra/tree/master/prow/pod-utilities.md). The pod utils decorate jobs with additional containers that transparently provide source code checkout and log/metadata/artifact uploading to GCS. +If you are adding a new job that will execute on a Kubernetes cluster (`agent: kubernetes`, the default value) you should consider using the [Pod Utilities](/docs/components/pod-utilities/). The pod utils decorate jobs with additional containers that transparently provide source code checkout and log/metadata/artifact uploading to GCS. ## Job Environment Variables @@ -333,28 +337,32 @@ Examples of the JSON-encoded job specification follow for the different job types: Periodic Job: + ```json {"type":"periodic","job":"job-name","buildid":"0","prowjobid":"uuid","refs":{}} ``` Postsubmit Job: + ```json {"type":"postsubmit","job":"job-name","buildid":"0","prowjobid":"uuid","refs":{"org":"org-name","repo":"repo-name","base_ref":"base-ref","base_sha":"base-sha"}} ``` Presubmit Job: + ```json {"type":"presubmit","job":"job-name","buildid":"0","prowjobid":"uuid","refs":{"org":"org-name","repo":"repo-name","base_ref":"base-ref","base_sha":"base-sha","pulls":[{"number":1,"author":"author-name","sha":"pull-sha"}]}} ``` Batch Job: + ```json {"type":"batch","job":"job-name","buildid":"0","prowjobid":"uuid","refs":{"org":"org-name","repo":"repo-name","base_ref":"base-ref","base_sha":"base-sha","pulls":[{"number":1,"author":"author-name","sha":"pull-sha"},{"number":2,"author":"other-author-name","sha":"second-pull-sha"}]}} ``` ## Testing a new job -See ["How to test a ProwJob"](https://github.com/kubernetes/test-infra/tree/master/prow/build_test_update.md#How-to-test-a-ProwJob). +See ["How to test a ProwJob"](/docs/build-test-update/#how-to-test-a-prowjob). ## Badges @@ -368,5 +376,5 @@ The format to send your `deck` URL is `/badge.svg?jobs=single-job-name` or `/bad [PodSpec api reference]: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#podspec-v1-core [`Presets`]: https://github.com/kubernetes/test-infra/blob/3afb608d28630b99e49e09dd101a96c201268739/prow/config/jobs.go#L33-L40 [`plugins.yaml`]: https://github.com/kubernetes/test-infra/tree/master/config/prow/plugins.yaml -[deployed]: https://github.com/kubernetes/test-infra/blob/master/prow/getting_started_deploy.md +[deployed]: /docs/getting-started-deploy/ [regular expression]: https://golang.org/pkg/regexp/syntax/ diff --git a/site/content/en/docs/Legacy Snapshot/_index.md b/site/content/en/docs/legacy-snapshot/_index.md similarity index 98% rename from site/content/en/docs/Legacy Snapshot/_index.md rename to site/content/en/docs/legacy-snapshot/_index.md index 8676bd45f9..ac07799261 100644 --- a/site/content/en/docs/Legacy Snapshot/_index.md +++ b/site/content/en/docs/legacy-snapshot/_index.md @@ -2,7 +2,7 @@ title: "Legacy Snapshot" description: > Legacy docs -weight: 100 +weight: 1000 --- ## Legacy Snapshot diff --git a/site/content/en/docs/Legacy Snapshot/migrate.sh b/site/content/en/docs/legacy-snapshot/migrate.sh similarity index 100% rename from site/content/en/docs/Legacy Snapshot/migrate.sh rename to site/content/en/docs/legacy-snapshot/migrate.sh diff --git a/site/content/en/docs/Legacy Snapshot/prow/life_of_a_prow_job.md b/site/content/en/docs/life-of-a-prow-job.md similarity index 92% rename from site/content/en/docs/Legacy Snapshot/prow/life_of_a_prow_job.md rename to site/content/en/docs/life-of-a-prow-job.md index 5aa982bc32..322fb70f6e 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/life_of_a_prow_job.md +++ b/site/content/en/docs/life-of-a-prow-job.md @@ -1,9 +1,10 @@ --- -title: "prow/life_of_a_prow_job.md" +title: "Life of a Prow Job" +weight: 120 +description: > + --- -# Life of a Prow Job - NOTE: This document uses [5df7636b83cab54e248e550a31dbf1e4731197a6][prow-repo-sync-point] (July 21, 2021) as a reference point for all code links. Let's pretend a user comments `/test all` on a Pull Request (PR). @@ -28,15 +29,15 @@ Prow Plugins receive 2 objects: The `ClientAgent` object contains the following clients: -- GitHub client -- Prow job client -- Kubernetes client -- BuildClusterCoreV1 clients -- Git client -- Slack client -- Owners client -- Bugzilla client -- Jira client +* GitHub client +* Prow job client +* Kubernetes client +* BuildClusterCoreV1 clients +* Git client +* Slack client +* Owners client +* Bugzilla client +* Jira client These clients are initialized by **hook**, during start-up. @@ -83,11 +84,11 @@ status: state: triggered ``` -[**prow-controller-manager**][prow-controller-manager] runs ProwJobs by launching them by creating a new Kubernetes pod. +[**prow-controller-manager**](/docs/components/core/prow-controller-manager/) runs ProwJobs by launching them by creating a new Kubernetes pod. It knows how to schedule new ProwJobs onto the cluster, responding to changes in the ProwJob or cluster health. When the ProwJob finishes (the containers in the pod have finished running), **prow-controller-manager** updates the ProwJob. -[**crier**][crier] reports back the status of the ProwJob back to the various external services like GitHub (e.g., as a green check-mark on the PR where the original `/test all` comment was made). +[**crier**](/docs/components/core/crier/) reports back the status of the ProwJob back to the various external services like GitHub (e.g., as a green check-mark on the PR where the original `/test all` comment was made). A day later, [**sinker**][sinker] notices that the job and pod are a day old and [deletes them][sinker-clean] from the Kubernetes API server. @@ -141,7 +142,5 @@ Here is a summary of the above: [trigger-handleGenericComment-getPresubmits]: https://github.com/kubernetes/test-infra/blob/5df7636b83cab54e248e550a31dbf1e4731197a6/prow/plugins/trigger/generic-comment.go#L56 [prow-repo-sync-point]: https://github.com/kubernetes/test-infra/tree/5df7636b83cab54e248e550a31dbf1e4731197a6 -[crier]: https://github.com/kubernetes/test-infra/tree/5df7636b83cab54e248e550a31dbf1e4731197a6/prow/cmd/crier [sample-github-webhook-payloads]: https://github.com/kubernetes/test-infra/tree/5df7636b83cab54e248e550a31dbf1e4731197a6/prow/cmd/phony/examples -[prow-controller-manager]: https://github.com/kubernetes/test-infra/tree/5df7636b83cab54e248e550a31dbf1e4731197a6/prow/cmd/prow-controller-manager [prow-plugins-trigger]: https://github.com/kubernetes/test-infra/tree/5df7636b83cab54e248e550a31dbf1e4731197a6/prow/plugins/trigger diff --git a/site/content/en/docs/Legacy Snapshot/prow/metadata_artifacts.md b/site/content/en/docs/metadata-artifacts.md similarity index 98% rename from site/content/en/docs/Legacy Snapshot/prow/metadata_artifacts.md rename to site/content/en/docs/metadata-artifacts.md index 624846fa49..398e6809e7 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/metadata_artifacts.md +++ b/site/content/en/docs/metadata-artifacts.md @@ -1,9 +1,10 @@ --- -title: "prow/metadata_artifacts.md" +title: "Understanding Started.json and Finished.json" +weight: 200 +description: > + --- -# Understanding Started.json and Finished.json - ### Context Prow uploads a host of artifacts throughout the life cycle of a job. Two of these artifacts that are present in each run are `started.json` and `finished.json` which contain a host of information pertaining to the job/run. These files have existed through the evolution of Kubernetes CI: from Jenkins -> Containerized Jenkins -> Bootstrap Containerized Jenkins -> Bootstrap Prow -> PodUtils. As of 2021, all jobs exist within either Bootstrap Prow or PodUtils. As the CI has evolved, so has `started/finished.json` and it's function. diff --git a/site/content/en/docs/Metrics/_index.md b/site/content/en/docs/metrics/_index.md similarity index 99% rename from site/content/en/docs/Metrics/_index.md rename to site/content/en/docs/metrics/_index.md index db266ac423..b41be73dde 100644 --- a/site/content/en/docs/Metrics/_index.md +++ b/site/content/en/docs/metrics/_index.md @@ -1,6 +1,6 @@ --- title: "Metrics" -weight: 10 +weight: 60 description: > Prometheues metrics that can be used for monitoring and alerting. --- diff --git a/site/content/en/docs/Legacy Snapshot/prow/more_prow.md b/site/content/en/docs/more-prow.md similarity index 53% rename from site/content/en/docs/Legacy Snapshot/prow/more_prow.md rename to site/content/en/docs/more-prow.md index a1f96ee9a9..75798aba94 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/more_prow.md +++ b/site/content/en/docs/more-prow.md @@ -1,20 +1,21 @@ --- -title: "prow/more_prow.md" +title: "Getting more out of Prow" +weight: 100 +description: > + --- -# Getting more out of Prow - If you want more functionality from your Prow instance this guide is for you. It primarily links to other resources that catalogue existing components and features. ## Use more Prow components and plugins -Prow has a number of optional cluster components and a suite of plugins for `hook` that provide all sorts of automation. Check out the [README](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/README.md) in the [`prow/cmd`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd) directory for a list of cluster components and the [README](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/README.md) in the [`prow/plugins`](https://github.com/kubernetes/test-infra/tree/master/prow/plugins) directory for information about available plugins. +Prow has a number of optional cluster components and a suite of plugins for `hook` that provide all sorts of automation. Check out the [Components](/docs/components/) for a list of cluster components and the [Plugins](/docs/components/plugins/) for information about available plugins. ## Consume Prometheus metrics -Some Prow components expose prometheus metrics that can be used for monitoring, alerting, and pretty graphs. You can find details in the [README](https://github.com/kubernetes/test-infra/tree/master/prow/metrics/README.md) in the [`prow/metrics`](https://github.com/kubernetes/test-infra/tree/master/prow/metrics) directory. +Some Prow components expose prometheus metrics that can be used for monitoring, alerting, and pretty graphs. You can find details in the [Metrics](/docs/metrics/) document. -## Make Prow update and deploy itself! +## Make Prow update and deploy itself You can easily make your Prow instance automatically update itself when changes are made to its component's kubernetes resource files. This is achieved with a @@ -26,34 +27,33 @@ order to `kubectl apply` to the cluster, the job will need to supply credentials this job requires priviledged credentials to deploy to the cluster, it is important that it is run in a separate build cluster that is isolated from all presubmit jobs. See the -[documentation about separate build clusters](https://github.com/kubernetes/test-infra/tree/master/prow/scaling.md#separate-build-clusters) +[documentation about separate build clusters](/docs/scaling/#separate-build-clusters) for details. An example of such a job can be found [here](https://github.com/istio/test-infra/blob/45526926b4f1cd09147d54d23abc4a4258e62860/prow/cluster/jobs/istio/test-infra/istio.test-infra.trusted.master.yaml#L2-L28). Once you have a postsubmit deploy job, any changes to Prow component files are automatically applied to the cluster when the changes merge. In order to ensure that all changes to production are properly approved, you can use OWNERS files -with the [`approve` plugin](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/approve) and [`Tide`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide). +with the [`approve` plugin](/docs/components/plugins/approve/) and [`Tide`](/docs/components/core/tide/). -With the help of the [Prow Autobump utility](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/generic-autobumper/README.md) you can easily create commits that update all references to Prow images to the latest image version that has been vetted by the https://prow.k8s.io instance. If your Prow component resource files live in GitHub, this utility can even automatically create/update a Pull Request that includes these changes. This works great when run as a periodic job since it will maintain a single open PR that is periodically updated to reference the most recent upstream version. See the [config used to bump prow.k8s.io](https://github.com/kubernetes/test-infra/tree/master/config/prow/autobump-config/prow-component-autobump-config.yaml) for an example +With the help of the [Prow Autobump utility](/docs/components/cli-tools/generic-autobumper/) you can easily create commits that update all references to Prow images to the latest image version that has been vetted by the instance. If your Prow component resource files live in GitHub, this utility can even automatically create/update a Pull Request that includes these changes. This works great when run as a periodic job since it will maintain a single open PR that is periodically updated to reference the most recent upstream version. See the [config used to bump prow.k8s.io](https://github.com/kubernetes/test-infra/tree/master/config/prow/autobump-config/prow-component-autobump-config.yaml) for an example Combining a postsubmit deploy job with a periodic job that runs the Prow Autobump utility allows Prow to be updated to the latest version by simply merging the automatically created Pull Request (or letting Tide merge it after it has been approved). ### Deploy config changes automatically -Prow can also automatically upload changes to files that correspond to Kubernetes ConfigMaps. This includes its own `config`, `plugins` and `job-config` config maps. Take a look at the [`updateconfig` plugin](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/updateconfig) and [`config-bootstrapper`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/config-bootstrapper) for more details. Both of these tools share the [`updateconfig` plugin's plugin configuration](https://github.com/kubernetes/test-infra/blob/531f2a5e6b6fb60e3262340a86992029aa59808f/prow/plugins/config.go#L69). The plugin provides slightly better GitHub integration and is simpler to enable, but the config-bootstrapper is more flexible. It can be run in a postsubmit job to provide config upload on non-GitHub Prow instances or run after custom config generation is executed. +Prow can also automatically upload changes to files that correspond to Kubernetes ConfigMaps. This includes its own `config`, `plugins` and `job-config` config maps. Take a look at the [`updateconfig` plugin](/docs/components/plugins/updateconfig/) and [`config-bootstrapper`](/docs/components/cli-tools/config-bootstrapper/) for more details. Both of these tools share the [`updateconfig` plugin's plugin configuration](https://github.com/kubernetes/test-infra/blob/531f2a5e6b6fb60e3262340a86992029aa59808f/prow/plugins/config.go#L69). The plugin provides slightly better GitHub integration and is simpler to enable, but the config-bootstrapper is more flexible. It can be run in a postsubmit job to provide config upload on non-GitHub Prow instances or run after custom config generation is executed. ## Use other tools with Prow * If you find that your GitHub bot is running low on API tokens consider using [`ghproxy`](https://github.com/kubernetes/test-infra/tree/master/ghproxy) to cache requests to GitHub and take advantage of the strange re-validation rules that allow for additional API token savings. -* [Testgrid](https://github.com/kubernetes/test-infra/tree/master/testgrid) provides a highly configurable visual overview of test results and can be configured to send alerts for failing or stale results. Testgrid is in the process of being open sourced, but until it has completely made the switch OSS users will need to use the https://testgrid.k8s.io instance that is managed by the GKE-Engprod team. +* [Testgrid](https://github.com/kubernetes/test-infra/tree/master/testgrid) provides a highly configurable visual overview of test results and can be configured to send alerts for failing or stale results. Testgrid is in the process of being open sourced, but until it has completely made the switch OSS users will need to use the instance that is managed by the GKE-Engprod team. * [Kind](https://github.com/kubernetes-sigs/kind) lets you run an entire Kubernetes cluster in a container. This makes it fast and easy for ProwJobs to test anything that runs on Kubernetes (or Kubernetes itself). * [label_sync](https://github.com/kubernetes/test-infra/tree/master/label_sync) maintains GitHub labels across orgs and repos based on yaml configuration. ## Handle scale -If your Prow instance operates on a lot of GitHub repos or runs lots of jobs you should review the ["Scaling Prow"](https://github.com/kubernetes/test-infra/tree/master/prow/scaling.md) guide for tips and best practices. - +If your Prow instance operates on a lot of GitHub repos or runs lots of jobs you should review the ["Scaling Prow"](/docs/scaling/) guide for tips and best practices. ## Private Front end -If you want to create a private Deck instnace that contains a subset of prowjobs, you should review the ["Private Deck"](https://github.com/kubernetes/test-infra/tree/master/prow/private_deck.md) guide. +If you want to create a private Deck instnace that contains a subset of prowjobs, you should review the ["Private Deck"](/docs/private-deck/) guide. diff --git a/site/content/en/docs/overview/_index.md b/site/content/en/docs/overview/_index.md new file mode 100644 index 0000000000..4ff5e0dc59 --- /dev/null +++ b/site/content/en/docs/overview/_index.md @@ -0,0 +1,135 @@ +--- +title: "Overview" +weight: 20 +description: > + A brief guide about Prow and its ecosystem +--- + +# ![Prow](/images/logo_horizontal_solid.png) + +Prow is a Kubernetes based CI/CD system. Jobs can be triggered by various types of events and report their status to many different services. In addition to job execution, Prow provides GitHub automation in the form of policy enforcement, chat-ops via `/foo` style commands, and automatic PR merging. + +See the [GoDoc](https://pkg.go.dev/k8s.io/test-infra/prow) for library docs. +Please note that these libraries are intended for use by prow only, and we do +not make any attempt to preserve backwards compatibility. + +For a brief overview of how Prow runs jobs take a look at ["Life of a Prow Job"](/docs/life-of-a-prow-job/). + +To see common Prow usage and interactions flow, see the pull request interactions [sequence diagram](/images/pr-interactions-sequence.svg). + +#### Functions and Features + +* Job execution for testing, batch processing, artifact publishing. + * GitHub events are used to trigger post-PR-merge (postsubmit) jobs and on-PR-update (presubmit) jobs. + * Support for multiple execution platforms and source code review sites. +* Pluggable GitHub bot automation that implements `/foo` style commands and enforces configured policies/processes. +* GitHub merge automation with batch testing logic. +* Front end for viewing jobs, merge queue status, dynamically generated help information, and more. +* Automatic deployment of source control based config. +* Automatic GitHub org/repo administration configured in source control. +* Designed for multi-org scale with dozens of repositories. (The Kubernetes Prow instance uses only 1 GitHub bot token!) +* High availability as benefit of running on Kubernetes. (replication, load balancing, rolling updates...) +* JSON structured logs. +* Prometheus metrics. + +## Documentation + +### Getting started + +* With your own Prow deployment: ["Deploying Prow"](/docs/getting-started-deploy/) +* With developing for Prow: ["Developing and Contributing to Prow"](/docs/getting-started-develop/) +* As a job author: [ProwJobs](/docs/jobs/) + +### More details + +* [Components](/docs/components/) + +* [Plugins](/docs/components/plugins/) +* [ProwJobs](/docs/jobs/) +* [Building, Testing, and Updating](/docs/build-test-update/) +* [General Configuration](/docs/config/) +* [Pod Utilities](/docs/components/pod-utilities/) +* [Scaling Prow](/docs/scaling/) +* [Tide](/docs/components/core/tide/) +* [Metrics](/docs/metrics/) +* ["Life of a Prow Job"](/docs/life-of-a-prow-job/) +* [Getting more out of Prow](/docs/more-prow/) + +### Tests + +The stability of prow is heavily relying on unit tests and integration tests. + +* Unit tests are co-located with prow source code +* [Integration tests](/docs/test/integration/) utilizes [kind](https://kind.sigs.k8s.io/) with hermetic integration tests. See [instructions for adding new integration tests](/docs/test/integration/#adding-new-integration-tests) for more details + +## Useful Talks + +### KubeCon 2020 EU virtual + +* [Going Beyond CI/CD with Prow](https://youtu.be/qQvoImxHydk) + +### KubeCon 2018 EU + +* [Automation and the Kubernetes Contributor Experience](https://www.youtube.com/watch?v=BsIC7gPkH5M) +* [SIG Testing Deep Dive](https://www.youtube.com/watch?v=M32NIHRKaOI) + +### KubeCon 2018 China + +* [SIG Testing Intro](https://youtu.be/WFvC_VdkDFk) + +### KubeCon 2018 Seattle + +* [Behind you PR: K8s with K8s on K8s](https://www.youtube.com/watch?v=pz0lpl6h-Gc) +* [Using Prow for Testing Outside of K8s](https://www.youtube.com/watch?v=DBrkSC6nS8A) +* [Jenkins X (featuring Tide)](https://www.youtube.com/watch?v=IDEa8seAzVc) +* [SIG Testing Intro](https://www.youtube.com/watch?v=7-_O41W3FRU) +* [SIG Testing Deep Dive](https://www.youtube.com/watch?v=1rwiKDTJILY) + +### Misc + +* [Deploy Your Own Kubernetes Prow](https://www.youtube.com/watch?v=eMNwB96A1Qc) + +## Prow in the wild + +Prow is used by the following organizations and projects: + +* [Kubernetes](https://prow.k8s.io) + * This includes [kubernetes](https://github.com/kubernetes), [kubernetes-client](https://github.com/kubernetes-client), [kubernetes-csi](https://github.com/kubernetes-csi), and [kubernetes-sigs](https://github.com/kubernetes-sigs). +* [OpenShift](https://prow.ci.openshift.org/) + * This includes [openshift](https://github.com/openshift), [openshift-s2i](https://github.com/openshift-s2i), [operator-framework](https://github.com/operator-framework), and some repos in [containers](https://github.com/containers) and [heketi](https://github.com/heketi). +* [Istio](https://prow.istio.io/) +* [Knative](https://prow.knative.dev/) +* [Jetstack](https://prow.build-infra.jetstack.net/) +* [Kyma](https://status.build.kyma-project.io/) +* [Metal³](https://prow.apps.test.metal3.io/) +* [Prometheus](http://prombench.prometheus.io/) +* [Caicloud](https://github.com/caicloud) +* [Kubeflow](https://github.com/kubeflow) +* [Azure AKS Engine](https://github.com/Azure/aks-engine/tree/master/.prowci) +* [tensorflow/minigo](https://github.com/tensorflow/minigo#automated-tests) +* [Daisy (Google Compute Image Tools)](https://github.com/GoogleCloudPlatform/compute-image-tools/tree/master/test-infra#prow-and-gubenator) +* [KubeEdge (Kubernetes Native Edge Computing Framework)](https://github.com/kubeedge/kubeedge) +* [Volcano (Kubernetes Native Batch System)](https://github.com/volcano-sh/volcano) +* [Loodse](https://public-prow.loodse.com/) +* [Feast](https://github.com/gojek/feast) +* [Falco](http://prow.falco.org) +* [TiDB](https://prow.tidb.io) +* [Amazon EKS Distro and Amazon EKS Anywhere](https://prow.eks.amazonaws.com/) +* [KubeSphere](https://prow.kubesphere.io) +* [OpenYurt](https://github.com/openyurtio/openyurt) +* [KubeVirt](https://prow.ci.kubevirt.io/) +* [AWS Controllers for Kubernetes](https://prow.ack.aws.dev/) + +[Jenkins X](https://jenkins-x.io/) uses [Prow as part of Serverless Jenkins](https://medium.com/@jdrawlings/serverless-jenkins-with-jenkins-x-9134cbfe6870). + +## Contact us + +If you need to contact the maintainers of Prow you have a few options: + +1. Open an issue in the [kubernetes/test-infra](https://github.com/kubernetes/test-infra) repo. +1. Reach out to the `#prow` channel of the [Kubernetes Slack](https://github.com/kubernetes/community/tree/master/communication#social-media). +1. Contact one of the code owners in [prow/OWNERS](https://github.com/kubernetes/test-infra/tree/master/prow/OWNERS) or in a more specifically scoped OWNERS file. + +### Bots home + +[@k8s-ci-robot](https://github.com/k8s-ci-robot) lives here and is the face of the Kubernetes Prow instance. Here is a [command list](https://go.k8s.io/bot-commands) for interacting with @k8s-ci-robot and other Prow bots. diff --git a/site/content/en/docs/Overview/architecture.md b/site/content/en/docs/overview/architecture.md similarity index 99% rename from site/content/en/docs/Overview/architecture.md rename to site/content/en/docs/overview/architecture.md index 23757b1089..2df7c38323 100644 --- a/site/content/en/docs/Overview/architecture.md +++ b/site/content/en/docs/overview/architecture.md @@ -1,6 +1,6 @@ --- title: "Architecture" -weight: 1 +weight: 10 description: > Prow is made up of a collection of microservices (aka "Prow components") that work together in a **service cluster**, leveraging one or more **build clusters** to schedule Prow Jobs (or just "jobs"). --- diff --git a/site/content/en/docs/Legacy Snapshot/prow/private_deck.md b/site/content/en/docs/private-deck.md similarity index 97% rename from site/content/en/docs/Legacy Snapshot/prow/private_deck.md rename to site/content/en/docs/private-deck.md index 914791b74e..976d33b4c4 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/private_deck.md +++ b/site/content/en/docs/private-deck.md @@ -1,9 +1,10 @@ --- -title: "prow/private_deck.md" +title: "Setting up Private Deck" +weight: 170 +description: > + --- -# Setting up Private Deck - ## 1) [User] Create a PR to Set up TenantIDs for prowjobs and Repos Prow users should create a PR creating tenantID defaults for their org/repos and clusters. Once you set up a tenantID, all prowjobs labelled with that tenantID will only be visible on Deck instances created with the same tenantID. If you already have prowjobs that you don't want to lose access to on Deck, do this step last. If not, do it first to make sure prowjobs you want to keep sequestered do not appear on other instances of Deck. @@ -81,7 +82,7 @@ data: cookieSecret: ... ``` -For more information on how to make these secrets take a look at the [Secrets Documentation](https://github.com/kubernetes/test-infra/tree/master/prow/prow_secrets.md) +For more information on how to make these secrets take a look at the [Secrets Documentation](/docs/prow-secrets/) ## 4 [User] Create the new Deck Deployment diff --git a/site/content/en/docs/Legacy Snapshot/prow/prow_secrets.md b/site/content/en/docs/prow-secrets.md similarity index 98% rename from site/content/en/docs/Legacy Snapshot/prow/prow_secrets.md rename to site/content/en/docs/prow-secrets.md index 6785d83327..01e7620764 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/prow_secrets.md +++ b/site/content/en/docs/prow-secrets.md @@ -1,9 +1,10 @@ --- -title: "prow/prow_secrets.md" +title: "Prow Secrets Management" +weight: 140 +description: > + --- -# Prow Secrets Management - Secrets in prow service/build clusters are managed with [Kubernetes External Secrets](https://github.com/external-secrets/kubernetes-external-secrets), which is responsible for one-way syncing secret values from major secret manager providers such as GCP, Azure, and AWS secret managers into diff --git a/site/content/en/docs/Legacy Snapshot/prow/scaling.md b/site/content/en/docs/scaling.md similarity index 78% rename from site/content/en/docs/Legacy Snapshot/prow/scaling.md rename to site/content/en/docs/scaling.md index 6449e63c40..4ff68b3055 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/scaling.md +++ b/site/content/en/docs/scaling.md @@ -1,12 +1,13 @@ --- -title: "prow/scaling.md" +title: "Using Prow at Scale" +weight: 190 +description: > + --- -# Using Prow at Scale - If you are maintaining a Prow instance that will need to scale to handle a large load, consider using the following best practices, features, and additional tools. -You may also be interested in ["Getting more out of Prow"](https://github.com/kubernetes/test-infra/tree/master/prow/more_prow.md). +You may also be interested in ["Getting more out of Prow"](/docs/more-prow/). ## Features and Tools @@ -24,7 +25,7 @@ Any number of build clusters can be used in order to isolate specific jobs from each other, improve scalability, or allow tenants to provide and manage their own execution environments. Instructions for configuring jobs to run in different clusters can be found -[here](https://github.com/kubernetes/test-infra/tree/master/prow/getting_started_deploy.md#Run-test-pods-in-different-clusters). +[here](/docs/getting-started-deploy/#run-test-pods-in-different-clusters). Production Prow instances should run most ProwJobs in a build cluster separate from the Prow service cluster (the cluster where the Prow components live). Any @@ -38,7 +39,7 @@ and managing an additional build cluster would be wasteful. ### Pull Request Merge Automation Pull Requests can be automatically merged when they satisfy configured merge -requirements using [`tide`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide/). Automating merge is critical for +requirements using [`tide`](/docs/components/core/tide/). Automating merge is critical for large projects where allowing human to click the merge button is either a bottle neck, a security concern, or both. Tide ensures that PRs have been tested against the most recent base branch commit before merging (retesting if @@ -50,8 +51,8 @@ batch whenever possible. If your Prow config starts to grow too large, consider splitting the job config files into more specific and easily reviewed files. This is particularly useful for delegating ownership of ProwJob config to different users or groups via the -use of OWNERS files with the [`approve` plugin](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/approve) and -[`Tide`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide). It is common to enforce custom config policies for +use of OWNERS files with the [`approve` plugin](/docs/components/plugins/approve/) and +[`Tide`](/docs/components/core/tide/). It is common to enforce custom config policies for jobs defined in certain files or directories via presubmit unit tests. This makes it safe for Prow admins to delegate job config ownership by enforcing limitations on what can be configured and by whom. For example, we use a golang @@ -62,8 +63,8 @@ by test-infra oncall. To use this pattern simply aggregate all job configs in a directory of files with unique base names and supply the directory path to components via -`--job-config-path`. The [`updateconfig` plugin](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/updateconfig) and -[`config-bootstrapper`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/config-bootstrapper) support this pattern by +`--job-config-path`. The [`updateconfig` plugin](/docs/components/plugins/updateconfig/) and +[`config-bootstrapper`](/docs/components/cli-tools/config-bootstrapper/) support this pattern by allowing multiple files to be loaded into a single configmap under different keys (different files once mounted to a container). @@ -91,14 +92,14 @@ with Prow: - [`label_sync`](https://github.com/kubernetes/test-infra/tree/master/label_sync/) is a tool that synchronizes labels and their metadata across multiple orgs and repos in order to provide a consistent user experience in a multi-repo project. -- [`branch_protector`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/branchprotector) is a Prow component that +- [`branchprotector`](/docs/components/optional/branchprotector/) is a Prow component that synchronizes GitHub branch requirements and restrictions based on config. -- [`peribolos`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/peribolos) is a tool that synchronizes org settings, +- [`peribolos`](/docs/components/cli-tools/peribolos/) is a tool that synchronizes org settings, teams, and memberships based on config. ### Metrics -Prow exposes some [Prometheus metrics](https://github.com/kubernetes/test-infra/tree/master/prow/metrics/README.md) that can be used to generate graphs and +Prow exposes some [Prometheus metrics](/docs/metrics/) that can be used to generate graphs and alerts. If you are maintaining a Prow instance that handles important workloads you should consider using these metrics for monitoring. @@ -127,13 +128,13 @@ around GitHub's limited permission system: - Permission to trigger presubmit tests is determined based on org membership as configured in the [`triggers`](https://github.com/kubernetes/test-infra/blob/526195d3e22cb90d784c1e4db1c43041a006c848/prow/plugins/plugins.go#L180) plugin config section. - File ownership is described with OWNERS files and change approval is - enforced with the [`approve` plugin](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/approve). See the [docs](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/approve/approvers/README.md) for details. + enforced with the [`approve` plugin](/docs/components/plugins/approve/). See the [docs](/docs/components/plugins/approve/approvers/) for details. - Org member review of the most recent version of the PR is enforced with the - [`lgtm` plugin](https://github.com/kubernetes/test-infra/tree/master/prow/plugins/lgtm). + [`lgtm` plugin](/docs/components/plugins/lgtm/). - Various other plugins manage labels, milestone, and issue state based on `/foo` style commands from authorized users. Authorization may be based on org membership, GitHub team membership, or OWNERS file membership. - - [`Tide`](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/tide) provides PR merge automation so that humans do not need to (and are not + - [`Tide`](/docs/components/core/tide/) provides PR merge automation so that humans do not need to (and are not allowed to) merge PRs. Without Tide, a user either has no permission to merge or they have repo write access which grants permission to merge any PR in the entire repo. Additionally, Tide enforces merge requirements like diff --git a/site/content/en/docs/Legacy Snapshot/prow/spyglass/README.md b/site/content/en/docs/spyglass/_index.md similarity index 93% rename from site/content/en/docs/Legacy Snapshot/prow/spyglass/README.md rename to site/content/en/docs/spyglass/_index.md index 8b77144f20..256f87a1ec 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/spyglass/README.md +++ b/site/content/en/docs/spyglass/_index.md @@ -1,18 +1,21 @@ --- -title: "prow/spyglass/README.md" +title: "Spyglass" +weight: 180 +description: > + --- # Spyglass -Spyglass is a pluggable artifact viewer framework for [Prow](https://github.com/kubernetes/test-infra/tree/master/prow). It collects artifacts +Spyglass is a pluggable artifact viewer framework for [Prow](/docs/overview/). It collects artifacts (usually files in a storage bucket) from various sources and distributes them to registered viewers, which are responsible for consuming them and rendering a view. A typical Spyglass page might look something like this: -![](https://github.com/kubernetes/test-infra/tree/master/prow/spyglass/spyglass-example.png) +![](./spyglass-example.png) -If you want to know how to write a Spyglass lens, check [the lens-writing guide](https://github.com/kubernetes/test-infra/tree/master/prow/spyglass/write-a-lens.md). -If you're interested in how Spyglass works, check [the architecture summary](https://github.com/kubernetes/test-infra/tree/master/prow/spyglass/architecture.md). +If you want to know how to write a Spyglass lens, check [the lens-writing guide](/docs/spyglass/write-a-lens/). +If you're interested in how Spyglass works, check [the architecture summary](/docs/spyglass/architecture/). ## Configuration diff --git a/site/content/en/docs/Legacy Snapshot/prow/spyglass/architecture.md b/site/content/en/docs/spyglass/architecture.md similarity index 98% rename from site/content/en/docs/Legacy Snapshot/prow/spyglass/architecture.md rename to site/content/en/docs/spyglass/architecture.md index e953f172b9..712d3d8bc9 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/spyglass/architecture.md +++ b/site/content/en/docs/spyglass/architecture.md @@ -1,9 +1,10 @@ --- -title: "prow/spyglass/architecture.md" +title: "Spyglass Architecture" +weight: 10 +description: > + --- -# Spyglass Architecture - Spyglass is split into two major parts: the Spyglass core, and a set of independent lenses. Lenses are designed to run statelessly and without any knowledge of the world outside being provided with a list of artifacts. The core is responsible for selecting lenses and providing them diff --git a/site/content/en/docs/Legacy Snapshot/prow/spyglass/lenses/restcoverage/README.md b/site/content/en/docs/spyglass/lenses/restcoverage.md similarity index 96% rename from site/content/en/docs/Legacy Snapshot/prow/spyglass/lenses/restcoverage/README.md rename to site/content/en/docs/spyglass/lenses/restcoverage.md index c477167fc9..4930b6b5b8 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/spyglass/lenses/restcoverage/README.md +++ b/site/content/en/docs/spyglass/lenses/restcoverage.md @@ -1,9 +1,10 @@ --- -title: "prow/spyglass/lenses/restcoverage/README.md" +title: "REST API coverage lens" +weight: 30 +description: > + --- -# REST API coverage lens - Presents REST endpoints statistics ## Configuration diff --git a/site/content/en/docs/spyglass/spyglass-example.png b/site/content/en/docs/spyglass/spyglass-example.png new file mode 100644 index 0000000000..928584d8c5 Binary files /dev/null and b/site/content/en/docs/spyglass/spyglass-example.png differ diff --git a/site/content/en/docs/Legacy Snapshot/prow/spyglass/write-a-lens.md b/site/content/en/docs/spyglass/write-a-lens.md similarity index 99% rename from site/content/en/docs/Legacy Snapshot/prow/spyglass/write-a-lens.md rename to site/content/en/docs/spyglass/write-a-lens.md index e53e5ef918..f1e9b081e6 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/spyglass/write-a-lens.md +++ b/site/content/en/docs/spyglass/write-a-lens.md @@ -1,9 +1,10 @@ --- -title: "prow/spyglass/write-a-lens.md" +title: "Build a Spyglass Lens" +weight: 20 +description: > + --- -# Build a Spyglass Lens - Spyglass lenses consist of two components: a frontend (which may be trivial) and a backend. ## Lens backend @@ -19,7 +20,6 @@ An instance of the struct implementing the `lenses.Lens` interface must then be spyglass, by calling [`lenses.RegisterLens`](https://godoc.org/k8s.io/test-infra/prow/spyglass/lenses#RegisterLens). A minimal example of a lens called `samplelens`, located at `lenses/samplelens`, might look like this: - ```go package samplelens diff --git a/site/content/en/docs/test/_index.md b/site/content/en/docs/test/_index.md new file mode 100644 index 0000000000..c71c8d7b2b --- /dev/null +++ b/site/content/en/docs/test/_index.md @@ -0,0 +1,6 @@ +--- +title: "Testing Prow" +weight: 210 +description: > + +--- diff --git a/site/content/en/docs/Legacy Snapshot/prow/test/integration/README.md b/site/content/en/docs/test/integration/_index.md similarity index 97% rename from site/content/en/docs/Legacy Snapshot/prow/test/integration/README.md rename to site/content/en/docs/test/integration/_index.md index cff0c15fca..c79527b063 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/test/integration/README.md +++ b/site/content/en/docs/test/integration/_index.md @@ -1,9 +1,10 @@ --- -title: "prow/test/integration/README.md" +title: "Run Prow integration tests" +weight: 10 +description: > + --- -# Run Prow integration tests - ## Run all integration tests ```bash @@ -50,7 +51,7 @@ Assume we want to add `most-awesome-component` (source code in `prow/cmd/most-aw components](https://github.com/kubernetes/test-infra/tree/master/prow/test/integration/setup-prow-components.sh). - If you want to deploy an existing Prow component used in production (i.e., - https://prow.k8s.io), you can reuse (symlink) the configurations used in + ), you can reuse (symlink) the configurations used in production. See the examples in [configuration folder][config/prow/cluster]. - Remember to use `localhost:5001/most-awesome-component` for the `image: ...` diff --git a/site/content/en/docs/Legacy Snapshot/prow/test/integration/cmd/fakegitserver/README.md b/site/content/en/docs/test/integration/fakegitserver.md similarity index 98% rename from site/content/en/docs/Legacy Snapshot/prow/test/integration/cmd/fakegitserver/README.md rename to site/content/en/docs/test/integration/fakegitserver.md index 52b13ce408..29f9a23ab6 100644 --- a/site/content/en/docs/Legacy Snapshot/prow/test/integration/cmd/fakegitserver/README.md +++ b/site/content/en/docs/test/integration/fakegitserver.md @@ -1,9 +1,10 @@ --- -title: "prow/test/integration/cmd/fakegitserver/README.md" +title: "Fake Git Server (FGS)" +weight: 40 +description: > + --- -# Fake Git Server (FGS) - FGS is actually not a fake at all. It is a real web server that serves real Git repositories them over HTTP. FGS wraps around the vanilla `git http-backend` subcommand that comes with Git, calling it as a CGI executable. It supports both diff --git a/site/static/images/architecture.png b/site/static/images/architecture.png new file mode 100644 index 0000000000..5e7d92b3e8 Binary files /dev/null and b/site/static/images/architecture.png differ diff --git a/site/static/images/architecture.svg b/site/static/images/architecture.svg new file mode 100644 index 0000000000..70814e7f6b --- /dev/null +++ b/site/static/images/architecture.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/site/static/images/logo-horizontal.png b/site/static/images/logo-horizontal.png new file mode 100644 index 0000000000..90b7a2d91c Binary files /dev/null and b/site/static/images/logo-horizontal.png differ diff --git a/site/static/images/logo-horizontal.svg b/site/static/images/logo-horizontal.svg new file mode 100644 index 0000000000..f94cd98f14 --- /dev/null +++ b/site/static/images/logo-horizontal.svg @@ -0,0 +1,608 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/site/static/images/logo-vertical.png b/site/static/images/logo-vertical.png new file mode 100644 index 0000000000..d3710111e5 Binary files /dev/null and b/site/static/images/logo-vertical.png differ diff --git a/site/static/images/logo-vertical.svg b/site/static/images/logo-vertical.svg new file mode 100644 index 0000000000..f7fb72b7c4 --- /dev/null +++ b/site/static/images/logo-vertical.svg @@ -0,0 +1,239 @@ + +image/svg+xml \ No newline at end of file diff --git a/site/content/en/docs/Legacy Snapshot/prow/logo_horizontal_solid.png b/site/static/images/logo_horizontal_solid.png similarity index 100% rename from site/content/en/docs/Legacy Snapshot/prow/logo_horizontal_solid.png rename to site/static/images/logo_horizontal_solid.png diff --git a/site/static/images/pr-interactions-sequence.mmd b/site/static/images/pr-interactions-sequence.mmd new file mode 100644 index 0000000000..55a7d7dfb9 --- /dev/null +++ b/site/static/images/pr-interactions-sequence.mmd @@ -0,0 +1,51 @@ +sequenceDiagram + participant Contributor + participant Reviewer + participant Approver + participant GitHub + participant hook + participant approve as approve
(hook plugin) + participant size as size
(hook plugin) + participant cla as cla
(hook plugin) + participant lgtm as lgtm
(hook plugin) + participant trigger as trigger
(hook plugin) + participant tide + participant prowJob + participant k8s as Kubernetes
API Server + + Contributor->>GitHub: Creates Pull Request (PR) + GitHub->>hook: Sends PR create webhook + hook->>approve: Sends PR create webhook + approve->>approve: Analyzes repository approvers + approve->>GitHub: Create PR comment "PR not approved" + hook->>size: Send PR create webhook + size->>size: Analyzes PR changed line count + size->>GitHub: Adds "size/XS" label to PR + hook->>cla: Send PR create webhook + cla->>cla: Analyzes if Contributor is CLA approved + cla->>GitHub: Adds "cncf-cla: yes" label to PR + + hook->>GitHub: Adds Reviewers to PR + GitHub->>Reviewer: Requests Review (Normally an Email) + Reviewer->>GitHub: Requests PR Changes + Contributor->>GitHub: Make Changes Reviewer Requested + Reviewer->>GitHub: Creates a "/lgtm" Comment + GitHub->>hook: Sends comment Event + hook->>lgtm: Sends comment Event + lgtm->>lgtm: Analyzes OWNERS files + lgtm->>GitHub: Adds "lgtm" label to PR + + Reviewer->>GitHub: Creates an "/ok-to-test" Comment + hook->>GitHub: Adds "ok-to-test" label to PR + trigger->>k8s: Create presubmit prowJob + k8s->>prowJob: Run presubmit prowJob + prowJob->>GitHub: Update PR "statuses" based on job exit code + + Approver->>GitHub: Creates an "/approved" Comment + GitHub->>approve: Sends comment Event + approve->>approve: Analyzes if Approver can approve PR + approve->>GitHub: Adds an "approved" label to PR + + tide->>GitHub: Checks if All Merge Labels Exist on PR + tide->>GitHub: Merges PR + diff --git a/site/static/images/pr-interactions-sequence.svg b/site/static/images/pr-interactions-sequence.svg new file mode 100644 index 0000000000..64e3c765b8 --- /dev/null +++ b/site/static/images/pr-interactions-sequence.svg @@ -0,0 +1,360 @@ +ContributorReviewerApproverGitHubhookapprove(hook plugin)size(hook plugin)cla(hook plugin)lgtm(hook plugin)trigger(hook plugin)tideprowJobKubernetesAPI ServerCreates Pull Request (PR)Sends PR create webhookSends PR create webhookAnalyzes repository approversCreate PR comment "PR not approved"Send PR create webhookAnalyzes PR changed line countAdds "size/XS" label to PRSend PR create webhookAnalyzes if Contributor is CLA approvedAdds "cncf-cla: yes" label to PRAdds Reviewers to PRRequests Review (Normally an Email)Requests PR ChangesMake Changes Reviewer RequestedCreates a "/lgtm" CommentSends comment EventSends comment EventAnalyzes OWNERS filesAdds "lgtm" label to PRCreates an "/ok-to-test" CommentAdds "ok-to-test" label to PRCreate presubmit prowJobRun presubmit prowJobUpdate PR "statuses" based on job exit codeCreates an "/approved" CommentSends comment EventAnalyzes if Approver can approve PRAdds an "approved" label to PRChecks if All Merge Labels Exist on PRMerges PRContributorReviewerApproverGitHubhookapprove(hook plugin)size(hook plugin)cla(hook plugin)lgtm(hook plugin)trigger(hook plugin)tideprowJobKubernetesAPI Server \ No newline at end of file