Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deploy prow to k8s-infra-prow cluster #7141

Merged
merged 12 commits into from
Aug 22, 2024
Merged

deploy prow to k8s-infra-prow cluster #7141

merged 12 commits into from
Aug 22, 2024

Conversation

upodroid
Copy link
Member

@upodroid upodroid commented Aug 8, 2024

This is the prow specific piece of #7127

I copied the manifests from https://github.com/kubernetes/test-infra/tree/master/config/prow/cluster and adjusted a few things:

  1. I consolidated the prow compoment files in to a single file. so deck-service, deck-rbac and deck-deployment are now just deck.
  2. I commented out a few services that we don't use
  3. The metrics will be handled separately by using Managed Prometheus and visualising it in Grafana https://github.com/knative/infra/blob/002a2bd25f5b3488c258baeb93b260e34d8956ee/prow/cluster/control-plane/200-monitoring.yaml
  4. Prow will be deployed to the prow namespace instead of default. (I need to adjust the EKS IAM roles)
  5. The controllers need to be scaled down before this PR is merged.
  6. The secrets needs to be loaded into the project and their manifests rewritten

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 8, 2024
@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. labels Aug 8, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 8, 2024
@upodroid upodroid mentioned this pull request Aug 8, 2024
Closed
xmudrii
xmudrii reviewed Aug 15, 2024
View reviewed changes
kubernetes/apps/prow.yaml
- clusters:
selector:
matchLabels:
clusterType: 'prow'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should have prow-control-plane and prow-build as Argo will be used for build clusters as well eventually. That way we avoid deploying stuff into wrong cluster by accident.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each cluster will have its own prow folder to deploy prow specific resources.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I see, then it's all good.

kubernetes/apps/prow.yaml Show resolved Hide resolved
kubernetes/gke-prow/prow/external-secrets.yaml Outdated Show resolved Hide resolved
kubernetes/apps/prow.yaml Outdated
server: "{{ .server }}"
project: default
source:
path: kubernetes/{{ .name }}/prow
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't path supposed to be gke-prow?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{{.name}} is based on the clusterName defined in ArgoCD.

https://github.com/kubernetes/k8s.io/blob/main/kubernetes/gke-utility/argocd/clusters.yaml#L4

kubernetes/gke-prow/prow/prow-controller-manager.yaml Outdated Show resolved Hide resolved
@upodroid upodroid changed the title [WIP] deploy prow to k8s-infra-prow cluster deploy prow to k8s-infra-prow cluster Aug 16, 2024
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 16, 2024
@michelle192837 michelle192837 mentioned this pull request Aug 21, 2024
Open
44 tasks
xmudrii
xmudrii reviewed Aug 22, 2024
View reviewed changes
Copy link
Member

@xmudrii xmudrii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two final nits. Can't comment much on the Prow manifests, but given everything is up and running, I'm sure we're fine. :D

kubernetes/gke-prow/prow/hook.yaml
Comment on lines +55 to +61
# AWS_ variables needed to assume role to access the prow-build-cluster EKS cluster.
- name: AWS_ROLE_ARN
value: arn:aws:iam::468814281478:role/Prow-EKS-Admin
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/aws-iam-token/serviceaccount/token
- name: AWS_REGION
value: us-east-2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Random note/blurb: we should look into if there's a better way to handle this. I'm wondering how we would handle it if we had multiple Prow build clusters on AWS.

Copy link
Member Author

@upodroid upodroid Aug 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have multiple AWS clusters. We allow this role arn:aws:iam::468814281478:role/Prow-EKS-Admin to access the cluster

kubernetes/gke-prow/prow/kustomization.yaml
kind: Kustomization
namespace: default

resources:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit: would be nice to sort this file A-Z, gateway.yaml and monitoring.yaml are in the wrong place.

xmudrii
xmudrii reviewed Aug 22, 2024
View reviewed changes
kubernetes/gke-prow/prow/crds/prowjob-crd.yaml Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we apply this file? I don't see it being mentioned in kustomization.yaml unless I missed it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check the prow.yaml argo app

@upodroid upodroid added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Aug 22, 2024
@upodroid
Copy link
Member Author

This is ready to be merged

BenTheElder
BenTheElder reviewed Aug 22, 2024
View reviewed changes
Copy link
Member

@BenTheElder BenTheElder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/hold

NOTE: we're going to have reduced availability to fix things for a bit, I am out tomorrow and probably some portion of the next two weeks, Arnaud is out and Dims is out already.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 22, 2024
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 22, 2024
@upodroid
Copy link
Member Author

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 22, 2024
@k8s-ci-robot k8s-ci-robot merged commit 258cfde into kubernetes:main Aug 22, 2024
3 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.32 milestone Aug 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BenTheElder BenTheElder BenTheElder left review comments

@xmudrii xmudrii xmudrii left review comments

@cjwagner cjwagner Awaiting requested review from cjwagner

Assignees

@BenTheElder BenTheElder

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.
Projects
None yet
Milestone
v1.32
Development

Successfully merging this pull request may close these issues.
4 participants
@upodroid @k8s-ci-robot @BenTheElder @xmudrii