Bootstrap the new clusters in k8s-infra-prow project. #7127
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| /* | ||
| Copyright 2024 The Kubernetes Authors. | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); | ||
| you may not use this file except in compliance with the License. | ||
| You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| */ | ||
|
|
||
| module "iam" { | ||
| source = "terraform-google-modules/iam/google//modules/projects_iam" | ||
| version = "~> 7" | ||
|
|
||
| projects = ["kubernetes-public"] | ||
|
|
||
| mode = "authoritative" | ||
|
|
||
| bindings = { | ||
| "roles/secretmanager.secretAccessor" = [ | ||
| "serviceAccount:kubernetes-external-secrets@kubernetes-public.iam.gserviceaccount.com", | ||
| "principal://iam.googleapis.com/projects/16065310909/locations/global/workloadIdentityPools/k8s-infra-prow.svc.id.goog/subject/ns/external-secrets/sa/external-secrets", | ||
| ] | ||
| } | ||
| } |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| apiVersion: argoproj.io/v1alpha1 | ||
| kind: Application | ||
| metadata: | ||
| name: argocd | ||
| spec: | ||
| destination: | ||
| namespace: argocd | ||
| server: https://kubernetes.default.svc | ||
| project: default | ||
| source: | ||
| path: kubernetes/gke-utility/argocd | ||
| repoURL: https://github.com/kubernetes/k8s.io | ||
| targetRevision: main | ||
| syncPolicy: | ||
| automated: | ||
| prune: false | ||
| selfHeal: true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| apiVersion: argoproj.io/v1alpha1 | ||
| kind: ApplicationSet | ||
| metadata: | ||
| name: cert-manager | ||
| spec: | ||
| goTemplate: true | ||
| generators: | ||
| - clusters: | ||
| selector: | ||
| matchLabels: | ||
| clusterType: 'utility' | ||
| template: | ||
| metadata: | ||
| name: 'cert-manager-{{ .name }}' | ||
| spec: | ||
| destination: | ||
| namespace: cert-manager | ||
| server: "{{ .server }}" | ||
| project: default | ||
| sources: | ||
| - chart: cert-manager | ||
| repoURL: 'https://charts.jetstack.io' | ||
| targetRevision: v1.14.5 | ||
| helm: | ||
| releaseName: cert-manager | ||
| valueFiles: | ||
| - $values/kubernetes/{{ .name }}/helm/cert-manager.yaml | ||
| - repoURL: 'https://github.com/kubernetes/k8s.io.git' | ||
| targetRevision: main | ||
| ref: values | ||
| syncPolicy: | ||
| automated: | ||
| prune: true | ||
| selfHeal: true | ||
| syncOptions: | ||
| - CreateNamespace=true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| apiVersion: argoproj.io/v1alpha1 | ||
| kind: ApplicationSet | ||
| metadata: | ||
| name: external-secrets | ||
| spec: | ||
| goTemplate: true | ||
| generators: | ||
| # targets all clusters | ||
| - clusters: | ||
| selector: | ||
| matchExpressions: | ||
| - key: clusterType | ||
| operator: Exists | ||
| template: | ||
| metadata: | ||
| name: 'external-secrets-{{ .name }}' | ||
| spec: | ||
| destination: | ||
| namespace: external-secrets | ||
| server: "{{ .server }}" | ||
| project: default | ||
| sources: | ||
| - chart: external-secrets | ||
| repoURL: 'https://charts.external-secrets.io' | ||
| targetRevision: v0.9.18 | ||
| helm: | ||
| releaseName: external-secrets | ||
| parameters: | ||
| - name: installCRDs | ||
| value: 'true' | ||
| - name: serviceAccount.name | ||
| value: external-secrets | ||
| valueFiles: | ||
| - $values/kubernetes/{{ .name }}/helm/external-secrets.yaml | ||
|
There was a problem hiding this comment. How does this There was a problem hiding this comment. It's resolved from the values defined below. |
||
| - repoURL: 'https://github.com/kubernetes/k8s.io.git' | ||
| targetRevision: main | ||
| ref: values | ||
| syncPolicy: | ||
| automated: | ||
| prune: true | ||
| selfHeal: true | ||
| syncOptions: | ||
| - CreateNamespace=true | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| apiVersion: argoproj.io/v1alpha1 | ||
| kind: ApplicationSet | ||
| metadata: | ||
| name: ingress-nginx | ||
| spec: | ||
| goTemplate: true | ||
| generators: | ||
| - clusters: | ||
| selector: | ||
| matchLabels: | ||
| clusterType: 'utility' | ||
| template: | ||
| metadata: | ||
| name: 'ingress-nginx-{{ .name }}' | ||
| spec: | ||
| destination: | ||
| namespace: ingress-nginx | ||
| server: "{{ .server }}" | ||
| project: default | ||
| sources: | ||
| - chart: ingress-nginx | ||
| repoURL: 'https://kubernetes.github.io/ingress-nginx' | ||
| targetRevision: v4.10.1 | ||
| helm: | ||
| releaseName: ingress-nginx | ||
| valueFiles: | ||
| - $values/kubernetes/{{ .name }}/helm/ingress-nginx.yaml | ||
| - repoURL: 'https://github.com/kubernetes/k8s.io.git' | ||
| targetRevision: main | ||
| ref: values | ||
| syncPolicy: | ||
| automated: | ||
| prune: true | ||
| selfHeal: true | ||
| syncOptions: | ||
| - CreateNamespace=true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
| resources: | ||
| # - argocd.yaml This has been manually applied to fix sync issues | ||
| - external-secrets.yaml | ||
| - cert-manager.yaml | ||
| - ingress-nginx.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| extraObjects: | ||
| - apiVersion: external-secrets.io/v1beta1 | ||
| kind: ClusterSecretStore | ||
| metadata: | ||
| name: k8s-infra-prow | ||
| spec: | ||
| provider: | ||
| gcpsm: | ||
| projectID: k8s-infra-prow | ||
| - apiVersion: external-secrets.io/v1beta1 | ||
| kind: ClusterSecretStore | ||
| metadata: | ||
| name: kubernetes-public | ||
| spec: | ||
| provider: | ||
| gcpsm: | ||
| projectID: kubernetes-public | ||
| - apiVersion: external-secrets.io/v1beta1 | ||
| kind: ClusterSecretStore | ||
| metadata: | ||
| name: k8s-infra-prow-build-trusted | ||
| spec: | ||
| provider: | ||
| gcpsm: | ||
| projectID: k8s-infra-prow-build-trusted |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: argocd-rbac-cm | ||
| data: | ||
| policy.default: role:readonly | ||
| policy.csv: | | ||
| g, kubernetes:sig-k8s-infra-leads, role:admin | ||
|
There was a problem hiding this comment. We should think about having a way to grant other folks access to Argo, e.g. @koksay and I will need it once we extend it to the EKS cluster. |
||
| scopes: '[groups, email]' | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: argocd-cm | ||
| data: | ||
|
There was a problem hiding this comment. We should include some comments what are different options doing and why did we enable/disable these options. |
||
| url: https://argo.k8s.io | ||
| application.instanceLabelKey: infra.k8s.io/instance | ||
| resource.compareoptions: | | ||
| ignoreAggregatedRoles: true | ||
| resource.customizations: | | ||
| admissionregistration.k8s.io/MutatingWebhookConfiguration: | ||
| ignoreDifferences: | | ||
| jqPathExpressions: | ||
| - '.webhooks[]?.clientConfig.caBundle' | ||
| kustomize.buildOptions: --load-restrictor LoadRestrictionsNone --enable-alpha-plugins | ||
| dex.config: | | ||
| connectors: | ||
| - type: github | ||
| id: github | ||
| name: GitHub | ||
| config: | ||
| clientID: $dex.github.clientId | ||
| clientSecret: $dex.github.clientSecret | ||
| orgs: | ||
| - name: kubernetes | ||
| useLoginAsID: true | ||
| loadAllGroups: true | ||
| teamNameField: slug | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| apiVersion: v1 | ||
| kind: ServiceAccount | ||
| metadata: | ||
| annotations: | ||
| iam.gke.io/gcp-service-account: argocd@k8s-infra-prow.iam.gserviceaccount.com | ||
| name: argocd-application-controller | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ServiceAccount | ||
| metadata: | ||
| annotations: | ||
| iam.gke.io/gcp-service-account: argocd@k8s-infra-prow.iam.gserviceaccount.com | ||
upodroid marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| name: argocd-server | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| apiVersion: v1 | ||
| kind: Secret | ||
| metadata: | ||
| name: gke-prow | ||
| labels: | ||
| argocd.argoproj.io/secret-type: cluster | ||
| clusterType: prow | ||
| environment: prod | ||
| cloud: gke | ||
| type: Opaque | ||
| stringData: | ||
| name: gke-prow | ||
| server: https://10.254.0.18 | ||
| config: | | ||
| { | ||
| "execProviderConfig": { | ||
| "command": "argocd-k8s-auth", | ||
| "args": ["gcp"], | ||
| "apiVersion": "client.authentication.k8s.io/v1beta1" | ||
| }, | ||
| "tlsClientConfig": { | ||
| "insecure": true | ||
| } | ||
| } | ||
| --- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| apiVersion: v1 | ||
| kind: Secret | ||
| metadata: | ||
| name: kubernetes-repo | ||
| namespace: argocd | ||
| labels: | ||
| argocd.argoproj.io/secret-type: repository | ||
| stringData: | ||
| url: https://github.com/kubernetes | ||
|
There was a problem hiding this comment. Looking at the ArgoCD docs, it appears that this should be the full repository URL, not the organization URL. How do we exactly use this Secret? There was a problem hiding this comment. Look at the third example called private-repo-creds and the paragraph below it. There was a problem hiding this comment. I'm not sure I fully understand why do we need this, because the document is describing this in the context of private repositories, but all repositories in k-org that we use are public. Does this repository configure Argo to connect to k-org without authentication or there's something else to it? |
||
| name: kubernetes | ||
| type: git | ||
| --- | ||
| apiVersion: argoproj.io/v1alpha1 | ||
| kind: Application | ||
| metadata: | ||
| name: apps | ||
| spec: | ||
| destination: | ||
| namespace: argocd | ||
| server: https://kubernetes.default.svc | ||
| project: default | ||
| source: | ||
| path: kubernetes/apps | ||
| repoURL: https://github.com/borg-land/k8s.io | ||
| targetRevision: utility-dev | ||
| syncPolicy: | ||
| automated: | ||
| prune: false | ||
|
There was a problem hiding this comment. I see There was a problem hiding this comment. ArgoCD will be the one service where pruning is disabled. All other deployments will be automated with pruning enabled. |
||
| selfHeal: true | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
| namespace: argocd | ||
|
|
||
| resources: | ||
| - github.com/argoproj/argo-cd/manifests/ha/cluster-install?ref=v2.11.2 | ||
|
There was a problem hiding this comment. I'm wondering if we should keep the source manifests committed to this repo. That way we're sure we're always going to have access to the manifest we deployed. Also, we can easier track what's changed in the manifest and if there's anything that we need to pay attention to. There was a problem hiding this comment. Another comment: v2.11.2 is not too old, but v2.12.0 has just been released. |
||
| - extras.yaml | ||
|
|
||
| patches: | ||
| - path: argocd-cm.yaml | ||
| - path: argocd-cm-rbac.yaml | ||
| - path: argocd-sa.yaml | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| installCRDs: true | ||
| extraObjects: | ||
| - | | ||
| apiVersion: cert-manager.io/v1 | ||
| kind: ClusterIssuer | ||
| metadata: | ||
| name: letsencrypt-prod | ||
| spec: | ||
| acme: | ||
| server: https://acme-v02.api.letsencrypt.org/directory | ||
| email: sig-k8s-infra-leads+certificates@kubernetes.io | ||
| privateKeySecretRef: | ||
| name: letsencrypt-prod | ||
| solvers: | ||
| - http01: | ||
| ingress: | ||
| ingressClassName: nginx |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| extraObjects: | ||
| - apiVersion: external-secrets.io/v1beta1 | ||
| kind: ClusterSecretStore | ||
| metadata: | ||
| name: k8s-infra-prow | ||
| spec: | ||
| provider: | ||
| gcpsm: | ||
| projectID: k8s-infra-prow |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| controller: | ||
| publishService: | ||
| enabled: true | ||
| service: | ||
| annotations: | ||
| networking.gke.io/load-balancer-ip-addresses: utility-ingress-v4,utility-ingress-v6 | ||
| cloud.google.com/l4-rbs: "enabled" | ||
| externalTrafficPolicy: Local | ||
| ipFamilyPolicy: RequireDualStack | ||
| ipFamilies: | ||
| - IPv6 | ||
| - IPv4 |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How are clusters defined? I see that we have the
gke-prowcluster defined, but I don't see thegke-utilitycluster defined and labeled. Is it done manually/via click-ops?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
gke-utility is the default cluster. it was missing labels so I renamed it in the Argo Console and added the labels you see on gke-prow
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, thanks for the clarification! I think we should document this as part of the README.