From cf9a8e0b88fa14a13fc83035bfb588e9283871eb Mon Sep 17 00:00:00 2001 From: hc-github-team-secure-vault-core <82990506+hc-github-team-secure-vault-core@users.noreply.github.com> Date: Thu, 20 Jan 2022 22:25:21 -0500 Subject: [PATCH] Docs to clarify k8s auth options with short-lived tokens (#13275) (#13720) * Rework 1.21 content into one heading and add note at top * Add notes about extended k8s token duration * Add example of ClusterRoleBinding for using client JWTs Co-authored-by: Tom Proctor --- .../content/docs/auth/jwt/oidc_providers.mdx | 214 ++++++++++++++++++ website/content/docs/auth/kubernetes.mdx | 196 +++++++++++++--- 2 files changed, 373 insertions(+), 37 deletions(-) diff --git a/website/content/docs/auth/jwt/oidc_providers.mdx b/website/content/docs/auth/jwt/oidc_providers.mdx index b8dd58dc369a..cbc8141eae7e 100644 --- a/website/content/docs/auth/jwt/oidc_providers.mdx +++ b/website/content/docs/auth/jwt/oidc_providers.mdx @@ -253,6 +253,220 @@ vault write auth/oidc/role/your_default_role \ 1. Save. 1. Visit Credentials. Select Client ID and Secret and note the generated secret. +## Kubernetes + +Kubernetes can function as an OIDC provider such that Vault can validate its +service account tokens using JWT/OIDC auth. + +-> **Note:** The JWT auth engine does **not** use Kubernetes' `TokenReview` API +during authentication, and instead uses public key cryptography to verify the +contents of JWTs. This means tokens that have been revoked by Kubernetes will +still be considered valid by Vault until their expiry time. To mitigate this +risk, use short TTLs for service account tokens or use +[Kubernetes auth](/docs/auth/kubernetes) which _does_ use the `TokenReview` API. + +### Using service account issuer discovery + +When using service account issuer discovery, you only need to provide the JWT +auth mount with an OIDC discovery URL, and sometimes a TLS certificate authority +to trust. This makes it the most straightforward method to configure if your +Kubernetes cluster meets the requirements. + +Kubernetes cluster requirements: + +* [`ServiceAccountIssuerDiscovery`][k8s-sa-issuer-discovery] feature enabled. + * Present from 1.18, defaults to enabled from 1.20. +* kube-apiserver's `--service-account-issuer` flag is set to a URL that is + reachable from Vault. Public by default for most managed Kubernetes solutions. +* Must use short-lived service account tokens when logging in. + * Tokens mounted into pods default to short-lived from 1.21. + +Configuration steps: + +1. Ensure OIDC discovery URLs do not require authentication, as detailed + [here][k8s-sa-issuer-discovery]: + + ```bash + kubectl create clusterrolebinding oidc-reviewer \ + --clusterrole=system:service-account-issuer-discovery \ + --group=system:unauthenticated + ``` + +1. Find the issuer URL of the cluster. + + ```bash + kubectl proxy & + ISSUER="$(curl --fail --silent --show-error 127.0.0.1:8001/.well-known/openid-configuration | jq -r '.issuer')" + + # Kill the background proxy process when you're done + kill %% + ``` + +1. Enable and configure JWT auth in Vault. + + 1. If Vault is running in Kubernetes: + + ```bash + kubectl exec vault-0 -- vault auth enable jwt + kubectl exec vault-0 -- vault write auth/jwt/config \ + oidc_discovery_url=https://kubernetes.default.svc.cluster.local \ + oidc_discovery_ca_pem=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt + ``` + + 1. Alternatively, if Vault is _not_ running in Kubernetes: + + -> **Note:** When Vault is outside the cluster, the `$ISSUER` endpoint below may + or may not be reachable. If not, you can configure JWT auth using + [`jwt_validation_pubkeys`](#using-jwt-validation-public-keys) instead. + + ```bash + vault auth enable jwt + vault write auth/jwt/config oidc_discovery_url="${ISSUER}" + ``` + +1. Configure a role and log in as detailed [below](#creating-a-role-and-logging-in). + +[k8s-sa-issuer-discovery]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-issuer-discovery + +### Using JWT validation public keys + +This method can be useful if Kubernetes' API is not reachable from Vault or if +you would like a single JWT auth mount to service multiple Kubernetes clusters +by chaining their public signing keys. + +Kubernetes cluster requirements: + +* [`ServiceAccountIssuerDiscovery`][k8s-sa-issuer-discovery] feature enabled. + * Present from 1.18, defaults to enabled from 1.20. + * This requirement can be avoided if you can access the Kubernetes master + nodes to read the public signing key directly from disk at + `/etc/kubernetes/pki/sa.pub`. In this case, you can skip the steps to + retrieve and then convert the key as it will already be in PEM format. +* Must use short-lived service account tokens when logging in. + * Tokens mounted into pods default to short-lived from 1.21. + +Configuration steps: + +1. Fetch the service account signing public key from your cluster's JWKS URI. + + ```bash + # 1. Find the issuer URL of the cluster. + kubectl proxy & + ISSUER="$(curl --fail --silent --show-error 127.0.0.1:8001/.well-known/openid-configuration | jq -r '.issuer')" + + # 2. Query the jwks_uri specified in /.well-known/openid-configuration + # NB: You may need to run this from a pod within the cluster if the $ISSUER + # URL is not available outside the cluster. + curl "$(curl --fail --silent --show-error "${ISSUER}/.well-known/openid-configuration" | jq -r '.jwks_uri')" + + # Kill the background proxy process when you're done + kill %% + ``` + +1. Convert the keys from JWK format to PEM. You can use a CLI tool or an online + converter such as [this one][jwk-to-pem]. + +1. Configure the JWT auth mount with those public keys. + + ```bash + vault write auth/jwt/config \ + jwt_validation_pubkeys="-----BEGIN PUBLIC KEY----- + MIIBIjANBgkqhkiG9... + -----END PUBLIC KEY-----","-----BEGIN PUBLIC KEY----- + MIIBIjANBgkqhkiG9... + -----END PUBLIC KEY-----" + ``` + +1. Configure a role and log in as detailed [below](#creating-a-role-and-logging-in). + +[jwk-to-pem]: https://8gwifi.org/jwkconvertfunctions.jsp + +### Creating a role and logging in + +Once your JWT auth mount is configured, you're ready to configure a role and +log in. + +1. Create a role for JWT auth that the `default` service account from the + `default` namespace can use. The audience of tokens defaults to the same as + the issuer, but it is configurable. + + ```bash + vault write auth/jwt/role/my-role \ + role_type="jwt" \ + bound_audiences="${ISSUER}" \ + user_claim="sub" \ + bound_subject="system:serviceaccount:default:default" \ + policies="default" \ + ttl="1h" + ``` + +1. Pods or other clients with access to a service account JWT can then log in. + + ```bash + vault write auth/jwt/login \ + role=my-role \ + jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token + # OR equivalent to: + curl \ + --fail \ + --request PUT \ + --header "X-Vault-Request: true" \ + --data '{"jwt":"","role":"my-role"}' \ + "${VAULT_ADDR}/v1/auth/jwt/login" + ``` + +### Specifying TTL and audience + +If you would like to specify a custom TTL or audience for service account tokens, +the following pod spec illustrates a volume mount that overrides the default +admission injected token. This is especially relevant if you are unable to +disable the [--service-account-extend-token-expiration][k8s-extended-tokens] +flag for `kube-apiserver` and want to use short TTLs. + +When using the resulting token, you will need to set `bound_audiences=vault` +when creating roles in Vault's JWT auth mount. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: nginx +spec: + # automountServiceAccountToken is redundant in this example because the + # mountPath overlapping with the default path below will already stop the + # default admission injected token from being created. Use this option if you + # choose a different mount path. + automountServiceAccountToken: false + containers: + - name: nginx + image: nginx + volumeMounts: + - name: custom-token + mountPath: /var/run/secrets/kubernetes.io/serviceaccount + volumes: + - name: custom-token + projected: + defaultMode: 420 + sources: + - serviceAccountToken: + path: token + expirationSeconds: 600 # 10 minutes is the minimum TTL + audience: vault + - configMap: + name: kube-root-ca.crt + items: + - key: ca.crt + path: ca.crt + - downwardAPI: + items: + - fieldRef: + apiVersion: v1 + fieldPath: metadata.namespace + path: namespace +``` + +[k8s-extended-tokens]: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/#options + ## Okta 1. Make sure an Authorization Server has been created. The "Issuer" field shown on the Setting page diff --git a/website/content/docs/auth/kubernetes.mdx b/website/content/docs/auth/kubernetes.mdx index b3b4519398de..ad377d3ca192 100644 --- a/website/content/docs/auth/kubernetes.mdx +++ b/website/content/docs/auth/kubernetes.mdx @@ -12,6 +12,16 @@ The `kubernetes` auth method can be used to authenticate with Vault using a Kubernetes Service Account Token. This method of authentication makes it easy to introduce a Vault token into a Kubernetes Pod. +You can also use a Kubernetes Service Account Token to [log in via JWT auth][k8s-jwt-auth]. +See the section on [How to work with short-lived Kubernetes tokens][short-lived-tokens] +for a summary of why you might want to use JWT auth instead and how it compares to +Kubernetes auth. + +-> **Note:** If you are upgrading to Kubernetes v1.21+, ensure the config option +`disable_iss_validation` is set to true. Assuming the default mount path, you +can check with `vault read -field disable_iss_validation auth/kubernetes/config`. +See [Kubernetes 1.21](#kubernetes-1.21) below for more details. + ## Authentication ### Via the CLI @@ -62,48 +72,157 @@ Auth methods must be configured in advance before users or machines can authenticate. These steps are usually completed by an operator or configuration management tool. -1. Enable the Kubernetes auth method: +1. Enable the Kubernetes auth method: + + ```bash + vault auth enable kubernetes + ``` + +1. Use the `/config` endpoint to configure Vault to talk to Kubernetes. Use + `kubectl cluster-info` to validate the Kubernetes host address and TCP port. + For the list of available configuration options, please see the + [API documentation](/api/auth/kubernetes). + + ```bash + vault write auth/kubernetes/config \ + token_reviewer_jwt="" \ + kubernetes_host=https://192.168.99.100: \ + kubernetes_ca_cert=@ca.crt + ``` + + !> **Note:** The pattern Vault uses to authenticate Pods depends on sharing + the JWT token over the network. Given the [security model of + Vault](/docs/internals/security), this is allowable because Vault is + part of the trusted compute base. In general, Kubernetes applications should + **not** share this JWT with other applications, as it allows API calls to be + made on behalf of the Pod and can result in unintended access being granted + to 3rd parties. + +1. Create a named role: + + ```text + vault write auth/kubernetes/role/demo \ + bound_service_account_names=vault-auth \ + bound_service_account_namespaces=default \ + policies=default \ + ttl=1h + ``` + + This role authorizes the "vault-auth" service account in the default + namespace and it gives it the default policy. + + For the complete list of configuration options, please see the [API + documentation](/api/auth/kubernetes). + +## Kubernetes 1.21 + +Starting in version [1.21][k8s-1.21-changelog], the Kubernetes +`BoundServiceAccountTokenVolume` feature defaults to enabled. This changes the +JWT token mounted into containers by default in two ways that are important for +Kubernetes auth: + +* It has an expiry time and is bound to the lifetime of the pod and service account. +* The value of the JWT's `"iss"` claim depends on the cluster's configuration. + +The changes to token lifetime are important when configuring the +[`token_reviewer_jwt`](/api-docs/auth/kubernetes#token_reviewer_jwt) option. You +must avoid using a short-lived token because Vault stores that token in Vault +storage and does not automatically refresh it. If a short-lived token is used, +Kubernetes will revoke it as soon as the pod or service account are deleted, or +if the expiry time passes, and Vault will no longer be able to use the +`TokenReview` API. See [How to work with short-lived Kubernetes tokens][short-lived-tokens] +below for details on handling this change. + +In response to the issuer changes, Kubernetes auth has been updated in Vault +1.9.0 to not validate the issuer by default. The Kubernetes API does the same +validation when reviewing tokens, so enabling issuer validation on the Vault +side is duplicated work. Without disabling Vault's issuer validation, it is not +possible for a single Kubernetes auth configuration to work for default mounted +pod tokens with both Kubernetes 1.20 and 1.21. Note that auth mounts created +before Vault 1.9 will maintain the old default, and you will need to explicitly +set `disable_iss_validation=true` before upgrading Kubernetes to 1.21. See +[Discovering the service account `issuer`](#discovering-the-service-account-issuer) +below for guidance if you wish to enable issuer validation in Vault. + +[k8s-1.21-changelog]: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#api-change-2 +[short-lived-tokens]: #how-to-work-with-short-lived-kubernetes-tokens + +### How to work with short-lived Kubernetes tokens + +There are a few different ways to configure auth for Kubernetes pods when +default mounted pod tokens are short-lived, each with their own tradeoffs. This +table summarizes the options, each of which is explained in more detail below. + +| Option | All tokens are short-lived | Can revoke tokens early | Other considerations | +| ------------------------------------ | -------------------------- | ----------------------- | -------------------- | +| Use client JWT as reviewer JWT | Yes | Yes | Operational overhead | +| Use long-lived token as reviewer JWT | No | Yes | | +| Use JWT auth instead | Yes | No | | + +-> **Note:** By default, Kubernetes currently extends the lifetime of admission +injected service account tokens to a year to help smooth the transition to +short-lived tokens. If you would like to disable this, set +[--service-account-extend-token-expiration=false][k8s-extended-tokens] for +`kube-apiserver` or specify your own `serviceAccountToken` volume mount. See +[here](/docs/auth/jwt/oidc_providers#specifying-ttl-and-audience) for an example. + +[k8s-extended-tokens]: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/#options + +#### Use the Vault client's JWT as the reviewer JWT + +When configuring Kubernetes auth, you can omit the `token_reviewer_jwt`, and Vault +will use the Vault client's JWT as its own auth token when communicating with +the Kubernetes `TokenReview` API. If Vault is running in Kubernetes, you also need +to set `disable_local_ca_jwt=true`. + +This means Vault does not store any JWTs and allows you to use short-lived tokens +everywhere but adds some operational overhead to maintain the cluster role +bindings on the set of service accounts you want to be able to authenticate with +Vault. Each client of Vault would need the `system:auth-delegator` ClusterRole: - ```text - $ vault auth enable kubernetes - ``` +```bash +kubectl create clusterrolebinding vault-client-auth-delegator \ + --clusterrole=system:auth-delegator \ + --group=group1 \ + --serviceaccount=default:svcaccount1 \ + ... +``` -1. Use the `/config` endpoint to configure Vault to talk to Kubernetes. Use `kubectl cluster-info` to validate the Kubernetes host address and TCP port. Kubernetes 1.21+ clusters may require setting the service account `issuer`, [as described here](/docs/auth/kubernetes#discovering-the-service-account-issuer). For the list of available configuration options, please see the [API documentation](/api/auth/kubernetes). +#### Continue using long-lived tokens - ```text - $ vault write auth/kubernetes/config \ - token_reviewer_jwt="" \ - kubernetes_host=https://192.168.99.100: \ - kubernetes_ca_cert=@ca.crt - ``` +The default Kubernetes secret created for a service account is still long lived, +and can be used as the `token_reviewer_jwt` without needing to refresh it. To +find the secret, run: - !> **NOTE:** The pattern Vault uses to authenticate Pods depends on sharing - the JWT token over the network. Given the [security model of - Vault](/docs/internals/security), this is allowable because Vault is - part of the trusted compute base. In general, Kubernetes applications should - **not** share this JWT with other applications, as it allows API calls to be - made on behalf of the Pod and can result in unintended access being granted - to 3rd parties. +```bash +kubectl get secret "$(kubectl get serviceaccount default -o jsonpath='{.secrets[0].name}')" +``` + +Using this maintains previous workflows but does not fully take advantage of the +new default short-lived tokens. -1. Create a named role: +#### Use JWT auth - ```text - vault write auth/kubernetes/role/demo \ - bound_service_account_names=vault-auth \ - bound_service_account_namespaces=default \ - policies=default \ - ttl=1h - ``` +Kubernetes auth is specialized to use Kubernetes' `TokenReview` API. However, the +JWT tokens Kubernetes generates can also be verified using Kubernetes as an OIDC +provider. The JWT auth method documentation has [instructions][k8s-jwt-auth] for +setting up JWT auth with Kubernetes as the OIDC provider. - This role authorizes the "vault-auth" service account in the default - namespace and it gives it the default policy. +[k8s-jwt-auth]: /docs/auth/jwt/oidc_providers#kubernetes - For the complete list of configuration options, please see the [API - documentation](/api/auth/kubernetes). +This solution allows you to use short-lived tokens for all clients and removes +the need for a reviewer JWT. However, the client tokens cannot be revoked before +their TTL expires, so it is recommended to keep the TTL short with that +limitation in mind. ### Discovering the service account `issuer` --> **Deprecated:** The `issuer` parameter has been deprecated as of Vault 1.9 and will be removed in a future release. +-> **Note:** From Vault 1.9.0, `disable_iss_validation` and `issuer` are deprecated +and the default for `disable_iss_validation` has changed to `true` for new +Kubernetes auth mounts. The following section only applies if you have set +`disable_iss_validation=false` or created your mount before 1.9 with the default +value, but `disable_iss_validation=true` is the new recommended value for all +versions of Vault. Kubernetes 1.21+ clusters may require setting the service account [`issuer`](/api-docs/auth/kubernetes#issuer) to the same value as @@ -122,6 +241,9 @@ curl --silent http://127.0.0.1:8001/api/v1/namespaces/default/serviceaccounts/de | jq -r '.status.token' \ | cut -d . -f2 \ | base64 -D + +# Kill the background proxy process when you're done +kill %% ``` Most clusters will also have that information available at the @@ -130,15 +252,16 @@ Most clusters will also have that information available at the ```bash kubectl proxy & curl --silent http://127.0.0.1:8001/.well-known/openid-configuration | jq -r .issuer + +# Kill the background proxy process when you're done +kill %% ``` This value is then used when configuring Kubernetes auth, e.g.: ```bash vault write auth/kubernetes/config \ - token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \ - kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \ issuer="\"test-aks-cluster-dns-d6cbb78e.hcp.uksouth.azmk8s.io\"" ``` @@ -146,9 +269,8 @@ vault write auth/kubernetes/config \ This auth method accesses the [Kubernetes TokenReview API][k8s-tokenreview] to validate the provided JWT is still valid. Kubernetes should be running with -`--service-account-lookup`. This is defaulted to true in Kubernetes 1.7, but any -versions prior should ensure the Kubernetes API server is started with this -setting. Otherwise deleted tokens in Kubernetes will not be properly revoked and +`--service-account-lookup`. This is defaulted to true from Kubernetes 1.7. +Otherwise deleted tokens in Kubernetes will not be properly revoked and will be able to authenticate to this auth method. Service Accounts used in this auth method will need to have access to the @@ -177,7 +299,7 @@ subjects: The Kubernetes Auth Plugin has a full HTTP API. Please see the [API docs](/api/auth/kubernetes) for more details. -[k8s-tokenreview]: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#tokenreview-v1beta1-authentication-k8s-io +[k8s-tokenreview]: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#tokenreview-v1-authentication-k8s-io ## Code Example