Authentication failures Hashicorp Vault on Kubernetes 1.21

[sjoel]

Describe the bug
ARO (Azure Redhat Openshift) cluster is upgraded from 4.7 to 4.8 version. However, we see issue with pods stuck in "init 0/1" state indicating connection errors.
Error from server (BadRequest): container "anm" in pod "anm-xxx-xxx" is waiting to start: PodInitializing

• claim "iss" is invalid" backoff=2.160920171
  2022-02-01T09:29:24.556Z [INFO] auth.handler: authenticating
  2022-02-01T09:29:24.564Z [ERROR] auth.handler: error authenticating: error="Error making API request.

Vault pod logs show:
[INFO] expiration: revoked lease:
lease_id=auth/kubernetes/login

To Reproduce
Steps to reproduce the behavior:

1. Run Hashicorp on Kubernetes 1.21
   We are referring to articles that say when kubernetes is upgraded to >= 1.21, there are issues with the vault and it needs a repair https://particule.io/en/blog/vault-1.21/

Expected behavior
All application pods should be running instead they are stuck in init 0/1 state after the kubernetes (ARO) cluster upgrade

Environment:
Vault Server Version : HA Mode active
Vault CLI Version : 1.42
Server Operating System/Architecture: Vault Pods deployed in Openshift

Vault server configuration file(s):
Will share the vault yaml config files to the Developer once issue is looked at.

Please suggest on the next actions.

[NLRemco]

@sjoel
Hi! Good that you are reaching out. Luckily enough this has been looked in fairly recently and documented on the Vault website. Multiple implementations are now available and I will list some of them.

Depending on whether you work with an internal or external Vault cluster either refer to:

1. (If working with an internal cluster) The documentation for working with the newly introduced bound service account token volume (introduced through beta in Kubernetes 1.21). On top of this you would have to upgrade to the latest version of Vault to have the latest Kubernetes auth plugin which supports dynamically reloading short-lived tokens for internal clusters.
2. (For both internal and external clusters) The option disable_iss_validation to disable local issuer validation (mentioned in the article).
   This option is enabled by default since this commit and deprecated since v0.7.1 (included in Vault v1.9.0) of the Kubernetes auth plugin. With this option your cluster can still use secret-based service account tokens.
   Keep in mind that this option is mostly realizable when working with an external cluster.
3. (For both internal and external clusters) Personally I would recommend removing the token_reviewer_jwt (part of the documentation), using disable_iss_validation to disable local issuer validation, and using a token that is capable of acting on behalf of Vault when validating the sent-in service account token while upgrading an external cluster.

Since you are upgrading it might be hard to fulfil the first option. In that case to fulfil this option downgrading your Openshift cluster might be your last resort, keep this in mind.
If downgrading is not an option, I would attempt the second or third option in the list.

Next to this: the article linked is no longer completely up-to-date and even holds bad practices in my opinion (such as recommending against disabling local issuer validation). For future references always troubleshoot the Vault documentation prior to using external sources.

I hope this fixes your issue!

[sjoel]

Thanks Remco [RemcoBuddelmeijer] for the above info.. This helped us a lot!!

We were able to fix the issue and did not downgrade our cluster.
Followed the guide that you have shared - https://www.vaultproject.io/docs/auth/kubernetes#kubernetes-1-21
Added the additional config to Vault using the vault CLI

$vault write auth/kubernetes/config
issuer=""
token_reviewer_jwt=""
kubernetes_host=https://IP:<your TCP port or blank for 443>
kubernetes_ca_cert=@ca.crt
disable_iss_validation=true

And verified if issuer is updated
$vault read auth/kubernetes/config

Tada! It worked!

Thanks again for the quick response.

[kalafut]

Glad you got it working. Thanks for the assistance @RemcoBuddelmeijer!