Can't set log level to error for vault agent

[kwyczesany]

Hi, I'm quite new to vault and k8s, but I think I found a bug, which can be reproduced even on tutorial with vault agent as sidecar link

Describe the bug
Setting vault.hashicorp.com/log-level annotation either to "err" or "error" for vault agent causes failed startup:

[ERROR] runtime error encountered: error="template server failed to runner generate config: invalid log level "ERROR", valid log levels are TRACE, DEBUG, INFO, WARN, ERR"

To Reproduce
Steps to reproduce the behavior:

1. Deploy application
2. Set vault.hashicorp.com/log-level to err in patch deployment file
3. Patch deployment with vault agent injector
4. See error in vault-agent-init logs

5. Deployment can't start new version

Application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orgchart
  labels:
    app: orgchart
spec:
  selector:
    matchLabels:
      app: orgchart
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: orgchart
    spec:
      serviceAccountName: internal-app
      containers:
        - name: orgchart
          image: jweissig/app:0.0.1

and patch with vault agent:

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/log-level: "err"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"

kubectl describe deployment <app>:

kubectl describe deployment orgchart
Name:                   orgchart
Namespace:              default
CreationTimestamp:      Mon, 22 Feb 2021 07:42:58 +0000
Labels:                 app=orgchart
Annotations:            deployment.kubernetes.io/revision: 2
Selector:               app=orgchart
Replicas:               1 desired | 1 updated | 2 total | 1 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=orgchart
  Annotations:      vault.hashicorp.com/agent-inject: true
                    vault.hashicorp.com/agent-inject-secret-database-config.txt: internal/data/database/config
                    vault.hashicorp.com/log-level: err
                    vault.hashicorp.com/role: internal-app
  Service Account:  internal-app
  Containers:
   orgchart:
    Image:        jweissig/app:0.0.1
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    ReplicaSetUpdated
OldReplicaSets:  orgchart-7f6b86f74f (1/1 replicas created)
NewReplicaSet:   orgchart-5b46cc4b5d (1/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  15m    deployment-controller  Scaled up replica set orgchart-7f6b86f74f to 1
  Normal  ScalingReplicaSet  8m41s  deployment-controller  Scaled up replica set orgchart-5b46cc4b5d to 1

kubectl describe replicaset <app>:

kubectl describe replicaset orgchart
Name:           orgchart-5b46cc4b5d
Namespace:      default
Selector:       app=orgchart,pod-template-hash=5b46cc4b5d
Labels:         app=orgchart
                pod-template-hash=5b46cc4b5d
Annotations:    deployment.kubernetes.io/desired-replicas: 1
                deployment.kubernetes.io/max-replicas: 2
                deployment.kubernetes.io/revision: 2
Controlled By:  Deployment/orgchart
Replicas:       1 current / 1 desired
Pods Status:    0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=orgchart
                    pod-template-hash=5b46cc4b5d
  Annotations:      vault.hashicorp.com/agent-inject: true
                    vault.hashicorp.com/agent-inject-secret-database-config.txt: internal/data/database/config
                    vault.hashicorp.com/log-level: err
                    vault.hashicorp.com/role: internal-app
  Service Account:  internal-app
  Containers:
   orgchart:
    Image:        jweissig/app:0.0.1
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Events:
  Type    Reason            Age    From                   Message
  ----    ------            ----   ----                   -------
  Normal  SuccessfulCreate  9m56s  replicaset-controller  Created pod: orgchart-5b46cc4b5d-q7n8d

Name:           orgchart-7f6b86f74f
Namespace:      default
Selector:       app=orgchart,pod-template-hash=7f6b86f74f
Labels:         app=orgchart
                pod-template-hash=7f6b86f74f
Annotations:    deployment.kubernetes.io/desired-replicas: 1
                deployment.kubernetes.io/max-replicas: 2
                deployment.kubernetes.io/revision: 1
Controlled By:  Deployment/orgchart
Replicas:       1 current / 1 desired
Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=orgchart
                    pod-template-hash=7f6b86f74f
  Service Account:  internal-app
  Containers:
   orgchart:
    Image:        jweissig/app:0.0.1
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Events:
  Type    Reason            Age   From                   Message
  ----    ------            ----  ----                   -------
  Normal  SuccessfulCreate  16m   replicaset-controller  Created pod: orgchart-7f6b86f74f-htflv

Expected behavior
Vault agent should set itself log level to error and start application without issues

Environment

• Kubernetes version:
  • minikube version: v1.12.0
• vault-k8s version: v1.6.2

Additional context
Other log levels works as expected

[kofl91]

encountered the same issue. err seems to be translated to error somewhere along the line and then checked against an enum that contains "ERR" but not "ERROR"

[dntosas]

same here +1

[tomhjp]

Thanks for the report, this should be fixed for Agent versions 1.10.1/1.11.0+: hashicorp/vault#14424. That corresponds to versions 0.16.0+ by default for the Agent injector, or 0.20.0+ for vault-helm