Proposal: APISIX Supports Secrets Management

[soulbird]

Background

Secrets refer to any sensitive information required during the running process of APISIX, which may be part of the core configuration (such as the etcd's password) or some sensitive information in the plugin. Common types of Secrets in APISIX include:

• username, the password for some components (etcd, Redis, Kafka, etc.)
• the private key of the certificate
• API key
• Sensitive plugin configuration fields, typically used for authentication, hashing, signing, or encryption

Secrets Management refers to allowing users to store Secrets in APISIX through some secrets management services (vault, etc.), and read them according to the key when using them to ensure that Secrets do not exist in plaintext in the entire platform.

Currently, APISIX only supports the use of vault to manage secrets in the jwt-auth plugin. We plan to use more secrets management services in more plugins.

Scheme

Overall Design

In order to conveniently use the Secrets Management capability in various plugins of APISIX to protect secrets or other sensitive data, we will implement the following steps:

1. Add the environment variable module to support referencing environment variables in plugins and other APISIX resources in a specific way
2. Design a general KMS component, compatible with different secrets management services (vault, aws, etc.), and support referring to the KMS component in a specific way in the plugin
3. Add a new KMS resource in APISIX to configure and manage KMS components
4. Use a "non-intrusive" way to reference KMS components in plugins
5. We will first use secrets Management in the authentication class plugin

It works as follows:

Detailed Design

Environment variable

APISIX loads all environment variables at startup, so there is no need to add them through custom resources. Environment variables can be referenced in the following ways:

$ENV://$env_name/$sub_key

• env_name: environment variable name
• sub_key: get the value of a property when the value of the environment variable is a JSON string

If the value of the environment variable is of type string, such as:

export JACK_JWT_KEY=abc

It can be referenced as follows:

$ENV://JACK_JWT_KEY

If the value of the environment variable is a JSON string like:

export JACK={"jwt-key":"abc","openid-key": "def"}

It can be referenced as follows:

# Get the jwt-key of the environment variable JACK
$ENV://JACK/jwt-key
# Get the openid-key of the environment variable JACK
$ENV://JACK/openid-key

Example: use in jwt-auth plugin

First, create environment variables before the APISIX instance starts

export JACK_JWT_KEY=abc

Reference environment variables in the jwt-auth plugin

curl http://127.0.0.1:9180/apisix/admin/consumers \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "username": "jack",
    "plugins": {
        "jwt-auth": {
            "key": "user-key",
            "secret": "$ENV://JACK_JWT_KEY"
        }
    }
}'

Through the steps shown above, you can save the secret configuration in the jwt-auth plugin in an environment variable instead of displaying it in plaintext when configuring the plugin.

KMS Component Design

Core Points

1. In order to facilitate the management of KMS components, combined with the characteristics of APISIX itself, KMS resources are introduced to configure KMS components, and configuration files are also supported to configure KMS components.
2. The KMS component is the encapsulation of the specific secrets management components (env, vault, etc.), and the specific secrets management components are not exposed to the outside world
3. The KMS component exposes the get operation, which must meet the following core functions:

• Parse a string similar to $KMS://vault/jwt/secret_id/jwt to get the secrets management service object and key
• Call the secrets management service to obtain the secret corresponding to the key
• The secret is loaded lazily and supports caching for a certain period of time. This can not only avoid stress on secrets management services but also be compatible with the "secret rotation" function of some secrets management services

4. The KMS component does not provide write operations to the secrets management service, as this may expose the user's secret on APISIX.
5. The secret support required by each secrets management service in the KMS component is configured through environment variables.
6. The communication method between the KMS component and the secrets management service is determined by the final selected Lua library, and we do not care about the specific implementation protocol (HTTP or gRPC).

Reference Method

In the plugin, KMS components can be referenced through special variables in the following format:

$KMS://$secretmanager/$id/$secret_id/$key

• secretmanager: secrets management service, could be the vault, aws, etc.
• id: KMS resource id, which needs to be consistent with the one specified when adding the KMS resource
• secret_id: the secret id in the secrets management service
• key: the key corresponding to the secret in the secrets management service

Example: use in jwt-auth plugin

First, create the corresponding secret in the vault. You can use the following command:

vault kv put apisix/jack jwt-key=value

Next, you can configure APISIX through the following steps:
Step 1: Add KMS resources through the Admin API, and configure the connection information such as the address of the vault:

curl http://127.0.0.1:9180/apisix/admin/kms/vault/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri": "https://127.0.0.1:8200"，
    "prefix": "apisix",
    "token": "root"
}'

Step 2: Refer to the KMS resource in the jwt-auth plugin and fill in the secret information

curl http://127.0.0.1:9180/apisix/admin/consumers \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "username": "jack",
    "plugins": {
        "jwt-auth": {
            "key": "user-key",
            "secret": "$KMS://vault/1/jack/jwt-key"
        }
    }
}'

Through the above two steps, when the user request hits the jwt-auth plugin, the user-configured secrets management service will be called through the interface provided by the KMS component to obtain the real value of the secret in the vault.
In addition, the token required for APISIX to interact with the vault can also be stored in the environment variable. Before APISIX starts, the environment variable can be set by the following command:

export VAULT_TOKEN="root"

Reference the environment variable when adding the KMS resource:

curl http://127.0.0.1:9180/apisix/admin/kms/vault/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri": "https://127.0.0.1:8200"，
    "prefix": "apisix",
    "token": "$ENV://VAULT_TOKEN"
}'

Through the above steps, you can configure the secret in the jwt-auth plugin to the vault instead of displaying it in plaintext when configuring the plugin. Among them, the token used by APISIX to connect to the vault can also be saved in the environment variable.

KMS Resource Admin API Design

When adding KMS resources through the Admin API, the request is designed as follows:

Request URI

http://127.0.0.1:9180/apisix/admin/kms/${secretmanager}/${id}

• secretmanager: secrets management service, currently only vault, more secrets management services can be expanded in the future
• id: resource id

• APISIX can obtain all environment variables at startup, so users can directly refer to it without adding additional environment variable resources.

Request Method

Method	URI	Description
GET	apisix/admin/kms	Get a list of KMS resources
GET	apisix/admin/kms/${secretmanager}/${id}	Get the specified KMS resource
PUT	apisix/admin/kms/${secretmanager}/${id}	Create KMS resource based on id
DELETE	apisix/admin/kms/${secretmanager}/${id}	Delete the specified KMS resource
PATCH	apisix/admin/kms/${secretmanager}/${id}	Modify the attributes in the specified KMS resource, the attributes not involved in the request will be kept as they are
PATCH	apisix/admin/kms/${secretmanager}/${id}/${path}	SubPath PATCH， Specify the attribute to be updated by KMS through the path, update the data of the attribute in full, and other attributes that are not involved will remain as they are

• Note: According to the current design of APISIX, the POST method will automatically generate an id. When referring to a KMS component, an id needs to be used. The automatically generated id is not easy to manage and use, so the POST method is not provided.

Request Body

The secrets management service currently only supports vaults, so the request URI can currently only be:

http://127.0.0.1:9180/apisix/admin/kms/vault/${id}

The definitions of the fields in the corresponding request body are as follows:

Name	Optional	Type	Description	Example
uri	No	string	Address of vault	https://127.0.0.1:8200
prefix	No	string	The prefix for storing secrets in the vault. The -path parameter specified when enabling the vault engine with vault secrets enable	apisix
token	No	string	The token to use when connecting to the vault	root

[soulbird]

We noticed that cloud vendors such as aws have Secret management service and Key management service, which are two different products. The function we implemented this time is more in line with the definition of Secret management. There is obvious ambiguity in using "KMS" here, so we will rename "KMS" to "Secret".

[Sn0rt]

Define the Data Range That Secret Manager Can Protect

The following configuration is designed to obtain values from the secert manager, which is called referenceable.

• authentication plugin
  • Basic-auth 's password
  • Key-auth key
  • ldap-auth user_dn
  • appid of Wolf-rbac
  • Hmac-auth’s access_key and secret_key
  • Jwt-auth secret
  • client_secret of authz-keycloak (need test case
  • client_secret of authz-casdoor (need test case
  • client_secret of openid-connect (need test case
  • The RBAC configuration file information of authz-casbin requires secondary confirmation.
• logger plugin
  • http-logger: auth_header (option)
  • tcp-logger: tls_options(option)
  • kafka-logger: brokers.sasl_config.user, brokers.sasl_config.password, key(option) of
  • rocketmq-logger: access_key, secret_key of
  • clickhouse-logger: user, password
  • error-log-logger: clickhouse.user, clickhouse.password,
  • error-log-logger: kafka.brokers.sasl_config.user, kafka.brokers.sasl_config.password, kafka.key
  • sls-logger: access_key_id, access_key_secret
  • google-cloud-logging: auth_config.private_key
  • loggly: customer_token
  • elasticsearch-logger: auth.password
  • tencent-cloud-cls: secret_id, secret_key
• Other plugin
  • kafka-proxy: sasl.password
• Security plugin
  • csrf: key

Sensitive data at startup needs to be protected by secret manager

more info: #10221

• config-default.yaml
  • apisix.ssl.key_encrypt_salt is used to encrypt data and does not write the data key to etcd naked.
  • apisix.data_encryption is used to encrypt data, without writing the data key to etcd naked.
  • apisix.discovery.nacos.host

[github-actions]

This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.