help request: Apisix3 decoupled mode does not automatically refresh CP data without DP

[XW512]

Description

I have set up the decoupled mode of Apisix3, but found that the DP node will not automatically refresh the modified data of the CP node. If I restart the DP node and the data can be synchronized to the DP node, is there a problem with my configuration

the DP config

apisix:
ssl:
ssl_trusted_certificate: /app/apisix-3.3.0/conf/cert/mtls_ca.crt
deployment:
role: data_plane
role_data_plane:
config_provider: control_plane
control_plane:
host:
- "https://gw-cp.intsit.local:9280"
timeout: 5
prefix: "/apisix"
admin:
admin_key:
- name: admin
key: 7b0d04aef23327824710da9ce10c55f0 # using fixed API token has security risk, please update it when you deploy to production environment
role: admin

this CP config

deployment:
role: control_plane
role_control_plane:
config_provider: etcd
conf_server:
listen: 0.0.0.0:9280
cert: /app/apisix-3.3.0/conf/cert/mtls_server.crt
cert_key: /app/apisix-3.3.0/conf/cert/mtls_server.key
admin:
admin_key:
- name: admin
key: 7b0d04aef23327824710da9ce10c55f0 # using fixed API token has security risk, please update it when you deploy to production environment
role: admin

Environment

• APISIX version (run apisix version): 3.3.0
• Operating system (run uname -a): centos8
• OpenResty / Nginx version (run openresty -V or nginx -V): 1.19.9.1
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5.9
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version): 3.8.0

[XW512]

I want to ask, is this definitely a bug? If it's a bug, why hasn't anyone else encountered it，Or is there a problem with my configuration？

[shreemaan-abhishek]

@XW512 please provide the whole config.yaml file with proper formatting thanks so that I can try to reproduce it, thanks.

[XW512]

@XW512 please provide the whole config.yaml file with proper formatting thanks so that I can try to reproduce it, thanks.

OK，thank you

this is data_plane config：

apisix:
ssl:
ssl_trusted_certificate: /app/apisix-3.3.0/t/certs/mtls_ca.crt
deployment:
role: data_plane
role_data_plane:
config_provider: control_plane
control_plane:
host:
- "https://gw-cp.intsit.cloud.local:9280"
timeout: 30
prefix: "/apisix"
admin:
admin_key:
- name: admin
key: 6b0d04aef24327864740da9ce10c55f0
role: admin

this is control_plane config:

deployment:
role: control_plane
role_control_plane:
config_provider: etcd
conf_server:
listen: 0.0.0.0:9280
cert: /app/apisix-3.3.0/t/certs/mtls_server.crt
cert_key: /app/apisix-3.3.0/t/certs/mtls_server.key
admin:
admin_key:
- name: admin
key: 6b0d04aef24327864740da9ce10c55f0
role: admin

producer mtls from https://apisix.apache.org/zh/docs/apisix/tutorials/client-to-apisix-mtls/

[shreemaan-abhishek]

@XW512, please paste your config.yaml file with proper formatting here. 🤦🏼‍♂️ 🤦🏼‍♂️

[XW512]

@XW512, please paste your config.yaml file with proper formatting here. 🤦🏼‍♂️ 🤦🏼‍♂️

OK
this is data_plane config.yaml

apisix:
ssl:
ssl_trusted_certificate: /app/apisix-3.3.0/t/certs/mtls_ca.crt
deployment:
role: data_plane
role_data_plane:
config_provider: control_plane
control_plane:
host:
- "https://gw-cp.intsit.cloud.local:9280"
timeout: 30
prefix: "/apisix"
#certs:
#cert: /app/apisix-3.3.0/conf/cert/mtls_client.crt
#cert_key: /app/apisix-3.3.0/conf/cert/mtls_client.key
admin:
admin_key:
- name: admin
key: 6b0d04aef24327864740da9ce10c55f0
role: admin

this is control_plane config.yaml

deployment:
role: control_plane
role_control_plane:
config_provider: etcd
conf_server:
listen: 0.0.0.0:9280
cert: /app/apisix-3.3.0/t/certs/mtls_server.crt
cert_key: /app/apisix-3.3.0/t/certs/mtls_server.key
admin:
admin_key:
- name: admin
key: 6b0d04aef24327864740da9ce10c55f0
role: admin

producer mtls from https://apisix.apache.org/zh/docs/apisix/tutorials/client-to-apisix-mtls/

[kingluo]

The conf server is removed, so the etcd communication rolls back to what 2.x versions behave like, i.e. no matter CP or DP, connects to etcd directly. Closed by #10012

[XW512]

The conf server is removed, so the etcd communication rolls back to what 2.x versions behave like, i.e. no matter CP or DP, connects to etcd directly. Closed by #10012

thank you，I would like to ask how to configure the Decoupled mode for DP and CP isolation in version 3.5.0. I see that the configuration method after conf server is removed is not described in the document 3.5.0

[kingluo]

The conf server is removed, so the etcd communication rolls back to what 2.x versions behave like, i.e. no matter CP or DP, connects to etcd directly. Closed by #10012

thank you，I would like to ask how to configure the Decoupled mode for DP and CP isolation in version 3.5.0. I see that the configuration method after conf server is removed is not described in the document 3.5.0

No, we have changed the doc:

apisix/docs/en/latest/deployment-modes.md

Line 68 in c757baf

## Decoupled

As well as the conf:

apisix/conf/config-default.yaml

Lines 616 to 625 in c757baf

	deployment: # Deployment configurations
	role: traditional # Set deployment mode: traditional, control_plane, or data_plane.
	role_traditional:
	config_provider: etcd # Set the configuration center.
	#role_data_plane: # Set data plane details if role is data_plane.
	# config_provider: etcd # Set the configuration center: etcd, xds, or yaml.
	#role_control_plane: # Set control plane details if role is control_plane.
	# config_provider: etcd # Set the configuration center.

The configuration of etcd, no matter which deployment mode, is configured just like 2.x version, except the role field.

[XW512]

The conf server is removed, so the etcd communication rolls back to what 2.x versions behave like, i.e. no matter CP or DP, connects to etcd directly. Closed by #10012

thank you，I would like to ask how to configure the Decoupled mode for DP and CP isolation in version 3.5.0. I see that the configuration method after conf server is removed is not described in the document 3.5.0

No, we have changed the doc:

apisix/docs/en/latest/deployment-modes.md

Line 68 in c757baf

## Decoupled

As well as the conf:

apisix/conf/config-default.yaml

Lines 616 to 625 in c757baf

	deployment: # Deployment configurations
	role: traditional # Set deployment mode: traditional, control_plane, or data_plane.
	role_traditional:
	config_provider: etcd # Set the configuration center.
	#role_data_plane: # Set data plane details if role is data_plane.
	# config_provider: etcd # Set the configuration center: etcd, xds, or yaml.
	#role_control_plane: # Set control plane details if role is control_plane.
	# config_provider: etcd # Set the configuration center.

The configuration of etcd, no matter which deployment mode, is configured just like 2.x version, except the role field.

thank you, I tried using the method you mentioned above and found that DP can get up, while CP cannot. The error is as follows:
invalid deployment control_plane configuration: property "role_control_plane" validation failed: property "conf_server" is required

The configuration information of my CP is as follows:

deployment:
role: control_plane
role_control_plane:
config_provider: etcd

I added information and found that it is necessary to configure the path related to the certificate, otherwise an error will be reported

deployment:
role: control_plane
role_control_plane:
config_provider: etcd
conf_server:
listen: 0.0.0.0:9280
cert: /path/to/server.crt
cert_key: /path/to/server.key
client_ca_cert: /path/to/ca.crt

[XW512]

Is this modification not in version 3.5.0, but in the master branch？When is the next release expected to be released

[kingluo]

Is this modification not in version 3.5.0, but in the master branch？When is the next release expected to be released

Yes, it's only on the master branch. It'll be available in 3.6 or backport to 3.5 if necessary.

[XW512]

Is this modification not in version 3.5.0, but in the master branch？When is the next release expected to be released

Yes, it's only on the master branch. It'll be available in 3.6 or backport to 3.5 if necessary.

I would like to try this new mode now. Could you please create a release version first? Thank you very much

[kingluo]

You could try it from source code, but yes, it's a bit tough job.
So maybe you could use the release or dev version of the docker image:
https://hub.docker.com/r/apache/apisix/tags
Here is an example:
https://gist.github.com/kingluo/addf27f7dce5f9133043e8f1e5ab3a6a

[XW512]

You could try it from source code, but yes, it's a bit tough job. So maybe you could use the release or dev version of the docker image: https://hub.docker.com/r/apache/apisix/tags Here is an example: https://gist.github.com/kingluo/addf27f7dce5f9133043e8f1e5ab3a6a

Thank you very much. I will give it a try