bug: The apisix data plane does not update the upstream properly when etcd recovers from the failure

[linyongluis]

Current Behavior

The apisix data plane does not update the upstream properly when etcd recovers from the failure

Expected Behavior

The apisix data plane can be updated correctly and recovered when etcd fails

Error Logs

No response

Steps to Reproduce

1、Stop etcd server
2、Start etcd server
3、PUT admin api to upate upstream node ip and update successed
4、Access the updated URL and return 502
5、Check the APISIX logs, and the route still requests the old IP address

Environment

• APISIX version (run apisix version): 3.7.0
• Operating system (run uname -a): 5.10.134-13.1.al8.x86_64 change: added doc of how to load plugin. #1 SMP Mon Feb 6 14:54:50 CST 2023 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): 1.21.4.2
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version): 5.1

[shreemaan-abhishek]

please share the steps to reproduce this issue.

[linyongluis]

please share the steps to reproduce this issue.

1、Running a apisix cluster
2、Configure a route by the admin api and access it successfully
3、Simulating etcd failures： First stop the etcd service, then delete all the data in etcd, then start etcd.
4、At this point, because apisix has not been rebooted, the previous route is still accessible to the
5、Modify the previous routing configuration (create) via the admin api: modify the node ip that corresponds to the upstream
6、Accessing this route reveals that although the configuration was successful and the data in etcd is correct, the apisix datasurface is still requesting the same node ip as before.

[bstasz]

Hello!
I am having the same issue. Tried looking into the source of the issue but no luck. Am I missing a configuration or is there a bug?
Both 3.3.0 and 3.9.1. Deployed with the Helm chart on EKS.

[JacobLi11]

mark