bug: all requests on the upstream return 500 When the one node domain dns resolution timeout , even if the other node is ok

[redynasc]

Issue description
all requests on the upstream return 500 When the one node’s domain dns resolution timeout , even if the other node is ok

Environment

apisix version : 1.5
OS: Linux d1-beta-node-0004 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
OpenResty / Nginx version: (cmd: nginx -V or openresty -V)
nginx version: openresty/1.17.8.2
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)
built with OpenSSL 1.1.1g 21 Apr 2020
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DNGX_MAX_ERROR_STR=49152 -I/usr/local/openresty/pcre/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.17 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.8 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/pcre/lib -Wl,-rpath,/usr/local/openresty/pcre/lib' --with-openssl=/tmp/openssl-1.1.1g --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-md5-asm --with-sha1-asm --with-pcre-jit --with-stream --with-stream_ssl_module --with-threads --with-openssl-opt=-g --with-stream --with-stream_ssl_preread_module

Minimal test code / Steps to reproduce the issue

1.Create an upstream, like this
"nodes":{
"domain1-ok:8080":100,
"domain2-ok:8080":100,
"domain3-ok:8080":100
}

2. then requests return 500

What's the actual result? (including assertion message & call stack if applicable)
one node DNS timeout ,all requests return 500

What's the expected result?
one node DNS timeout ,the request should be forwarded to other node

[spacewander]

@membphis
Maybe we can ignore the failed DNS response?

[membphis]

Maybe we can ignore the failed DNS response?

agree, and need to write an error log for this case.

if there is no healthy node, we should return 503 for this case.

[spacewander]

We returns 502 when no endpoint can be selected, maybe returns 502 for this case is better?

[membphis]

502 means the upstream is running but cannot respond to requests normally
503 means the upstream can not working (stoped or restarting)

so I think we should use 503 if there is no healthy upstream node.

[spacewander]

@redynasc
Would you contribute a PR for it?

[spacewander]

Solved by #3398