Howto get upstreams state

[adrimt]

Hi,
the upstream state is always 'UP' even when the backend serer is down and the nginx return code is 502.
Did I miss something ?

vhost_traffic_status_zone;
geoip_country /usr/share/GeoIP/GeoIP.dat;
vhost_traffic_status_filter_by_set_key $geoip_country_code country::*;
vhost_traffic_status_zone shared:vhost_traffic_status:32m;

server {
listen 10.10.10.1:80;
server_name $hostname;
location /status {
vhost_traffic_status_display;
vhost_traffic_status_display_format html;
access_log off;
}
}
server {
listen 10.10.10.10:80;
server_name webserver1;
location / {
proxy_pass http://192.168.10.10;
...
}
}

[u5surf]

@adrimt Thanks reporting!
The other days, we had catch the same issue #258 (comment)
Then it could resolve the upstream zone.
Can you check this setting is existed on your nginx.conf (I could not find out upstream directive above config)

[adrimt]

I change the config to define an "upstream" but it didn't behave differently.
The "check" directive is unavailable on my free nginx.

upstream web01-http {
zone backend 64k;
server 192.168.10.10:80;
}

server {
listen 10.10.10.10:80;
server_name webserver1;
location / {
proxy_pass http://web01-http;
...
}

[u5surf]

@adrimt
This module seems to display whether the upstream state is down or not to watch the peer.down when display mode is html ( json mode is slightly different behavior on that #258 (comment))

nginx-module-vts/share/status.template.html

Line 685 in 5b3812e

aHe('td', [sTh(peer.backup, peer.down), mTh(peer.responseMsec),

Moreover nginx oss doesn't change peer->down anywhere below.

% rg "peer->down" ./src
./src/stream/ngx_stream_upstream_round_robin.c
461:        if (peer->down) {
559:        if (peer->down) {

./src/stream/ngx_stream_upstream_hash_module.c
235:        if (peer->down) {
549:            if (peer->down) {

./src/stream/ngx_stream_upstream_least_conn_module.c
142:        if (peer->down) {
197:            if (peer->down) {

./src/stream/ngx_stream_upstream_random_module.c
248:        if (peer->down) {
350:        if (peer->down) {

./src/http/ngx_http_upstream_round_robin.c
452:        if (peer->down) {
550:        if (peer->down) {

./src/http/modules/ngx_http_upstream_ip_hash_module.c
206:        if (peer->down) {

./src/http/modules/ngx_http_upstream_random_module.c
248:        if (peer->down) {
350:        if (peer->down) {

./src/http/modules/ngx_http_upstream_least_conn_module.c
146:        if (peer->down) {
201:            if (peer->down) {

./src/http/modules/ngx_http_upstream_hash_module.c
236:        if (peer->down) {
548:            if (peer->down) {

We should match the behavior due to display mode both html and json.
By fixing this behavior, we can observe the upstream down to use nginx_upstream_check_module on workaround.

[u5surf]

@adrimt
Hi, We end up checking the nginx upstream spec which the peer is single via nginx-dev ML. its answer is here.
https://mailman.nginx.org/pipermail/nginx-devel/2023-March/Q6MCXGQU56HDD7K2BWRYLCUEC7VBPUV4.html

As per nginx usage of upstreams with a single peer, such peers are
never considered unavailable, and to do this failures are never
counted (or, which is equivalent, immediately forgotten). If you are trying to show status of a peer as seen by nginx, then
"up" is perfectly correct for such peers.

Thus, we could not fix the view point of peer->fails with peers->single by any nginx upstream patch.

you should look into request results (and/or upstream states), and collect relevant stats yourself.

So we should embed other request based module to collect such stats, e.g. we can use nginx_upstream_check_module on workaround.

compile with pure nginx

$ git clone https://github.com/yaoweibin/nginx_upstream_check_module
$ cd /path/to/nginx # latest
$ patch -p1 < /path/to/nginx_upstream_check_module/check_1.20.1+patch
$ ./auto/configure --add-module=/path/to/nginx-module-vts --add-module=/path/to/nginx_upstream_check_module
$ make
$ sudo make install

nginx.conf

http {
...
    vhost_traffic_status_zone;
    upstream web {
            zone backend 64k; # <- here
            check interval=5000 rise=1 fall=3 timeout=4000;
            server 127.0.0.1:8080;
    }
    server {
        listen       8000;
        location / {
            proxy_pass http://web;
        }
    }
    server {
        listen       80;
        location /status/ {
            vhost_traffic_status_display;
            vhost_traffic_status_display_format html;
        }
    }
}

The upstream peer is up

otherwise

[adrimt]

Thank you for clarifying this issue.