Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

upstream status not correct #258

Closed
FrancisAston opened this issue Jan 13, 2023 · 5 comments
Closed

upstream status not correct #258

FrancisAston opened this issue Jan 13, 2023 · 5 comments

Comments

@FrancisAston
Copy link

FrancisAston commented Jan 13, 2023

nginx version: openresty/1.19.9.1
nginx-module-vts-0.2.1/

server alread down,but the status page still up

nginx_upstream_check_module page
image

nginx-module-vts status page
image

@u5surf
Copy link
Collaborator

u5surf commented Jan 14, 2023

@FrancisAston Hi, Thanks reporting!!
Can you try once to change display format json? Because the State is changed by upstream_server_node.down, it changes in below code, but it might not to call when format is html.

#if (NGX_HTTP_UPSTREAM_CHECK)
if (ngx_http_upstream_check_peer_down(peer->check_index)) {
usn.down = 1;
} else {
usn.down = 0;
}
#else
usn.down = (peer->fails >= peer->max_fails || peer->down);
#endif
#if nginx_version > 1007001
usn.name = peer->name;
#endif

I still don’t make sense obviously whether it is bug or not but it worth for diving deeply for this. I’m willing to investigate that.

@FrancisAston
Copy link
Author

the 127.0.0.1:8540 is down , response json data: "down": false
I'm not a developer,Please help. thank a lot.
"ali-oss": [ {
"server": "127.0.0.1:8540",
"requestCounter": 0,
"inBytes": 0,
"outBytes": 0,
"responses": {
"1xx": 0,
"2xx": 0,
"3xx": 0,
"4xx": 0,
"5xx": 0
},
"requestMsecCounter": 0,
"requestMsec": 0,
"requestMsecs": {
"times": [],
"msecs": []
},
"requestBuckets": {
"msecs": [],
"counters": []
},
"responseMsecCounter": 0,
"responseMsec": 0,
"responseMsecs": {
"times": [],
"msecs": []
},
"responseBuckets": {
"msecs": [],
"counters": []
},
"weight": 1,
"maxFails": 3,
"failTimeout": 5,
"backup": false,
"down": false,
"overCounts": {
"maxIntegerSize": 18446744073709551615,
"requestCounter": 0,
"inBytes": 0,
"outBytes": 0,
"1xx": 0,
"2xx": 0,
"3xx": 0,
"4xx": 0,
"5xx": 0,
"requestMsecCounter": 0,
"responseMsecCounter": 0
}
}],

@u5surf
Copy link
Collaborator

u5surf commented Jan 14, 2023

@FrancisAston
Is this the below module which you use vts module together?
https://github.com/yaoweibin/nginx_upstream_check_module
And we appreciate if you can give us the nginx.conf which can reproduce this behavior.

@u5surf
Copy link
Collaborator

u5surf commented Jan 14, 2023

@FrancisAston
I've investigated, I seem to found the solution of this issue.
Can you check your nginx.conf whether upstream zone directive is active or not.
e.g.

http {
...
    vhost_traffic_status_zone;
    upstream web {
            zone backend 64k; # <- here
            check interval=5000 rise=1 fall=3 timeout=4000;
            server 127.0.0.1:9000;
    }
    server {
        listen       8000;
        location / {
            proxy_pass http://web;
        }
    }
    server {
        listen       80;
        location /status/ {
            vhost_traffic_status_display;
            vhost_traffic_status_display_format json; #prometheus;
        }
    }
}

c.f. http://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone

if the directive is inactive, the upstream alive status check which I referred the comment in #258 (comment) can't pass for the goto is exist before.

if (uscf->shm_zone == NULL) {
goto not_supported;
}

As a test we activate the zone directive, the status down can return true in case of upstream is dead.

result

% curl -s http://localhost/status/ | jq '.upstreamZones.web[0].down'
true

gdb info:

Breakpoint 1, ngx_http_vhost_traffic_status_display_set_upstream_group (r=0x558bb9bb7180, buf=0x558bb9c094fe "") at ../ng
inx-module-vts/src/ngx_http_vhost_traffic_status_display_json.c:573
573                 if (uscf->shm_zone == NULL) {
(gdb) p uscf->shm_zone
$1 = (ngx_shm_zone_t *) 0x558bb9bc4320
(gdb) c

@FrancisAston
Copy link
Author

http {
...
vhost_traffic_status_zone;
upstream web {
zone backend 64k; # <- here
check interval=5000 rise=1 fall=3 timeout=4000;
server 127.0.0.1:9000;
}

add zone backend 64k; and reload nginx.
it work. thinks 。

u5surf added a commit to u5surf/nginx-module-vts that referenced this issue Mar 12, 2023
u5surf added a commit to u5surf/nginx-module-vts that referenced this issue Mar 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants