shm_add_upstream #228

hailiangchen · 2022-03-17T09:10:14Z

[error] 18811#0: *7019257 shm_add_upstream::peer failed while logging request,
[error] 18811#0: *7019257 handler::shm_add_upstream() failed while logging request,

The above two errors are logged on all requests

nginx version : nginx/1.18.0

upstream web{
dynamic_resolve fallback=stale fail_timeout=5s;
server web.xxxxx.local;
keepalive 200;
}

server {
listen 443 ssl;
server_name web.xxxxx.com;
resolver 10.200.0.10;
location / {
include proxy.conf;
proxy_pass http://web;
}
}

vozlt · 2022-09-05T16:19:34Z

@hailiangchen
It's a log that failed to get upstream information. It's not exact, but it looks like it could happen if you're using upstream changes using shared memory. Are you using upstream with dynamic settings?

hailiangchen · 2022-10-11T02:06:33Z

@hailiangchen It's a log that failed to get upstream information. It's not exact, but it looks like it could happen if you're using upstream changes using shared memory. Are you using upstream with dynamic settings?

Only dynamic_resolve is used , not shared memory

u5surf · 2022-10-15T23:58:28Z

@hailiangchen Hi, thanks for your report. At first, are you use the customized nginx such as tengine? As far as I know, the dynamic_resolve directive has defined as the tengine's original module which has been used in customized ngx_http_upstream_connect. I suspect that it may be factor that customized function get weird behavior.

@vozlt
BTW, Should we support the tengine which includes the customized upstream_connect function?

vozlt · 2022-10-17T15:00:29Z

@u5surf
If it is currently under maintenance and actively used, I think it need to find the problematic part.

u5surf · 2022-10-18T08:54:14Z

@hailiangchen Hi,
I could reproduce the issue by the minimum nginx.conf and on the tengine latest master package.

my nginx.conf

worker_processes  1;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    vhost_traffic_status_zone;
    upstream web {
            dynamic_resolve fallback=stale fail_timeout=5s;
            server www.yahoo.com;
            #keepalive 200;
    }
    server {
        listen       8000;
        location / {
            #root   html;
            #index  index.html index.htm;
            proxy_pass http://web;
        }
    }
    server {
        listen       80;
        gzip off;
        client_max_body_size 100M;
        location /status/ {
            vhost_traffic_status_display;
            vhost_traffic_status_display_format prometheus;
        }
    }
}

my nginx -V

% /usr/local/nginx/sbin/nginx -V
Tengine version: Tengine/2.3.3
nginx version: nginx/1.18.0
built by gcc 11.2.0 (Ubuntu 11.2.0-19ubuntu1)
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --add-module=./modules/ngx_http_upstream_dynamic_module --add-module=/home/u5surf/nginx-module-vts

And I consider that the issue has been fixed by #252.
Could you check this fix on your environment if you have the environment which can be compiled from the source?

u5surf · 2022-10-18T22:50:54Z

@hailiangchen Hi, the issue seems to be fixed.

Commit: 4bbac92

If you have any issue, can you close this?

hailiangchen · 2022-11-05T02:01:46Z

ok，let me check out my environmental problems. Think you

hailiangchen · 2022-11-05T03:15:08Z

Compile environment:

[root@localhost tengine-2.3.3]# ./configure --prefix=/usr/local/tengine  --add-module=./modules/ngx_http_upstream_dynamic_module/ --add-module=/opt/nginx-module-vts-0.2.1/ 
checking for OS
 + Linux 3.10.0-1062.18.1.el7.x86_64 x86_64
checking for C compiler ... found
 + using GNU C compiler
 + gcc version: 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) 
checking for gcc -pipe switch ... found
checking for -Wl,-E switch ... found
checking for gcc builtin atomic operations ... found
checking for C99 variadic macros ... found
checking for gcc variadic macros ... found
checking for gcc builtin 64 bit byteswap ... found
checking for unistd.h ... found
checking for inttypes.h ... found
checking for limits.h ... found
checking for sys/filio.h ... not found
checking for sys/param.h ... found
checking for sys/mount.h ... found
checking for sys/statvfs.h ... found
checking for crypt.h ... found
checking for Linux specific features
checking for epoll ... found
checking for EPOLLRDHUP ... found
checking for EPOLLEXCLUSIVE ... not found
checking for O_PATH ... found
checking for sendfile() ... found
checking for sendfile64() ... found
checking for sys/prctl.h ... found
checking for prctl(PR_SET_DUMPABLE) ... found
checking for prctl(PR_SET_KEEPCAPS) ... found
checking for capabilities ... found
checking for crypt_r() ... found
checking for sys/vfs.h ... found
checking for nobody group ... found
checking for poll() ... found
checking for /dev/poll ... not found
checking for kqueue ... not found
checking for crypt() ... not found
checking for crypt() in libcrypt ... found
checking for F_READAHEAD ... not found
checking for posix_fadvise() ... found
checking for O_DIRECT ... found
checking for F_NOCACHE ... not found
checking for directio() ... not found
checking for statfs() ... found
checking for statvfs() ... found
checking for dlopen() ... not found
checking for dlopen() in libdl ... found
checking for sched_yield() ... found
checking for sched_setaffinity() ... found
checking for SO_SETFIB ... not found
checking for SO_REUSEPORT ... found
checking for SO_ACCEPTFILTER ... not found
checking for SO_BINDANY ... not found
checking for IP_TRANSPARENT ... found
checking for IP_BINDANY ... not found
checking for IP_BIND_ADDRESS_NO_PORT ... found
checking for IP_RECVDSTADDR ... not found
checking for IP_SENDSRCADDR ... not found
checking for IP_PKTINFO ... found
checking for IPV6_RECVPKTINFO ... found
checking for TCP_DEFER_ACCEPT ... found
checking for TCP_KEEPIDLE ... found
checking for TCP_FASTOPEN ... found
checking for TCP_INFO ... found
checking for accept4() ... found
checking for eventfd() ... found
checking for int size ... 4 bytes
checking for long size ... 8 bytes
checking for long long size ... 8 bytes
checking for void * size ... 8 bytes
checking for uint32_t ... found
checking for uint64_t ... found
checking for sig_atomic_t ... found
checking for sig_atomic_t size ... 4 bytes
checking for socklen_t ... found
checking for in_addr_t ... found
checking for in_port_t ... found
checking for rlim_t ... found
checking for uintptr_t ... uintptr_t found
checking for system byte ordering ... little endian
checking for size_t size ... 8 bytes
checking for off_t size ... 8 bytes
checking for time_t size ... 8 bytes
checking for AF_INET6 ... found
checking for setproctitle() ... not found
checking for pread() ... found
checking for pwrite() ... found
checking for pwritev() ... found
checking for sys_nerr ... found
checking for localtime_r() ... found
checking for clock_gettime(CLOCK_MONOTONIC) ... found
checking for posix_memalign() ... found
checking for memalign() ... found
checking for mmap(MAP_ANON|MAP_SHARED) ... found
checking for mmap("/dev/zero", MAP_SHARED) ... found
checking for System V shared memory ... found
checking for POSIX semaphores ... not found
checking for POSIX semaphores in libpthread ... found
checking for struct msghdr.msg_control ... found
checking for ioctl(FIONBIO) ... found
checking for ioctl(FIONREAD) ... found
checking for struct tm.tm_gmtoff ... found
checking for struct dirent.d_namlen ... not found
checking for struct dirent.d_type ... found
checking for sysconf(_SC_NPROCESSORS_ONLN) ... found
checking for sysconf(_SC_LEVEL1_DCACHE_LINESIZE) ... found
checking for openat(), fstatat() ... found
checking for getaddrinfo() ... found
checking for sysinfo() ... found
checking for getloadavg() ... found
checking for /proc/meminfo ... found
checking for /proc/stat ... found
configuring additional modules
adding module in ./modules/ngx_http_upstream_dynamic_module/
 + ngx_http_upstream_dynamic_module was configured
adding module in /opt/nginx-module-vts-0.2.1/
 + ngx_http_vhost_traffic_status_module was configured
checking for PCRE library ... found
checking for PCRE JIT support ... found
checking for OpenSSL library ... found
checking for zlib library ... found
creating objs/Makefile

Configuration summary
  + using system PCRE library
  + using system OpenSSL library
  + using system zlib library
  + jemalloc library is disabled

  nginx path prefix: "/usr/local/tengine"
  nginx binary file: "/usr/local/tengine/sbin/nginx"
  nginx modules path: "/usr/local/tengine/modules"
  nginx configuration prefix: "/usr/local/tengine/conf"
  nginx configuration file: "/usr/local/tengine/conf/nginx.conf"
  nginx pid file: "/usr/local/tengine/logs/nginx.pid"
  nginx error log file: "/usr/local/tengine/logs/error.log"
  nginx http access log file: "/usr/local/tengine/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"

conf file :

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;

    vhost_traffic_status_zone;
    server {
        listen 80;
        server_name localhost;

        access_log off;
        error_log off;

        location /status {
            vhost_traffic_status_display;
            vhost_traffic_status_display_format prometheus;
        }
    }

    upstream web{
        dynamic_resolve fallback=stale fail_timeout=5s;
        server web1.server1;
        keepalive 200;
    }

    server {
        listen 443 ssl;
        server_name web.xxxxxx.com;
        include ssl.conf;
        resolver 10.200.0.10;

        location / {
                include proxy.conf;
                client_max_body_size 100m;
                proxy_pass http://web;
        }
    }
}

error.log

2022/11/05 11:01:05 [error] 1606#0: *35 handler::shm_add_upstream() failed while logging request, client: xx.xx.xx.xx, server: web.xxxx.com, request: "GET /css/bundle.login.min.css?v=gbyvjfarbgWT1e-cOZIwih91Xv4U_hbdhZ7gdruQOfE HTTP/1.1", upstream: "http://xx.xx.xx.xx:80/css/bundle.login.min.css?v=gbyvjfarbgWT1e-cOZIwih91Xv4U_hbdhZ7gdruQOfE", host: "xx.xx.xx.xx", referrer: "https://xx.xx.xx.xx/Login?ReturnUrl=%2F"
2022/11/05 11:01:05 [error] 1606#0: *35 shm_add_upstream::peer failed while logging request, client: xx.xx.xx.xx, server: web.xxxx.com, request: "GET /Users/LoadCaptcha?id=e49459485b604dd58174624471a2cc0dEN2E&_t=1667617266348 HTTP/1.1", upstream: "http://xx.xx.xx.xx:80/Users/LoadCaptcha?id=e49459485b604dd58174624471a2cc0dEN2E&_t=1667617266348", host: "xx.xx.xx.xx", referrer: "https://xx.xx.xx.xx/Login?ReturnUrl=%2F"
2022/11/05 11:01:05 [error] 1606#0: *35 handler::shm_add_upstream() failed while logging request, client: xx.xx.xx.xx, server: web.xxxx.com, request: "GET /Users/LoadCaptcha?id=e49459485b604dd58174624471a2cc0dEN2E&_t=1667617266348 HTTP/1.1", upstream: "http://xx.xx.xx.xx:80/Users/LoadCaptcha?id=e49459485b604dd58174624471a2cc0dEN2E&_t=1667617266348", host: "xx.xx.xx.xx", referrer: "https://xx.xx.xx.xx/Login?ReturnUrl=%2F"
2022/11/05 11:01:05 [error] 1606#0: *35 shm_add_upstream::peer failed while logging request, client: xx.xx.xx.xx, server: web.xxxx.com, request: "GET /img/favicon.ico HTTP/1.1", upstream: "http://xx.xx.xx.xx:80/img/favicon.ico", host: "xx.xx.xx.xx", referrer: "https://xx.xx.xx.xx/Login?ReturnUrl=%2F"
2022/11/05 11:01:05 [error] 1606#0: *35 handler::shm_add_upstream() failed while logging request, client: xx.xx.xx.xx, server: web.xxxx.com, request: "GET /img/favicon.ico HTTP/1.1", upstream: "http://xx.xx.xx.xx:80/img/favicon.ico", host: "xx.xx.xx.xx", referrer: "https://xx.xx.xx.xx/Login?ReturnUrl=%2F"

hailiangchen · 2022-11-05T03:15:48Z

I still get errors using the latest version

u5surf · 2022-11-05T06:23:43Z

@hailiangchen The latest version doesn't include 4bbac92 yet.

hailiangchen · 2022-11-05T07:43:37Z

Ok, problem solved. Thank you

vozlt added the question label Sep 5, 2022

u5surf mentioned this issue Oct 18, 2022

Bugfix: fixed issues/228 Change the reffered source of upstream_state #252

Merged

u5surf added the compatibility label Oct 18, 2022

u5surf closed this as completed in #252 Oct 18, 2022

u5surf reopened this Oct 18, 2022

u5surf removed the question label Oct 18, 2022

hailiangchen closed this as completed Nov 5, 2022

hailiangchen reopened this Nov 5, 2022

hailiangchen closed this as completed Nov 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

shm_add_upstream #228

shm_add_upstream #228

hailiangchen commented Mar 17, 2022

vozlt commented Sep 5, 2022

hailiangchen commented Oct 11, 2022

u5surf commented Oct 15, 2022 •

edited

Loading

vozlt commented Oct 17, 2022

u5surf commented Oct 18, 2022

u5surf commented Oct 18, 2022

hailiangchen commented Nov 5, 2022

hailiangchen commented Nov 5, 2022

hailiangchen commented Nov 5, 2022

u5surf commented Nov 5, 2022

hailiangchen commented Nov 5, 2022

shm_add_upstream #228

shm_add_upstream #228

Comments

hailiangchen commented Mar 17, 2022

vozlt commented Sep 5, 2022

hailiangchen commented Oct 11, 2022

u5surf commented Oct 15, 2022 • edited Loading

vozlt commented Oct 17, 2022

u5surf commented Oct 18, 2022

u5surf commented Oct 18, 2022

hailiangchen commented Nov 5, 2022

hailiangchen commented Nov 5, 2022

hailiangchen commented Nov 5, 2022

u5surf commented Nov 5, 2022

hailiangchen commented Nov 5, 2022

u5surf commented Oct 15, 2022 •

edited

Loading