Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

restart docker error:[emerg] 1#1: bind() to unix:/usr/local/apisix/conf/config_listen.sock failed (98: Address already in use) #373

Closed
qiaogw opened this issue Nov 6, 2022 · 30 comments
Labels

Comments

@qiaogw
Copy link

qiaogw commented Nov 6, 2022

docker-compose:
version: '3.5'

网络配置

networks:
dev_net:
driver: bridge

服务容器配置

services:
etcd:
# 自定义容器名称
image: quay.io/coreos/etcd:v3.5.1 # 镜像
container_name: etcd
environment:
- TZ=${TZ}
- ETCD_NAME=etcd
- ALLOW_NONE_AUTHENTICATION=yes
- ETCD_ADVERTISE_CLIENT_URLS=http://etcd:2379,http://127.0.0.1:2379
- ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379 # 用于监听客户端通讯的URL列表
ports:
# 设置端口映射
- ${ETCD_PORT}:2379
networks:
- dev_net
restart: always
apisix:
image: apache/apisix:3.0.0-debian
restart: always
container_name: apisix
environment:
- LANG=zh_CN.UTF-8
# - LC_ALL=zh_CN.UTF-8
- TZ=Asia/Shanghai
volumes:
# - ./apisix/apisix_log:/usr/local/apisix/logs
- ./apisix/apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml
- /etc/localtime:/etc/localtime:ro
depends_on:
- etcd
ports:
- '9180:9180/tcp'
- '9080:9080/tcp'
- '19091:9091/tcp'
- '9443:9443/tcp'
networks:
- dev_net

web1:
  image: nginx:1.19.0-alpine
  restart: always
  container_name: apisix-web1
  environment:
    - LANG=zh_CN.UTF-8
    - LC_ALL=zh_CN.UTF-8
    - TZ=Asia/Shanghai
    - NGINX_PORT=80
  volumes:
    # - ./apisix/upstream/web1.conf:/etc/nginx/nginx.conf
    - /etc/localtime:/etc/localtime:ro
  ports:
    - '9081:80/tcp'
  networks:
    - dev_net

web2:
  image: nginx:1.19.0-alpine
  restart: always
  container_name: apisix-web2
  environment:
    - LANG=zh_CN.UTF-8
    - LC_ALL=zh_CN.UTF-8
    - TZ=Asia/Shanghai
    - NGINX_PORT=80
  volumes:
    # - ./apisix/upstream/web2.conf:/etc/nginx/nginx.conf
    - /etc/localtime:/etc/localtime:ro
  ports:
    - '9082:80/tcp'
  networks:
    - dev_net

docker env: wsl2 debain. docker start and service ok . 

But when docker service restart, apisix fail this error .
must edit the docker-compose.yml file anywhere then restart succes .

@tokers
Copy link
Contributor

tokers commented Nov 7, 2022

I remember this is a known issue. The config_listen.sock is not deleted when stop.

@qiaogw
Copy link
Author

qiaogw commented Nov 9, 2022

Image apache/apisix:3.0.0-debian is latest. The bug have closed. why????????????

@tokers
Copy link
Contributor

tokers commented Nov 9, 2022

@spacewander I forget the details about it, do you still remember?

@spacewander
Copy link
Member

It is expected to be fixed in apache/apisix#8022

@spacewander
Copy link
Member

Could you provide a complete reproducible example? I can't reproduce the issue because the current example lacks files like ./apisix/apisix_conf/config.yaml.

@qiaogw
Copy link
Author

qiaogw commented Nov 9, 2022

apisix:
node_listen: 9080 # APISIX listening port
enable_ipv6: false

enable_control: true
control:
ip: '0.0.0.0'
port: 9092

deployment:
admin:
allow_admin: # http://nginx.org/en/docs/http/ngx_http_access_module.html#allow
- 0.0.0.0/0 # We need to restrict ip access rules for security. 0.0.0.0/0 is for test.

admin_key:
  - name: 'admin'
    key: edd1c9f011134335f136f87ad84b625c8f1
    role: admin # admin: manage all configuration data

  - name: 'viewer'
    key: 4054f7cf01117e344346cd3f287985e76a2
    role: viewer

etcd:
host:
- 'http://etcd:2379'
prefix: '/apisix'
timeout: 30

@spacewander
Copy link
Member

I tried docker stop/start the APISIX docker container but everything works OK.

@wxbty
Copy link
Member

wxbty commented Nov 24, 2022

I use the latest docker apisix-docker, encountered the same problem

@spacewander
Copy link
Member

@wxbty
Could you provide an example step by step (including the materials), so that we can reproduce it?

@wxbty
Copy link
Member

wxbty commented Nov 24, 2022

I deleted everything and started again, can't reproduce now.
I think this may be the reason, using m1 mac, so I used the following command to start
“docker-compose -p docker-apisix -f docker-compose-arm64.yml up -d”

But when I updated the plugin to restart docker, I used this:
“docker-compose -p docker-apisix restart”
I'm not sure if the problem is caused by not specifying arm.yaml

@wxbty
Copy link
Member

wxbty commented Nov 24, 2022

@spacewander Can you help me look at this issue #apache/apisix#8386? I ran a demo, and the reported error was incomprehensible. I tried many methods, but nothing worked. thx!

@tzssangglass
Copy link
Member

I'm not sure if the problem is caused by not specifying arm.yaml

maybe, different images are used on the arm system.

@Hxhong
Copy link

Hxhong commented Nov 28, 2022

I deleted everything and started again, can't reproduce now. I think this may be the reason, using m1 mac, so I used the following command to start “docker-compose -p docker-apisix -f docker-compose-arm64.yml up -d”

But when I updated the plugin to restart docker, I used this: “docker-compose -p docker-apisix restart” I'm not sure if the problem is caused by not specifying arm.yaml

I have the same problem whether it is mac m1 or centos system. How did you solve it?

@wxbty
Copy link
Member

wxbty commented Nov 28, 2022

@Hxhong Not resolved, put on hold for now

@Hxhong
Copy link

Hxhong commented Nov 28, 2022

I'm not sure if the problem is caused by not specifying arm.yaml

maybe, different images are used on the arm system.

The image I used that matches the system has the same problem. Can you check whether there is a problem with the image?
image: apache/apisix:3.0.0-centos@sha: ee9d7a6224db5781b8561bd7ed3695e41c6808288008ae09307302dcfbeb6c07
env: mac m1
or
image: apache/apisix:3.0.0-centos@sha: 69552d3fab5de23ecca2bd3161b172b796db865458076b38bdc9e766c14a07dc
env: centos 7

@github-actions
Copy link

This issue has been marked as stale due to 30 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Dec 28, 2022
@github-actions github-actions bot removed the stale label Jan 10, 2023
@github-actions
Copy link

This issue has been marked as stale due to 30 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Feb 20, 2023
@github-actions
Copy link

github-actions bot commented Mar 7, 2023

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.

@github-actions github-actions bot closed this as completed Mar 7, 2023
@GeYiPeng
Copy link

我也是同样的问题,吐了

@WordlessEcho
Copy link

No. It doesn't fix.
I run apisix on WSL. WSL will shutdown automatically. It happens after I start WSL and docker again.

/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init

/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
2023/04/13 11:01:22 [emerg] 1#1: bind() to unix:/usr/local/apisix/conf/config_listen.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/conf/config_listen.sock failed (98: Address already in use)
2023/04/13 11:01:22 [emerg] 1#1: bind() to unix:/usr/local/apisix/conf/config_listen.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/conf/config_listen.sock failed (98: Address already in use)
2023/04/13 11:01:22 [emerg] 1#1: bind() to unix:/usr/local/apisix/conf/config_listen.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/conf/config_listen.sock failed (98: Address already in use)
2023/04/13 11:01:22 [emerg] 1#1: bind() to unix:/usr/local/apisix/conf/config_listen.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/conf/config_listen.sock failed (98: Address already in use)
2023/04/13 11:01:22 [emerg] 1#1: bind() to unix:/usr/local/apisix/conf/config_listen.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/usr/local/apisix/conf/config_listen.sock failed (98: Address already in use)
2023/04/13 11:01:22 [emerg] 1#1: still could not bind()
nginx: [emerg] still could not bind()

@jeffreys-cat
Copy link

I'm not sure if the problem is fixed or not, we use apache/apisix:2.99.0-debian, and while Aliyun ECS down makes
the error then docker auto restart does not work. It was a very bad case in Production Environment.

@chanble
Copy link

chanble commented May 6, 2023

Remove web1 and web2 image

@DennisGuo
Copy link

突然断电后,来电服务器自动启动,会导致这个问题:

1#1: bind() to unix:/usr/local/apisix/conf/config_listen.sock failed (98: Address already in use)

@mojtaba-roohi
Copy link

I use the latest docker apisix-docker, encountered the same problem

services:
apisix-dashboard:
image: apache/apisix-dashboard:3.0.1-alpine
restart: always
volumes:
- ./dashboard_conf/conf.yaml:/usr/local/apisix-dashboard/conf/conf.yaml
ports:
- "9000:9000"
networks:
apisix:

apisix:
image: apache/apisix:${APISIX_IMAGE_TAG:-3.3.0-debian}
restart: always
volumes:
- ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
depends_on:
- etcd
##network_mode: host
ports:
- "9180:9180/tcp"
- "9080:9080/tcp"
- "9091:9091/tcp"
- "9443:9443/tcp"
- "9092:9092/tcp"
networks:
apisix:

etcd:
image: bitnami/etcd:3.4.15
restart: always
volumes:
- etcd_data:/bitnami/etcd
environment:
ETCD_ENABLE_V2: "true"
ALLOW_NONE_AUTHENTICATION: "yes"
ETCD_ADVERTISE_CLIENT_URLS: "http://etcd:2379"
ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
ports:
- "2379:2379/tcp"
networks:
apisix:

web1:
image: nginx:1.19.0-alpine
restart: always
volumes:
- ./upstream/web1.conf:/etc/nginx/nginx.conf
ports:
- "9081:80/tcp"
environment:
- NGINX_PORT=80
networks:
apisix:

web2:
image: nginx:1.19.0-alpine
restart: always
volumes:
- ./upstream/web2.conf:/etc/nginx/nginx.conf
ports:
- "9082:80/tcp"
environment:
- NGINX_PORT=80
networks:
apisix:

prometheus:
image: prom/prometheus:v2.25.0
restart: always
volumes:
- ./prometheus_conf/prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
networks:
apisix:

grafana:
image: grafana/grafana:7.3.7
restart: always
ports:
- "3000:3000"
volumes:
- "./grafana_conf/provisioning:/etc/grafana/provisioning"
- "./grafana_conf/dashboards:/var/lib/grafana/dashboards"
- "./grafana_conf/config/grafana.ini:/etc/grafana/grafana.ini"
networks:
apisix:

networks:
apisix:
driver: bridge

volumes:
etcd_data:
driver: local

@WordlessEcho
Copy link

WordlessEcho commented Jun 4, 2023

Update: Just restart

docker restart <container_name>

And:

Do this during container start

docker exec -i <container_name> rm /usr/local/apisix/conf/config_listen.sock

@spatxos
Copy link

spatxos commented Jul 20, 2023

在容器启动期间执行此操作

docker exec -i <container_name> rm /usr/local/apisix/conf/config_listen.sock

在docker compose中该如何执行,这个情况应当避免手动操作,另一个办法是写一个systemd service去执行删除操作或者直接docker compose down然后docker compose up -d,但我认为这个并不是最好的解决办法

@wavemoroc001
Copy link

still occur on version 3.4.1

@WordlessEcho
Copy link

在容器启动期间执行此操作

docker exec -i <container_name> rm /usr/local/apisix/conf/config_listen.sock

在docker compose中该如何执行,这个情况应当避免手动操作,另一个办法是写一个systemd service去执行删除操作或者直接docker compose down然后docker compose up -d,但我认为这个并不是最好的解决办法

Compose部署出来的和普通的没什么区别的,搞清楚容器名就能执行
这个确实只是个workaround

@flavienbwk
Copy link

This issue should be reopened.

Here are the logs when running the bash /docker-entrypoint.sh docker-start command :

Attaching to server_apisix_1
apisix_1            | /usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init
apisix_1            | /usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
apisix_1            | 2023/08/27 13:40:10 [emerg] 107#107: bind() to 192.168.96.3:9091 failed (99: Cannot assign requested address)
apisix_1            | 2023/08/27 13:40:10 [alert] 107#107: failed to init privileged agent listeners (99: Cannot assign requested address)
apisix_1            | 2023/08/27 13:40:10 [alert] 9#9: privileged agent process 107 exited with fatal code 2 and cannot be respawned

@monkeyDluffy6017
Copy link
Contributor

The /usr/local/apisix/conf/config_listen.sock is used by the conf_server, and the conf_server is deleted by this pr: apache/apisix#10012, so the problem won't happen again in the latest version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests