Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: apisix 2.15.1 do not accept private CA (worked in apisix 2.13.0) #8568

Closed
MirtoBusico opened this issue Dec 25, 2022 · 13 comments
Closed

bug: apisix 2.15.1 do not accept private CA (worked in apisix 2.13.0) #8568

MirtoBusico opened this issue Dec 25, 2022 · 13 comments

Comments

@MirtoBusico
Copy link

Current Behavior

I have a keycloak server with certificates signed by a private Certification Authority.
Apisix is installed via helm chart in kebernetes
The apisix pod is not able to access the keycloak server saying that the issuer is unknown

root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# wget https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration
--2022-12-25 11:51:20--  https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration
Resolving k6k.h.net (k6k.h.net)... 192.168.100.20
Connecting to k6k.h.net (k6k.h.net)|192.168.100.20|:443... connected.
ERROR: The certificate of 'k6k.h.net' is not trusted.
ERROR: The certificate of 'k6k.h.net' doesn't have a known issuer.
root@apisix-54cdc68f89-wtl8w:/usr/local/apisix#

I created a kubernetes secret:

sysop@hserv:~$ cd ~/H/hservcerts
sysop@hserv:~/H/hservcerts$ sudo chmod a+r hservca.key
sysop@hserv:~/H/hservcerts$ ls -lh hservca.*
-rw-r--r-- 1 root root 1,7K ago 15 17:37 hservca.key
-rw-r--r-- 1 root root 1,5K ago 15 17:40 hservca.pem
sysop@hserv:~/H/hservcerts$ kubectl -n apisix create secret generic hservcacert --from-file=cert=./hservca.pem
secret/hservcacert created
sysop@hserv:~/H/hservcerts$ kubectl describe secret hservcacert -n apisix
Name:         hservcacert
Namespace:    apisix
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
cert:  1464 bytes
sysop@hserv:~/H/hservcerts$

and referenced the secret in the tls section of the gateway (in helm chart values.yaml)

gateway:
  type: LoadBalancer

...

  tls:
    enabled: true
    servicePort: 443
    containerPort: 9443
    existingCASecret: "hservcacert"
    certCAFilename: "cert"

What can I do?

Expected Behavior

In apisix 2.13.0 it worked correctly

I created the secret:

sysop@m01serv:~/m01certs$ cd ~/m01certs
sysop@m01serv:~/m01certs$ ls -lh m01ca.*
-rw-r--r-- 1 root root 1,7K gen 17 18:43 m01ca.key
-rw-r--r-- 1 root root 1,5K gen 17 18:46 m01ca.pem
-rw-r--r-- 1 root root   41 feb 11 14:01 m01ca.srl
sysop@m01serv:~/m01certs$ kubectl -n kube-system create secret generic m01cacert --from-file=cert=./m01ca.pem
secret/m01cacert created
sysop@m01serv:~/m01certs$ kubectl describe secret m01cacert -n kube-system
Name:         m01cacert
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
cert:  1464 bytes
sysop@m01serv:~/m01certs$

Instered the secret in the tls section of gateway (in values.yaml file of the helm chart)

gateway:
  type: LoadBalancer

...

  tls:
    enabled: true
    servicePort: 443
    containerPort: 9443
    existingCASecret: "m01cacert"
    certCAFilename: "cert"

Error Logs

Tryin to access the keycloak server gives the issuer unknown error

root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# wget https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration
--2022-12-25 11:51:20--  https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration
Resolving k6k.h.net (k6k.h.net)... 192.168.100.20
Connecting to k6k.h.net (k6k.h.net)|192.168.100.20|:443... connected.
ERROR: The certificate of 'k6k.h.net' is not trusted.
ERROR: The certificate of 'k6k.h.net' doesn't have a known issuer.
root@apisix-54cdc68f89-wtl8w:/usr/local/apisix#

Steps to Reproduce

  1. create a kubernetes secret with the private CA pem
sysop@hserv:~$ cd ~/H/hservcerts
sysop@hserv:~/H/hservcerts$ sudo chmod a+r hservca.key
sysop@hserv:~/H/hservcerts$ ls -lh hservca.*
-rw-r--r-- 1 root root 1,7K ago 15 17:37 hservca.key
-rw-r--r-- 1 root root 1,5K ago 15 17:40 hservca.pem
sysop@hserv:~/H/hservcerts$ kubectl -n apisix create secret generic hservcacert --from-file=cert=./hservca.pem
secret/hservcacert created
sysop@hserv:~/H/hservcerts$ kubectl describe secret hservcacert -n apisix
Name:         hservcacert
Namespace:    apisix
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
cert:  1464 bytes
sysop@hserv:~/H/hservcerts$
  1. modify the apisix helm chart values.yaml file
gateway:
  type: LoadBalancer
  
...

 http:
    enabled: true
    servicePort: 80
    containerPort: 9080
  tls:
    enabled: true
    servicePort: 443
    containerPort: 9443
    existingCASecret: "hservcacert"
    certCAFilename: "cert"
    http2:
      enabled: true
    sslProtocols: "TLSv1.2 TLSv1.3"
  1. install apisix
helm install apisix apisix/apisix -f apisix-values.yaml \
--set ingress-controller.config.apisix.serviceNamespace=apisix \
--set ingress-controller.config.apisix.serviceName=apisix-admin \
--set ingress-controller.config.kubernetes.apisixRouteVersion=apisix.apache.org/v2beta3 \
--namespace apisix
  1. try to use the openid-connect plugin on a route and gen an error
2022/12/05 08:19:27 [warn] 49#49: *48972 [lua] plugin.lua:934: run_plugin(): openid-connect exits with http status code 500, client: 127.0.0.6, server: _, request: "GET /*?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net"
2022/12/05 08:19:27 [alert] 49#49: *48972 ignoring stale global SSL error (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt), client: 127.0.0.6, server: _, request: "GET /*?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net"
2022/12/05 08:19:27 [alert] 47#47: *48973 ignoring stale global SSL error (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt), client: 127.0.0.6, server: _, request: "GET /favicon.ico HTTP/1.0", host: "apisix.h.net", referrer: "https://apisix.h.net/*?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756"
2022/12/05 08:19:27 [error] 48#48: *48980 [lua] openidc.lua:1475: authenticate(): request to the redirect_uri path but there's no session state found, client: 127.0.0.6, server: _, request: "GET /*?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net"
2022/12/05 08:19:27 [error] 48#48: *48980 [lua] openid-connect.lua:315: phase_func(): OIDC authentication failed: request to the redirect_uri path but there's no session state found, client: 127.0.0.6, server: _, request: "GET /*?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net"
2022/12/05 08:19:27 [warn] 48#48: *48980 [lua] plugin.lua:934: run_plugin(): openid-connect exits with http status code 500, client: 127.0.0.6, server: _, request: "GET /*?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net"
2022/12/05 08:19:27 [alert] 48#48: *48980 ignoring stale global SSL error (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt), client: 127.0.0.6, server: _, request: "GET /*?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net"
127.0.0.6 - - [05/Dec/2022:08:19:27 +0000] apisix.h.net "GET /*?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0" 500 553 0.000 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" - - - "http://apisix.h.net"
127.0.0.6 - - [05/Dec/2022:08:19:27 +0000] apisix.h.net "GET /favicon.ico HTTP/1.0" 302 217 0.000 "https://apisix.h.net/*?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756" "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" - - - "http://apisix.h.net"
127.0.0.6 - - [05/Dec/2022:08:19:27 +0000] apisix.h.net "GET /*?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0" 500 553 0.000 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" - - - "http://apisix.h.net"
  1. verify that the keycloak server is not reachable from the apisix pod
root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# wget https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration
--2022-12-25 12:11:16--  https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration
Resolving k6k.h.net (k6k.h.net)... 192.168.100.20
Connecting to k6k.h.net (k6k.h.net)|192.168.100.20|:443... connected.
ERROR: The certificate of 'k6k.h.net' is not trusted.
ERROR: The certificate of 'k6k.h.net' doesn't have a known issuer.
root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# 
root@apisix-54cdc68f89-wtl8w:/usr/local/apisix#

Environment

  • APISIX version (run apisix version):
root@apisix-64fffcfb4c-55vhw:/usr/local/apisix# apisix version
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua version
2.15.1
root@apisix-64fffcfb4c-55vhw:/usr/local/apisix#
  • Operating system (run uname -a):
root@apisix-64fffcfb4c-55vhw:/usr/local/apisix# uname -a
Linux apisix-64fffcfb4c-55vhw 5.15.0-53-generic #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64 GNU/Linux
root@apisix-64fffcfb4c-55vhw:/usr/local/apisix#
  • OpenResty / Nginx version (run openresty -V or nginx -V):
  • etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
  • APISIX Dashboard version, if relevant: 2.13.0
  • Plugin runner version, for issues related to plugin runners:
  • LuaRocks version, for installation issues (run luarocks --version):
@tokers
Copy link
Contributor

tokers commented Dec 26, 2022

  1. What you did is checking if the wget command in the APISIX pod can access your keycloak server, not Apache APISIX itself.
  2. If you want to let wget trust the private CA, try to use the option --ca-certificate.
  3. If you want APISIX to trust the private CA, specify via the ssl_trusted_certificate field

@MirtoBusico
Copy link
Author

Hi @tokers

  1. I know this; I'm trying to see if any program on apisix pod can access the keycloak server
  2. I want that wget can do the access correctly without using the --ca-certificate option
  3. this is exactly what I'm trying to do setting the existingCASecret and certCAFilename fields in the values.yaml file of the apisix helm chart

At the times of apisix 2.12.0 the tls settings of the helm chart were sufficient (see #6345 )
Now with apisix 2.15.1 seems that the step followed in the past are no more correct

What can I do?

@tokers
Copy link
Contributor

tokers commented Dec 26, 2022

I want that wget can do the access correctly without using the --ca-certificate option

I don't thinks this can be done unless you update the base image (adding your private CA to the system trusted CA bundles)

this is exactly what I'm trying to do setting the existingCASecret and certCAFilename fields in the values.yaml file of the apisix helm chart

Could you please paste the nginx.conf in the APISIX pod, the path is /usr/local/apisix/conf/nginx.conf. You may focus on the directive lua_ssl_trusted_certificate in the nginx.conf.

@MirtoBusico
Copy link
Author

Well the lua_ssl_trusted_certificate is

    resolver_timeout 5;

    lua_ssl_trusted_certificate /usr/local/apisix/conf/ssl/cert;

    # for stream logs, off by default

But I don't know how to set values in the helm chart to change this configuration

The complete nginx.conf is

root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# cat /usr/local/apisix/conf/nginx.conf
# Configuration File - Nginx Server Configs
# This is a read-only file, do not try to modify it.
master_process on;

worker_processes auto;
worker_cpu_affinity auto;

# main configuration snippet starts

# main configuration snippet ends

error_log /dev/stderr warn;
pid logs/nginx.pid;

worker_rlimit_nofile 20480;

events {
    accept_mutex off;
    worker_connections 10620;
}

worker_rlimit_core  16G;

worker_shutdown_timeout 240s;

env APISIX_PROFILE;
env PATH; # for searching external plugin runner's binary


lua {
}



stream {
    lua_package_path  "$prefix/deps/share/lua/5.1/?.lua;$prefix/deps/share/lua/5.1/?/init.lua;/usr/local/apisix/?.lua;/usr/local/apisix/?/init.lua;;/usr/local/apisix/?.lua;./?.lua;/usr/local/openresty/luajit/share/luajit-2.1.0-beta3/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;/usr/local/openresty/luajit/share/lua/5.1/?.lua;/usr/local/openresty/luajit/share/lua/5.1/?/init.lua;;";
    lua_package_cpath "$prefix/deps/lib64/lua/5.1/?.so;$prefix/deps/lib/lua/5.1/?.so;;./?.so;/usr/local/lib/lua/5.1/?.so;/usr/local/openresty/luajit/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so;";
    lua_socket_log_errors off;

    lua_max_pending_timers 16384;
    lua_max_running_timers 4096;

    lua_shared_dict lrucache-lock-stream 10m;
    lua_shared_dict etcd-cluster-health-check-stream 10m;

    lua_shared_dict plugin-limit-conn-stream 10m;

    resolver 10.43.0.10 valid=30 ipv6=on;
    resolver_timeout 5;

    lua_ssl_trusted_certificate /usr/local/apisix/conf/ssl/cert;

    # for stream logs, off by default

    # stream configuration snippet starts
    
    # stream configuration snippet ends

    upstream apisix_backend {
        server 127.0.0.1:80;
        balancer_by_lua_block {
            apisix.stream_balancer_phase()
        }
    }

    init_by_lua_block {
        require "resty.core"
        apisix = require("apisix")
        local dns_resolver = { "10.43.0.10", }
        local args = {
            dns_resolver = dns_resolver,
        }
        apisix.stream_init(args)
    }

    init_worker_by_lua_block {
        apisix.stream_init_worker()
    }

    server {
        listen 9100 reuseport;
        listen 9200 udp reuseport;



        preread_by_lua_block {
            apisix.stream_preread_phase()
        }

        proxy_pass apisix_backend;

        set $upstream_sni "apisix_backend";
        proxy_ssl_server_name on;
        proxy_ssl_name $upstream_sni;

        log_by_lua_block {
            apisix.stream_log_phase()
        }
    }
}

http {
    # put extra_lua_path in front of the builtin path
    # so user can override the source code
    lua_package_path  "$prefix/deps/share/lua/5.1/?.lua;$prefix/deps/share/lua/5.1/?/init.lua;/usr/local/apisix/?.lua;/usr/local/apisix/?/init.lua;;/usr/local/apisix/?.lua;./?.lua;/usr/local/openresty/luajit/share/luajit-2.1.0-beta3/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;/usr/local/openresty/luajit/share/lua/5.1/?.lua;/usr/local/openresty/luajit/share/lua/5.1/?/init.lua;;";
    lua_package_cpath "$prefix/deps/lib64/lua/5.1/?.so;$prefix/deps/lib/lua/5.1/?.so;;./?.so;/usr/local/lib/lua/5.1/?.so;/usr/local/openresty/luajit/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so;";

    lua_max_pending_timers 16384;
    lua_max_running_timers 4096;

    lua_shared_dict internal-status 10m;
    lua_shared_dict upstream-healthcheck 10m;
    lua_shared_dict worker-events 10m;
    lua_shared_dict lrucache-lock 10m;
    lua_shared_dict balancer-ewma 10m;
    lua_shared_dict balancer-ewma-locks 10m;
    lua_shared_dict balancer-ewma-last-touched-at 10m;
    lua_shared_dict etcd-cluster-health-check 10m; # etcd health check



    lua_shared_dict plugin-limit-conn 10m;

    lua_shared_dict plugin-limit-req 10m;

    lua_shared_dict plugin-limit-count 10m;
    lua_shared_dict plugin-limit-count-redis-cluster-slot-lock 1m;

    lua_shared_dict prometheus-metrics 10m;


    lua_shared_dict plugin-api-breaker 10m;

    # for openid-connect and authz-keycloak plugin
    lua_shared_dict discovery 1m; # cache for discovery metadata documents

    # for openid-connect plugin
    lua_shared_dict jwks 1m; # cache for JWKs
    lua_shared_dict introspection 10m; # cache for JWT verification results

    # for authz-keycloak
    lua_shared_dict access-tokens 1m; # cache for service account access tokens

    lua_shared_dict ext-plugin 1m; # cache for ext-plugin


    # for custom shared dict

    # for proxy cache
    proxy_cache_path /tmp/disk_cache_one levels=1:2 keys_zone=disk_cache_one:50m inactive=1d max_size=1G use_temp_path=off;

    # for proxy cache
    map $upstream_cache_zone $upstream_cache_zone_info {
        disk_cache_one /tmp/disk_cache_one,1:2;
    }


    lua_ssl_verify_depth 5;
    ssl_session_timeout 86400;

    underscores_in_headers on;

    lua_socket_log_errors off;

    resolver 10.43.0.10 valid=30 ipv6=on;
    resolver_timeout 5;

    lua_http10_buffering off;

    lua_regex_match_limit 100000;
    lua_regex_cache_max_entries 8192;

    log_format main escape=default '$remote_addr - $remote_user [$time_local] $http_host "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" $upstream_addr $upstream_status $upstream_response_time "$upstream_scheme://$upstream_host$upstream_uri"';
    uninitialized_variable_warn off;

    access_log /dev/stdout main buffer=16384 flush=3;
    open_file_cache  max=1000 inactive=60;
    client_max_body_size 0;
    keepalive_timeout 60s;
    client_header_timeout 60s;
    client_body_timeout 60s;
    send_timeout 10s;
    variables_hash_max_size 2048;

    server_tokens off;

    include mime.types;
    charset utf-8;

    # error_page
    error_page 500 @50x.html;

    real_ip_header X-Real-IP;

    real_ip_recursive off;

    set_real_ip_from 127.0.0.1;
    set_real_ip_from unix:;

    lua_ssl_trusted_certificate /usr/local/apisix/conf/ssl/cert;

    # http configuration snippet starts
    
    # http configuration snippet ends

    upstream apisix_backend {
        server 0.0.0.1;

        keepalive 320;
        keepalive_requests 1000;
        keepalive_timeout 60s;
        # we put the static configuration above so that we can override it in the Lua code

        balancer_by_lua_block {
            apisix.http_balancer_phase()
        }
    }


    apisix_delay_client_max_body_check on;
    apisix_mirror_on_demand on;


    init_by_lua_block {
        require "resty.core"
        apisix = require("apisix")

        local dns_resolver = { "10.43.0.10", }
        local args = {
            dns_resolver = dns_resolver,
        }
        apisix.http_init(args)
    }

    init_worker_by_lua_block {
        apisix.http_init_worker()
    }

    exit_worker_by_lua_block {
        apisix.http_exit_worker()
    }

    server {
        listen 127.0.0.1:9090;

        access_log off;

        location / {
            content_by_lua_block {
                apisix.http_control()
            }
        }

        location @50x.html {
            set $from_error_page 'true';
            content_by_lua_block {
                require("apisix.error_handling").handle_500()
            }
        }
    }

    server {
        listen 127.0.0.1:9091;

        access_log off;

        location / {
            content_by_lua_block {
                local prometheus = require("apisix.plugins.prometheus.exporter")
                prometheus.export_metrics()
            }
        }

        location = /apisix/nginx_status {
            allow 127.0.0.0/24;
            deny all;
            stub_status;
        }
    }

    server {
        listen 0.0.0.0:9180;
        log_not_found off;

        # admin configuration snippet starts
        
        # admin configuration snippet ends

        set $upstream_scheme             'http';
        set $upstream_host               $http_host;
        set $upstream_uri                '';

        location /apisix/admin {
                allow 127.0.0.1/24;
                allow 0.0.0.0/0;
                deny all;

            content_by_lua_block {
                apisix.http_admin()
            }
        }

        location @50x.html {
            set $from_error_page 'true';
            content_by_lua_block {
                require("apisix.error_handling").handle_500()
            }
        }
    }


    server {
        listen 0.0.0.0:9080 default_server reuseport;
        listen [::]:9080 default_server reuseport;
        listen 0.0.0.0:9443 ssl default_server http2 reuseport;
        listen [::]:9443 ssl default_server http2 reuseport;

        server_name _;

        ssl_certificate      cert/ssl_PLACE_HOLDER.crt;
        ssl_certificate_key  cert/ssl_PLACE_HOLDER.key;
        ssl_session_cache    shared:SSL:20m;
        ssl_session_timeout 10m;

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA;
        ssl_prefer_server_ciphers on;
        ssl_session_tickets off;

        proxy_ssl_trusted_certificate /usr/local/apisix/conf/ssl/cert;

        # http server configuration snippet starts
        set $session_secret 0123456789a5bac9bb3c868ec8b202e93;

        # http server configuration snippet ends

        location = /apisix/nginx_status {
            allow 127.0.0.0/24;
            deny all;
            access_log off;
            stub_status;
        }


        ssl_certificate_by_lua_block {
            apisix.http_ssl_phase()
        }

        proxy_ssl_name $upstream_host;
        proxy_ssl_server_name on;

        location / {
            set $upstream_mirror_uri         '';
            set $upstream_upgrade            '';
            set $upstream_connection         '';

            set $upstream_scheme             'http';
            set $upstream_host               $http_host;
            set $upstream_uri                '';
            set $ctx_ref                     '';
            set $from_error_page             '';


            # http server location configuration snippet starts
            
            # http server location configuration snippet ends


            access_by_lua_block {
                apisix.http_access_phase()
            }

            proxy_http_version 1.1;
            proxy_set_header   Host              $upstream_host;
            proxy_set_header   Upgrade           $upstream_upgrade;
            proxy_set_header   Connection        $upstream_connection;
            proxy_set_header   X-Real-IP         $remote_addr;
            proxy_pass_header  Date;

            ### the following x-forwarded-* headers is to send to upstream server

            set $var_x_forwarded_for        $remote_addr;
            set $var_x_forwarded_proto      $scheme;
            set $var_x_forwarded_host       $host;
            set $var_x_forwarded_port       $server_port;

            if ($http_x_forwarded_for != "") {
                set $var_x_forwarded_for "${http_x_forwarded_for}, ${realip_remote_addr}";
            }
            if ($http_x_forwarded_host != "") {
                set $var_x_forwarded_host $http_x_forwarded_host;
            }
            if ($http_x_forwarded_port != "") {
                set $var_x_forwarded_port $http_x_forwarded_port;
            }

            proxy_set_header   X-Forwarded-For      $var_x_forwarded_for;
            proxy_set_header   X-Forwarded-Proto    $var_x_forwarded_proto;
            proxy_set_header   X-Forwarded-Host     $var_x_forwarded_host;
            proxy_set_header   X-Forwarded-Port     $var_x_forwarded_port;

            ###  the following configuration is to cache response content from upstream server

            set $upstream_cache_zone            off;
            set $upstream_cache_key             '';
            set $upstream_cache_bypass          '';
            set $upstream_no_cache              '';

            proxy_cache                         $upstream_cache_zone;
            proxy_cache_valid                   any 10s;
            proxy_cache_min_uses                1;
            proxy_cache_methods                 GET HEAD POST;
            proxy_cache_lock_timeout            5s;
            proxy_cache_use_stale               off;
            proxy_cache_key                     $upstream_cache_key;
            proxy_no_cache                      $upstream_no_cache;
            proxy_cache_bypass                  $upstream_cache_bypass;


            proxy_pass      $upstream_scheme://apisix_backend$upstream_uri;

            mirror          /proxy_mirror;

            header_filter_by_lua_block {
                apisix.http_header_filter_phase()
            }

            body_filter_by_lua_block {
                apisix.http_body_filter_phase()
            }

            log_by_lua_block {
                apisix.http_log_phase()
            }
        }

        location @grpc_pass {

            access_by_lua_block {
                apisix.grpc_access_phase()
            }

            # For servers which obey the standard, when `:authority` is missing,
            # `host` will be used instead. When used with apisix-base, we can do
            # better by setting `:authority` directly
            grpc_set_header   ":authority" $upstream_host;
            grpc_set_header   Content-Type application/grpc;
            grpc_socket_keepalive on;
            grpc_pass         $upstream_scheme://apisix_backend;

            header_filter_by_lua_block {
                apisix.http_header_filter_phase()
            }

            body_filter_by_lua_block {
                apisix.http_body_filter_phase()
            }

            log_by_lua_block {
                apisix.http_log_phase()
            }
        }


        location = /proxy_mirror {
            internal;



            proxy_connect_timeout 60s;
            proxy_read_timeout 60s;
            proxy_send_timeout 60s;
            proxy_http_version 1.1;
            proxy_set_header Host $upstream_host;
            proxy_pass $upstream_mirror_uri;
        }

        location @50x.html {
            set $from_error_page 'true';
            content_by_lua_block {
                require("apisix.error_handling").handle_500()
            }
            header_filter_by_lua_block {
                apisix.http_header_filter_phase()
            }

            log_by_lua_block {
                apisix.http_log_phase()
            }
        }
    }
    # http end configuration snippet starts
    
    # http end configuration snippet ends
}

@tokers
Copy link
Contributor

tokers commented Dec 27, 2022

@MirtoBusico Hi, have you ever checked the contents in /usr/local/apisix/conf/ssl/cert? Also, could you try to send requests via wget and use the CA cert in /usr/local/apisix/conf/ssl/cert.

@MirtoBusico
Copy link
Author

Hi @tokers
the content of my CA certificate is

sysop@hserv:~/H/hservcerts$ cat hservca.pem 
-----BEGIN CERTIFICATE-----
MIIEDTCCAvWgAwIBAgIUNh8YcHarQe4SyFJd0W1nMLJXE/wwDQYJKoZIhvcNAQEL
BQAwgZUxCzAJBgNVBAYTAklUMQ4wDAYDVQQIDAVJdGFseTENMAsGA1UEBwwEUm9t
ZTEVMBMGA1UECgwMQnVzaWNvIE1pcnRvMRMwEQYDVQQLDApMYWJvcmF0b3J5MRUw
EwYDVQQDDAxCdXNpY28gTWlydG8xJDAiBgkqhkiG9w0BCQEWFW1pcnRvYnVzaWNv
QGdtYWlsLmNvbTAeFw0yMjA4MTUxNTQwMDZaFw0zMjA4MTIxNTQwMDZaMIGVMQsw
CQYDVQQGEwJJVDEOMAwGA1UECAwFSXRhbHkxDTALBgNVBAcMBFJvbWUxFTATBgNV
BAoMDEJ1c2ljbyBNaXJ0bzETMBEGA1UECwwKTGFib3JhdG9yeTEVMBMGA1UEAwwM
QnVzaWNvIE1pcnRvMSQwIgYJKoZIhvcNAQkBFhVtaXJ0b2J1c2ljb0BnbWFpbC5j
b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCYCAMZo12TlF8Vho+B
MBhyvTni6NLLorflgQgoRqCzcQq9yzMTUhgVQU1kzGaV98bzeIT4knJTjEmFVyJ7
qOIZhB2bdi5I8EpjLR/ooCwyo6rdTiymJECujxbUdl74aZ1pFfDsXrsazuoA2Ut+
dQPkRcNNovpuYsTehepOBFV3gts/TuT0WfalnZS//Rtz3sTEMIV4GTi5SrU1kCYR
QxO147c0NU9Q0HvVeV57+Y469O8DCm1ZumuRTiN6ZSojM4tGIIexyhyMg/V5j+zA
XO7Wnvq9myFYXM/e7hor67rAzpkwbagvOhhn1R/DlgRGikVSrv9UhqU74CG/7AAk
iTbpAgMBAAGjUzBRMB0GA1UdDgQWBBTbb3GEP28aGXTv7bQP9jf0YZfkmzAfBgNV
HSMEGDAWgBTbb3GEP28aGXTv7bQP9jf0YZfkmzAPBgNVHRMBAf8EBTADAQH/MA0G
CSqGSIb3DQEBCwUAA4IBAQCIlTXWDhU1D0t1RM41OvVl8EhfCRFbVaVoemItNtQB
nmJ0TFeb+ARl+c8IKCNifn9U3ftA3wKoCfaQvhrjoaVDBeFbK3b5ym6C0dX6ZKXa
LsT2CyMpAoTV3Y0QKMpKMR16ZfZbB5wCXxpBdnNveWL6cBOTaJzS1BfIU0XcTD6c
bAM+nhvQSMHTaaJnLksMAE1akHcpONtFeyLO13gwgo96f21bel852dWLo99xqQJr
q+YaYx7bRIktWW1WvBNpy9wuI4llUn7ovlYjl1T084v+++tbPL6NH7UlFA5WqXdK
4gDaR3SzcCWWgnw5moMAS6u278z5xos12bSH1Yv9p68B
-----END CERTIFICATE-----
sysop@hserv:~/H/hservcerts$

On the Apisix pod the /usr/local/apisix/conf/ssl/cert is

root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# cat /usr/local/apisix/conf/ssl/cert
-----BEGIN CERTIFICATE-----
MIIEDTCCAvWgAwIBAgIUNh8YcHarQe4SyFJd0W1nMLJXE/wwDQYJKoZIhvcNAQEL
BQAwgZUxCzAJBgNVBAYTAklUMQ4wDAYDVQQIDAVJdGFseTENMAsGA1UEBwwEUm9t
ZTEVMBMGA1UECgwMQnVzaWNvIE1pcnRvMRMwEQYDVQQLDApMYWJvcmF0b3J5MRUw
EwYDVQQDDAxCdXNpY28gTWlydG8xJDAiBgkqhkiG9w0BCQEWFW1pcnRvYnVzaWNv
QGdtYWlsLmNvbTAeFw0yMjA4MTUxNTQwMDZaFw0zMjA4MTIxNTQwMDZaMIGVMQsw
CQYDVQQGEwJJVDEOMAwGA1UECAwFSXRhbHkxDTALBgNVBAcMBFJvbWUxFTATBgNV
BAoMDEJ1c2ljbyBNaXJ0bzETMBEGA1UECwwKTGFib3JhdG9yeTEVMBMGA1UEAwwM
QnVzaWNvIE1pcnRvMSQwIgYJKoZIhvcNAQkBFhVtaXJ0b2J1c2ljb0BnbWFpbC5j
b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCYCAMZo12TlF8Vho+B
MBhyvTni6NLLorflgQgoRqCzcQq9yzMTUhgVQU1kzGaV98bzeIT4knJTjEmFVyJ7
qOIZhB2bdi5I8EpjLR/ooCwyo6rdTiymJECujxbUdl74aZ1pFfDsXrsazuoA2Ut+
dQPkRcNNovpuYsTehepOBFV3gts/TuT0WfalnZS//Rtz3sTEMIV4GTi5SrU1kCYR
QxO147c0NU9Q0HvVeV57+Y469O8DCm1ZumuRTiN6ZSojM4tGIIexyhyMg/V5j+zA
XO7Wnvq9myFYXM/e7hor67rAzpkwbagvOhhn1R/DlgRGikVSrv9UhqU74CG/7AAk
iTbpAgMBAAGjUzBRMB0GA1UdDgQWBBTbb3GEP28aGXTv7bQP9jf0YZfkmzAfBgNV
HSMEGDAWgBTbb3GEP28aGXTv7bQP9jf0YZfkmzAPBgNVHRMBAf8EBTADAQH/MA0G
CSqGSIb3DQEBCwUAA4IBAQCIlTXWDhU1D0t1RM41OvVl8EhfCRFbVaVoemItNtQB
nmJ0TFeb+ARl+c8IKCNifn9U3ftA3wKoCfaQvhrjoaVDBeFbK3b5ym6C0dX6ZKXa
LsT2CyMpAoTV3Y0QKMpKMR16ZfZbB5wCXxpBdnNveWL6cBOTaJzS1BfIU0XcTD6c
bAM+nhvQSMHTaaJnLksMAE1akHcpONtFeyLO13gwgo96f21bel852dWLo99xqQJr
q+YaYx7bRIktWW1WvBNpy9wuI4llUn7ovlYjl1T084v+++tbPL6NH7UlFA5WqXdK
4gDaR3SzcCWWgnw5moMAS6u278z5xos12bSH1Yv9p68B
-----END CERTIFICATE-----
root@apisix-54cdc68f89-wtl8w:/usr/local/apisix#

(Seems equal)

Issuing wget with --ca-certificate from Apisix pod says

root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# wget -v --ca-certificate /usr/local/apisix/conf/ssl/cert https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration
--2022-12-27 08:45:55--  https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration
Loaded CA certificate '/usr/local/apisix/conf/ssl/cert'
Resolving k6k.h.net (k6k.h.net)... 192.168.100.20
Connecting to k6k.h.net (k6k.h.net)|192.168.100.20|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5928 (5.8K) [application/json]
Saving to: 'openid-configuration'

openid-configuration                                             100%[=======================================================================================================================================================>]   5.79K  --.-KB/s    in 0s      

2022-12-27 08:45:55 (326 MB/s) - 'openid-configuration' saved [5928/5928]

root@apisix-54cdc68f89-wtl8w:/usr/local/apisix#

Issuing wget without --ca-certificate from Apisix pod says

root@apisix-54cdc68f89-wtl8w:/usr/local/apisix# wget -v https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration
--2022-12-27 08:48:21--  https://k6k.h.net/realms/hcluster_admins/.well-known/openid-configuration
Resolving k6k.h.net (k6k.h.net)... 192.168.100.20
Connecting to k6k.h.net (k6k.h.net)|192.168.100.20|:443... connected.
ERROR: The certificate of 'k6k.h.net' is not trusted.
ERROR: The certificate of 'k6k.h.net' doesn't have a known issuer.
root@apisix-54cdc68f89-wtl8w:/usr/local/apisix#

Can I look at someting else?

@tokers
Copy link
Contributor

tokers commented Dec 27, 2022

hservca.pem

That's a normal situation. I want to know after you configure the CA cert to APISIX, is it normal for APISIX to communicate with your keycloak server? From the error logs you pasted:

2022/12/05 08:19:27 [warn] 49#49: 48972 [lua] plugin.lua:934: run_plugin(): openid-connect exits with http status code 500, client: 127.0.0.6, server: _, request: "GET /?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net"
2022/12/05 08:19:27 [alert] 49#49: 48972 ignoring stale global SSL error (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt), client: 127.0.0.6, server: _, request: "GET /?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net"
2022/12/05 08:19:27 [alert] 47#47: 48973 ignoring stale global SSL error (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt), client: 127.0.0.6, server: _, request: "GET /favicon.ico HTTP/1.0", host: "apisix.h.net", referrer: "https://apisix.h.net/*?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756"
2022/12/05 08:19:27 [error] 48#48: 48980 [lua] openidc.lua:1475: authenticate(): request to the redirect_uri path but there's no session state found, client: 127.0.0.6, server: _, request: "GET /?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net"
2022/12/05 08:19:27 [error] 48#48: 48980 [lua] openid-connect.lua:315: phase_func(): OIDC authentication failed: request to the redirect_uri path but there's no session state found, client: 127.0.0.6, server: _, request: "GET /?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net"
2022/12/05 08:19:27 [warn] 48#48: 48980 [lua] plugin.lua:934: run_plugin(): openid-connect exits with http status code 500, client: 127.0.0.6, server: _, request: "GET /?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net"
2022/12/05 08:19:27 [alert] 48#48: 48980 ignoring stale global SSL error (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt), client: 127.0.0.6, server: _, request: "GET /?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0", host: "apisix.h.net"
127.0.0.6 - - [05/Dec/2022:08:19:27 +0000] apisix.h.net "GET /?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0" 500 553 0.000 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" - - - "http://apisix.h.net"
127.0.0.6 - - [05/Dec/2022:08:19:27 +0000] apisix.h.net "GET /favicon.ico HTTP/1.0" 302 217 0.000 "https://apisix.h.net/*?state=809e5a967452528b8549511068b99cb1&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=63022b8e-9545-4441-8272-d429d4c8a819.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756" "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" - - - "http://apisix.h.net"
127.0.0.6 - - [05/Dec/2022:08:19:27 +0000] apisix.h.net "GET /*?state=f4130a202c1dc0ec165657fab774df10&session_state=29ba412f-4e64-4533-8ce0-0d23ad64fbcd&code=055ab546-bf9a-42b9-b28d-f19a003a12f7.29ba412f-4e64-4533-8ce0-0d23ad64fbcd.755e9ac7-b5a6-46d4-9660-fc6aa23d3756 HTTP/1.0" 500 553 0.000 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" - - - "http://apisix.h.net"

There is no direct evidence shows it's caused by an unknown CA.

@MirtoBusico
Copy link
Author

MirtoBusico commented Dec 27, 2022 •
edited
Loading

Well @tokers when I tried to use openid-connect (see #8452 ) the cause was identified in #8068

But when I discovered that a wget in the apisix pod was not able to communicate with the keyclok server I supposed that there was another problem, so I reported here

Do you think that there is no relation between the failing wget and the apisix pod accessing the keycloak server?

BTW when in the past I encountered this issue (see #6345 ) in the apisix pod I used curl and I was able to reach the keycloak server (apisix image was tagged 2.13.0-alpine ). Now curl is not installed and I had to use wget (apisix image is tagged 2.15.1-debian )
Don't know if this is relevant

@tokers
Copy link
Contributor

tokers commented Dec 27, 2022

Well @tokers when I tried to use openid-connect (see #8452 ) the cause was identified in #8068

But when I discovered that a wget in the apisix pod was not able to communicate with the keyclok server I supposed that there was another problem, so I reported here

Do you think that there is no relation between the failing wget and the apisix pod accessing the keycloak server?

BTW when in the past I encountered this issue (see #6345 ) in the apisix pod I used curl and I was able to reach the keycloak server (apisix image was tagged 2.13.0-alpine ). Now curl is not installed and I had to use wget (apisix image is tagged 2.15.1-debian )

Don't know if this is relevant

From the error log's point of view, I think it's not related to the CA.

Not sure how you invoke the curl command. 😁

@MirtoBusico
Copy link
Author

Ok @tokers I think you can close the issue
Thanks for your time

@tokers
Copy link
Contributor

tokers commented Dec 28, 2022

Ok @tokers I think you can close the issue

Thanks for your time

Did you solve the issue? :)

@MirtoBusico
Copy link
Author

Well, no
But now that the CA problem is excluded I'm trying other instructions to solve the issue (see #8452 (comment))

@tokers
Copy link
Contributor

tokers commented Dec 28, 2022

Well, no But now that the CA problem is excluded I'm trying other instructions to solve the issue (see #8452 (comment))

Got it. Let's troubleshoot it in another issue.

@tokers tokers closed this as completed Dec 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tokers @MirtoBusico