[Bug]: Function ingress is not ready yet

[paul-wie]

Nuclio Version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Nuclio.

Issue Description

I have installed Nuclio 1.11.20 on Kubernetes and everything works as expected.
But when I deploy a function with ingress the function will not work properly anymore, it will get in an unhealthy state (without the ingress config the function works).

This is the ingress config:

            "triggers": {
                "default-http": {
                    "class": "nginx",
                    "kind": "http",
                    "name": "default-http",
                    "maxWorkers": 5,
                    "attributes": {
                        "ingresses": {
                            "my-ingress": {
                                "host": "10.195.9.71.sslip.io",
                                "pathType": "ImplementationSpecific",
                                "paths": [
                                    "/hallo"
                                ]
                            }
                        },
                        "serviceType": "ClusterIP"
                    }
                }
            }

The Kubernetes ingress is created properly:

Name:             nuclio-nuclio-function
Labels:           nuclio.io/app=functionres
                  nuclio.io/class=function
                  nuclio.io/function-name=nuclio-function
                  nuclio.io/function-version=latest
                  nuclio.io/project-name=default
Namespace:        default
Address:          
Ingress Class:    traefik
Default backend:  <default>
Rules:
  Host                  Path  Backends
  ----                  ----  --------
  10.195.9.71.sslip.io  
                        /hallo   nuclio-nuclio-function:http (10.244.2.68:8080)
Annotations:            nginx.ingress.kubernetes.io/configuration-snippet: proxy_set_header X-Nuclio-Target "nuclio-function";
Events:                 <none>

I can also invoke the function:

https://10.195.9.71.sslip.io:30100/hallo/latest

So the function and the ingress works properly.
But somehow the Nuclio-Controller keeps restarting the function as it thinks the Function ingress is not ready:

controller.functionres (W) Function ingress is not ready yet, continuing {"err": "Function ingress is not ready yet", "namespace": "default", "name": "nuclio-function"}

Nuclio will then mark the function as unhealthy. The Nginx ingress has the same behaviour.

The build logs of the functions says:

 [13:44:40.026]  (I)  Deploying function   [requestID: "nuclio-dashboard-66656cb8bc-kf767/6X4TQ3CAqA-001340"] 
 [13:44:40.026]  (I)  Skipping build   [requestID: "nuclio-dashboard-66656cb8bc-kf767/6X4TQ3CAqA-001340"] 
 [13:46:40.364]  (W)  Function creation failed, updating function status   [errorStack: "
Error - NuclioFunction in unhealthy state:

Error - context deadline exceeded
    .../platform/kube/controller/nucliofunction.go:232

Call stack:
Failed to wait for function resources to be available
    .../platform/kube/controller/nucliofunction.go:232

    ...//nuclio/pkg/platform/kube/client/deployer.go:569

Call stack:
NuclioFunction in unhealthy state:

Error - context deadline exceeded
    .../platform/kube/controller/nucliofunction.go:232

Call stack:
Failed to wait for function resources to be available
    .../platform/kube/controller/nucliofunction.go:232

    ...//nuclio/pkg/platform/kube/client/deployer.go:569
Failed to wait for function readiness.

Pod logs:

* nuclio-nuclio-function-5d69d89c94-6mqzd
Function is listening on port 8080

    ...//nuclio/pkg/platform/kube/client/deployer.go:154
", requestID: "nuclio-dashboard-66656cb8bc-kf767/6X4TQ3CAqA-001340"

So the error messages and the logs are very limited and it seems to be very hard to figure out what's the problem with the current information.
The setup was done with the information on https://nuclio.io/docs/latest/concepts/k8s/function-ingress/. But the documentation also seems to be outdated as the installation works not as described, one has to visit the official Traefik website for installation.

Expected Behavior

I expect Nuclio to properly create and recognise the ingress rule and to to mark the function as unhealthy.

Deployment Method

Kubernetes

Nuclio Version

1.11.20

Additional Information

No response

[OnceUponATimeMathley]

I have same issues when add ingress config in .yaml file, the function when using nuctl deploy have same output error but I still call to function endpoint but after amount of time, the function change to unhealthy and after it's change to running (this looping repeat)

[TomerShor]

Hey @paul-wie,
A couple of questions:

1. What kubernetes version do you have?
2. Are you using the Nuclio API or nuctl?

The way Nuclio verifies ingress readiness is by getting the ingress, and checking one of the following values is populated: ingress.Status.LoadBalancer.Ingress.IP and ingress.Status.LoadBalancer.Ingress.Host.
Can you please share the output of the above values of the created ingress you see in your cluster?
It should be something like this:

$ kubectl -n <tenant-name> get ingress nuclio-<function-name> -o yaml

...
status:
  loadBalancer:
    ingress:
    - ip: <some-ip>

While we investigate what's going on, there is a workaround to bypass the fail-fast mechanism and let the Nuclio controller keep waiting for the ingress to be created:

• In the function configuration, set the following: spec.waitReadinessTimeoutBeforeFailure: false.
• You can modify the readiness timeout duration by changing the value (integer) of spec.readinessTimeoutSeconds, e.g: readinessTimeoutSeconds: 300

Please try the above workaround and answer the questions so we can get more information on what's going on.
CC: @OnceUponATimeMathley

[csvlrs-lak]

Hey @TomerShor , even I'm facing same issue, here is the output of kubectl -n get ingress nuclio- -o yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: proxy_set_header X-Nuclio-Target
      "copilot1";
  creationTimestamp: "2024-02-27T08:12:51Z"
  generation: 1
  labels:
    nuclio.io/app: functionres
    nuclio.io/class: function
    nuclio.io/function-name: function1
    nuclio.io/function-version: latest
    nuclio.io/project-name: project1
  name: nuclio-function1
  namespace: nuclio
  resourceVersion: "54131536"
  uid: 6ba17dbe-2102-4fc3-bee8-1f30f597b360
spec:
  rules:
  - host: nucliocheck.centralus.cloudapp.azure.com
    http:
      paths:
      - backend:
          service:
            name: nuclio-function1
            port:
              name: http
        path: function1
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

[paul-wie]

I was using kubelet=1.25.3-00 kubeadm=1.25.3-00 kubectl=1.25.3-00
I evaluated Nuclio for a student project but I'm not using it anymore

[csvlrs-lak]

thanks for the reply @paul-wie. I was looking for the fix because issue still exists.