[Bug]: Function deployment is ready while ingress is not when deploy function with self-host k8s cluster

[OnceUponATimeMathley]

Nuclio Version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Nuclio.

Issue Description

I have created two k8s cluster with kubeadm for testing deploy function with k8s
1.Two physical machine
2. EC2 instance in AWS

In both case, I deploy ingress controller and nuclio for k8s successfully.

When I try deploying dummy yolov7 function for testing. This is all my source code
yolov7.zip

When research this error, I found this issues in nuclio: #2124
I try using --file argument and --verbose for debugging
This is my command: nuctl deploy --namespace nuclio --project-name cvat --path ./yolov7/nuclio/ --file ./yolov7/nuclio/function.yaml --registry docker.io/onceuponatimemathley --run-registry docker.io/onceuponatimemathley --platform kube --verbose

Below is my logs:

         processor\u001b[0m \u001b[32m(D)\u001b[0m Processor started"}}, "functionExists": true}

23.06.22 18:10:25.889 nuctl (D) Saving deployment logs
23.06.22 18:10:25.891 nuctl.platform.updater (I) Updating function {"name": "yolov7"}

Error - NuclioFunction in unhealthy state:

Error - Function deployment is ready while ingress is not
...//nuclio/pkg/platform/kube/functionres/lazy.go:328

Call stack:
Function deployment is ready while ingress is not
...//nuclio/pkg/platform/kube/functionres/lazy.go:328
Failed to wait for function resources to be available
.../platform/kube/controller/nucliofunction.go:232

...//nuclio/pkg/platform/kube/client/deployer.go:599

Call stack:
NuclioFunction in unhealthy state:

Error - Function deployment is ready while ingress is not
...//nuclio/pkg/platform/kube/functionres/lazy.go:328

Call stack:
Function deployment is ready while ingress is not
...//nuclio/pkg/platform/kube/functionres/lazy.go:328
Failed to wait for function resources to be available
.../platform/kube/controller/nucliofunction.go:232

...//nuclio/pkg/platform/kube/client/deployer.go:599

Failed to wait for function readiness.

Pod logs:

• nuclio-yolov7-6d589fcb8f-6krxg
  23.06.22 11:07:48.487 processor (D) No secret is not mounted to function pod, continuing without restoring function config
  23.06.22 11:07:48.488 processor (I) Starting processor {"version": "Label: 1.11.9, Git commit: 1f8f0c2, OS: linux, Arch: amd64, Go version: go1.17.10"}
  23.06.22 11:07:48.488 processor (D) Read configuration {"config": "{
  "metadata": {
  "name": "yolov7",
  "namespace": "nuclio",
  "labels": {
  "nuclio.io/app": "functionres",
  "nuclio.io/class": "function",
  "nuclio.io/function-name": "yolov7",
  "nuclio.io/function-version": "latest",
  "nuclio.io/project-name": "cvat"
  },
  "annotations": {
  "framework": "pytorch",
  "name": "Object Detection - YOLOv7",
  "spec": "[
  { "id": 0, "name": "person" },
  { "id": 1, "name": "bus" }
  ]
  ",
  "type": "detector"
  }
  },
  "spec": {
  "description": "YOLOv7",
  "handler": "main:handler",
  "runtime": "python:3.9",
  "env": [
  {
  "name": "PYTHONPATH",
  "value": "/opt/nuclio/yolov7"
  },
  {
  "name": "NUCLIO_CHECK_FUNCTION_CONTAINERS_HEALTHINESS",
  "value": "true"
  }
  ],
  "resources": {
  "limits": {
  "cpu": "4",
  "memory": "512M"
  },
  "requests": {
  "cpu": "2",
  "memory": "128M"
  }
  },
  "image": "docker.io/onceuponatimemathley/cvat.yolov7:latest",
  "imageHash": "1687432053147746326",
  "targetCPU": 75,
  "triggers": {
  "myHttpTrigger": {
  "class": "",
  "kind": "http",
  "name": "myHttpTrigger",
  "maxWorkers": 1,
  "workerAvailabilityTimeoutMilliseconds": 10000,
  "attributes": {
  "ingresses": {
  "path1": {
  "host": "rainscales.com",
  "pathType": "ImplementationSpecific",
  "paths": [
  "/yolov7/latest"
  ]
  }
  },
  "maxRequestBodySize": 33554432,
  "serviceType": "ClusterIP"
  }
  }
  },
  "version": -1,
  "alias": "latest",
  "build": {
  "functionConfigPath": "./yolov7/nuclio/function.yaml",
  "registry": "docker.io/onceuponatimemathley",
  "image": "cvat.yolov7",
  "baseImage": "ubuntu:22.04",
  "directives": {
  "postCopy": null,
  "preCopy": [
  {
  "kind": "ENV",
  "value": "DEBIAN_FRONTEND=noninteractive"
  },
  {
  "kind": "WORKDIR",
  "value": "/opt/nuclio"
  },
  {
  "kind": "RUN",
  "value": "apt-get update \u0026\u0026 apt-get -y install --no-install-recommends python3 python3-pip tmux ffmpeg libsm6 libxext6"
  },
  {
  "kind": "RUN",
  "value": "pip install Pillow numpy"
  },
  {
  "kind": "RUN",
  "value": "ln -s /usr/bin/python3 /usr/bin/python"
  }
  ]
  },
  "codeEntryType": "image",
  "timestamp": 1687432065
  },
  "runRegistry": "docker.io/onceuponatimemathley",
  "platform": {
  "attributes": {
  "mountMode": "volume",
  "restartPolicy": {
  "maximumRetryCount": 3,
  "name": "always"
  }
  }
  },
  "readinessTimeoutSeconds": 360,
  "securityContext": {},
  "eventTimeout": "60s"
  },
  "PlatformConfig": null
  }"}
  23.06.22 11:07:48.488 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
  23.06.22 11:07:48.488 processor.http (D) Creating worker pool {"num": 1}
  23.06.22 11:07:48.488 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cia2n12gofbc73837ls0.sock"}
  23.06.22 11:07:48.488 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cia2n12gofbc73837lsg.sock"}
  23.06.22 11:07:48.488 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
  23.06.22 11:07:48.488 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
  23.06.22 11:07:48.488 sor.http.w0.python.logger (D) Using Python executable {"path": "/usr/bin/python3"}
  23.06.22 11:07:48.488 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/yolov7"}
  23.06.22 11:07:48.488 sor.http.w0.python.logger (D) Running wrapper {"command": "/usr/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --event-socket-path /tmp/nuclio-rpc-cia2n12gofbc73837ls0.sock --control-socket-path /tmp/nuclio-rpc-cia2n12gofbc73837lsg.sock --platform-kind kube --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger --decode-event-strings"}
  /opt/nuclio/_nuclio_wrapper.py:442: DeprecationWarning: There is no current event loop
  loop = asyncio.get_event_loop()
  23.06.22 11:07:48.667 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 22}
  23.06.22 11:07:48.667 sor.http.w0.python.logger (D) Creating control connection {"wid": 0}
  23.06.22 11:07:48.667 sor.http.w0.python.logger (D) Control connection created {"wid": 0}
  23.06.22 11:07:48.667 sor.http.w0.python.logger (D) Waiting for start
  23.06.22 11:07:48.667 sor.http.w0.python.logger (D) Received control message {"messageKind": "wrapperInitialized"}
  23.06.22 11:07:48.667 sor.http.w0.python.logger (I) Init context... 0% {"worker_id": "0"}
  23.06.22 11:07:48.667 sor.http.w0.python.logger (I) Init context...100% {"worker_id": "0"}
  23.06.22 11:07:48.667 sor.http.w0.python.logger (D) Sending data on control socket {"data_length": 2, "worker_id": "0"}
  23.06.22 11:07:48.667 sor.http.w0.python.logger (D) Started
  23.06.22 11:07:48.667 processor (I) Starting event timeout watcher {"timeout": "1m0s"}
  23.06.22 11:07:48.667 .webadmin.server.triggers (D) Registered custom route {"routeName": "triggers", "stream": false, "pattern": "/{id}/stats", "method": "GET"}
  23.06.22 11:07:48.667 processor.webadmin.server (D) Registered resource {"name": "triggers"}
  23.06.22 11:07:48.667 processor (W) No metric sinks configured, metrics will not be published
  23.06.22 11:07:48.667 processor (D) Starting triggers {"triggersError": "json: unsupported value: encountered a cycle via *http.http"}
  23.06.22 11:07:48.671 processor.http (I) Starting {"listenAddress": ":8080", "readBufferSize": 16384, "maxRequestBodySize": 33554432, "reduceMemoryUsage": false, "cors": null}
  23.06.22 11:07:48.671 processor.webadmin.server (I) Listening {"listenAddress": ":8081"}
  23.06.22 11:07:48.671 processor (D) Processor started

  ...//nuclio/pkg/platform/kube/client/deployer.go:154
  Failed to deploy function
  ...//nuclio/pkg/platform/abstract/platform.go:215

---

This similar to recently issue: #2952

**The yolov7 function status is changing between running and unhealthy periodically, but when I check pod, is not restart and I can using ingress to call function by : http://node_ip:node_port/yolov7/latest despite of status of yolov7 function
NAMESPACE NAME READY STATUS RESTARTS AGE
nuclio nuclio-yolov7-65bb5b75f-pc9cd 1/1 Running 0 3m8s

root@master:~# kubectl get ingress -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
nuclio nuclio-yolov7 traefik rainscales.com 80 6m55s**

Expected Behavior

In my self-host k8s cluster, I have no using load balancer and pending with external ip but using nodeport I work in physical machine and in EC2 instance

Deployment Method

Kubernetes

Nuclio Version

1.8.14, 1.11.21

Additional Information

No response