unable to fetch metrics from external metrics API: no matches for kind "ScaledObject" in version "keda.k8s.io/v1alpha1" (Scaler : Prometheus)

[wamak9]

Autoscaling is working okay but I keep seeing error in hpa when describing it

Warning FailedComputeMetricsReplicas 44m (x12 over 47m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get nginx_ingress_controller_requests external metric: unable to get external metric demo/nginx_ingress_controller_requests/&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matches for kind "ScaledObject" in version "keda.k8s.io/v1alpha1" Warning FailedGetExternalMetric 37m (x40 over 47m) horizontal-pod-autoscaler unable to get external metric demo/nginx_ingress_controller_requests/&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matches for kind "ScaledObject" in version "keda.k8s.io/v1alpha1" Warning FailedGetExternalMetric 31m horizontal-pod-autoscaler unable to get external metric demo/nginx_ingress_controller_requests/&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get nginx_ingress_controller_requests.external.metrics.k8s.io) Normal SuccessfulRescale 21m horizontal-pod-autoscaler New size: 4; reason: external metric nginx_ingress_controller_requests(&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}) above target Normal SuccessfulRescale 21m horizontal-pod-autoscaler New size: 8; reason: external metric nginx_ingress_controller_requests(&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}) above target Normal SuccessfulRescale 8m51s horizontal-pod-autoscaler New size: 8; reason: All metrics below target Normal SuccessfulRescale 8m5s horizontal-pod-autoscaler New size: 7; reason: All metrics below target Normal SuccessfulRescale 38s horizontal-pod-autoscaler New size: 4; reason: external metric nginx_ingress_requests(&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}) above target Normal SuccessfulRescale 22s horizontal-pod-autoscaler New size: 8; reason: external metric nginx_ingress_requests(&LabelSelector{MatchLabels:map[string]string{deploymentName: autoscalingdemo,},MatchExpressions:[]LabelSelectorRequirement{},}) above target

Expected Behavior

Working as expected but would be more clear to understand what actual error is for on the hpa side.
Ignoring the error did not sound like a good option.

Actual Behavior

scaling is working as expected as mentioned below

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE keda-hpa-autoscalingdemo Deployment/autoscalingdemo 45/3 (avg) 1 10 4 26m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 22500m/3 (avg) 1 10 8 26m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 16600m/3 (avg) 1 10 10 27m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 13800m/3 (avg) 1 10 10 27m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 18/3 (avg) 1 10 10 27m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 8700m/3 (avg) 1 10 10 28m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 0/3 (avg) 1 10 10 28m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 1900m/3 (avg) 1 10 10 29m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 17900m/3 (avg) 1 10 10 29m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 18/3 (avg) 1 10 10 30m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 17900m/3 (avg) 1 10 10 30m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 6200m/3 (avg) 1 10 10 30m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 6200m/3 (avg) 1 10 10 30m keda-hpa-autoscalingdemo Deployment/autoscalingdemo 0/3 (avg) 1 10 10 30m

In this case when hpa is created throws me an error saying
unable to fetch metrics from external metrics API: no matches for kind "ScaledObject" in version "keda.k8s.io/v1alpha1"
unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get nginx_ingress_controller_requests.external.metrics.k8s.io)
and when you hit the application as you can see above autoscaling works.

Is this an expected behaviour when first creating hpa ?
Are there any additional troubleshooting steps which can be performed to see why this is happening ?

scaled-objects yaml file

Using prometheus version : 2.16.0

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaledobject
  labels:
    app: autoscalingdemo
    deploymentName: autoscalingdemo
spec:
  scaleTargetRef:
    deploymentName: autoscalingdemo
  pollingInterval: 15  # Optional. Default: 30 seconds
  cooldownPeriod:  30 # Optional. Default: 300 seconds
  minReplicaCount: 1 # Optional. Default: 0
  maxReplicaCount: 10 # Optional. Default: 100
  triggers:
  - type: prometheus
    metadata:
      # Required
      serverAddress: https://prometheus.<xxxx>.<xxxxxxx>.com
      metricName: nginx_ingress_requests
      threshold: '3'
      query: sum(rate(nginx_ingress_controller_requests[1m]))

Steps to Reproduce the Problem

1.deploy any demo app with ingress using helm and deploy it on a desired namespace(In this case used nginx alpine image)
2. create a scaled object and deploy it on a namespace desired(should be same as helm demo app)
3.kubectl describe hpa should show you above error message when you trigger a autoscale (used hey and vegeta)

Let me know if any additional information is needed ?

Other things

Installed keda using kubectl apply.

Specifications

• KEDA Version: 1.4.1
• Platform & Version: Azure AKS
• Kubernetes Version: 1.16.7
• Scaler(s): Prometheus Scaler

[zroubalik]

Strange, is your Keda Metrics Server pod running correctly?

[wamak9]

Strange, is your Keda Metrics Server pod running correctly?

They are working fine and autoscaling is also working fine but not sure about the error.

keda-metrics-apiserver-768c487965-87j96 1/1 Running 0 keda-operator-664db9c799-wzr57 1/1 Running 0
Logs looks okay too for Operator
2020-06-03T10:15:00.976848497Z {"level":"info","ts":1591179300.976541,"logger":"controller_scaledobject","msg":"Detected ScaleType = Deployment","Request.Namespace":"demo","Request.Name":"prometheus-scaledobject"} 2020-06-03T10:15:15.691784217Z {"level":"info","ts":1591179315.691503,"logger":"controller_scaledobject","msg":"Reconciling ScaledObject","Request.Namespace":"demo","Request.Name":"prometheus-scaledobject"} 2020-06-03T10:15:15.691834417Z {"level":"info","ts":1591179315.6915674,"logger":"controller_scaledobject","msg":"Detected ScaleType = Deployment","Request.Namespace":"demo","Request.Name":"prometheus-scaledobject"} 2020-06-03T10:20:09.573293816Z {"level":"info","ts":1591179609.573071,"logger":"controller_scaledobject","msg":"Reconciling ScaledObject","Request.Namespace":"demo","Request.Name":"prometheus-scaledobject"} 2020-06-03T10:20:09.573359317Z {"level":"info","ts":1591179609.5731223,"logger":"controller_scaledobject","msg":"Detected ScaleType = Deployment","Request.Namespace":"demo","Request.Name":"prometheus-scaledobject"}

Logs for Metric Server
: 2020-06-03 15:46:15.158191886 +0000 UTC m=+144970.720830419) (total time: 702.282623ms): 2020-06-03T15:46:15.860912312Z Trace[685758675]: [702.182222ms] [702.107622ms] Listing from storage done 2020-06-03T15:53:41.921134439Z I0603 15:53:41.920820 1 trace.go:116] Trace[1072422181]: "List /apis/external.metrics.k8s.io/v1beta1/namespaces/demo/nginx_ingress_requests" (started: 2020-06-03 15:53:40.40745358 +0000 UTC m=+145415.970092213) (total time: 1.513335257s): 2020-06-03T15:53:41.92118584Z Trace[1072422181]: [1.513249656s] [1.513164255s] Listing from storage done 2020-06-03T16:02:42.811045559Z I0603 16:02:42.810868 1 trace.go:116] Trace[859921853]: "List /apis/external.metrics.k8s.io/v1beta1/namespaces/demo/nginx_ingress_requests" (started: 2020-06-03 16:02:42.202681838 +0000 UTC m=+145957.765320471) (total time: 608.153919ms): 2020-06-03T16:02:42.811082959Z Trace[859921853]: [608.075219ms] [607.986419ms] Listing from storage done 2020-06-03T16:14:45.626292342Z I0603 16:14:45.626090 1 trace.go:116] Trace[42538795]: "List /apis/external.metrics.k8s.io/v1beta1/namespaces/demo/nginx_ingress_requests" (started: 2020-06-03 16:14:45.002629115 +0000 UTC m=+146680.565267548) (total time: 623.426225ms): 2020-06-03T16:14:45.626343242Z Trace[42538795]: [623.341224ms] [623.278624ms] Listing from storage done 2020-06-03T16:33:03.235852918Z I0603 16:33:03.235446 1 trace.go:116] Trace[1385210417]: "List /apis/external.metrics.k8s.io/v1beta1/namespaces/demo/nginx_ingress_requests" (started: 2020-06-03 16:33:02.53002827 +0000 UTC m=+147778.092666803) (total time: 705.382845ms): 2020-06-03T16:33:03.235895719Z Trace[1385210417]: [705.289044ms] [705.198643ms] Listing from storage done 2020-06-03T17:40:12.257012409Z I0603 17:40:12.256876 1 trace.go:116] Trace[1555270060]: "List /apis/external.metrics.k8s.io/v1beta1/namespaces/demo/nginx_ingress_requests" (started: 2020-06-03 17:40:10.903763642 +0000 UTC m=+151806.466402075) (total time: 1.353080065s): 2020-06-03T17:40:12.257054709Z Trace[1555270060]: [1.353000565s] [1.352925964s] Listing from storage done 2020-06-03T17:40:27.90467648Z I0603 17:40:27.904480 1 trace.go:116] Trace[1357217039]: "List /apis/external.metrics.k8s.io/v1beta1/namespaces/demo/nginx_ingress_requests" (started: 2020-06-03 17:40:27.289351966 +0000 UTC m=+151822.851990499) (total time: 615.104212ms): 2020-06-03T17:40:27.90471378Z Trace[1357217039]: [615.012512ms] [614.926011ms] Listing from storage done

[kalyan123i]

I also have the same problem , it works as expected in HPA but , lot of spam in logs. We are using promethus in another name space. Our promethus is centralized.

[kalyan123i]

@akramwaseem @zroubalik I was able to resolve it by using the setting in new operator.

metricsServer:
useHostNetwork: false

so it will not use your vnet DNS server to loop the prometheus host

[zroubalik]

@kalyan123i great! Thanks for sharing :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[adi658]

@akramwaseem @zroubalik I was able to resolve it by using the setting in new operator.

metricsServer: useHostNetwork: false

so it will not use your vnet DNS server to loop the prometheus host

@kalyan123i @zroubalik - where did you add that?? In keda-operator pod? if yes what line or after which line?? Pls help