/metrics (Prometheus) endpoint always returning a value of 0

[sillyfrog]

Report

Running Keda v2.8.0 the /metrics endpoint is always returning a value of zero. I have both AWS SQS Queue and Cron scalers.

My deployment is using the kedacore/keda v2.8.1 helm chart.

If I deploy the helm chart version v2.7.2, things work as expected.

Expected Behavior

Correct values be reported to the /metrics endpoint.

Actual Behavior

Values to the /metrics endpoint are always zero.

Steps to Reproduce the Problem

1. Configure a ScaledObject with a Cron Scaler, ensure the time selection is in range, running Keda v2.7.1, and deploy eg:

  triggers:
  - metadata:
      desiredReplicas: "3"
      end: 0 23 * * *
      start: 0 1 * * *
      timezone: Australia/Brisbane
    type: cron

2. Look at the /metrics endpoint, eg:

curl keda-operator-metrics-apiserver.keda.svc.cluster.local:9022/metrics
# HELP keda_metrics_adapter_scaler_metrics_value Metric Value used for HPA
# TYPE keda_metrics_adapter_scaler_metrics_value gauge
keda_metrics_adapter_scaler_metrics_value{metric="s0-cron-australia-brisbane-01xxx-023xxx",namespace="coreapi",scaledObject="coreapi-queue",scaler="cronScaler",scalerIndex="0"} 3

3. Update Keda to version v2.8.0, and run the curl again:

curl keda-operator-metrics-apiserver.keda.svc.cluster.local:9022/metrics
# HELP keda_metrics_adapter_scaler_metrics_value Metric Value used for HPA
# TYPE keda_metrics_adapter_scaler_metrics_value gauge
keda_metrics_adapter_scaler_metrics_value{metric="s0-cron-australia-brisbane-01xxx-023xxx",namespace="coreapi",scaledObject="coreapi-queue",scaler="cronScaler",scalerIndex="0"} 0

Note how the value is now zero at the end of the last line.

Note: These outputs have been trimmed for clarity.

Logs from KEDA operator

No response

KEDA Version

2.8.0

Kubernetes Version

1.22

Platform

Amazon Web Services

Scaler Details

Cron and AWS SQS

Anything else?

Scaling still works as expected.

Of note, if running a query via kubectl the value is correct, eg:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/coreapi/s0-cron-australia-brisbane-01xxx-023xxx?labelSelector=scaledobject.keda.sh%2Fname%3Dcoreapi-queue"
{"kind":"ExternalMetricValueList","apiVersion":"external.metrics.k8s.io/v1beta1","metadata":{},"items":[{"metricName":"s0-cron-australia-brisbane-01xxx-023xxx","metricLabels":null,"timestamp":"2022-08-15T05:17:43Z","value":"3"}]}

[zerodayyy]

I've got the same issue on Keda 2.8.0, with Azure Storage Queue scaler

# HELP keda_metrics_adapter_scaler_metrics_value Metric Value used for HPA
# TYPE keda_metrics_adapter_scaler_metrics_value gauge
keda_metrics_adapter_scaler_metrics_value{metric="s1-azure-queue-redacted-01,namespace="redacted01",scaledObject="redacted01",scaler="azureQueueScaler",scalerIndex="1"} 0
keda_metrics_adapter_scaler_metrics_value{metric="s1-azure-queue-redacted-02",namespace="redacted02",scaledObject="redacted02,scaler="azureQueueScaler",scalerIndex="1"} 0
keda_metrics_adapter_scaler_metrics_value{metric="s1-azure-queue-redacted-03",namespace="redacted03",scaledObject="redacted03",scaler="azureQueueScaler",scalerIndex="1"} 0

For comparison, same metrics on KEDA 2.7.1:

# HELP keda_metrics_adapter_scaler_metrics_value Metric Value used for HPA
# TYPE keda_metrics_adapter_scaler_metrics_value gauge
keda_metrics_adapter_scaler_metrics_value{metric="s1-azure-queue-redacted-01,namespace="redacted01",scaledObject="redacted01",scaler="azureQueueScaler",scalerIndex="1"} 2580
keda_metrics_adapter_scaler_metrics_value{metric="s1-azure-queue-redacted-02",namespace="redacted02",scaledObject="redacted02,scaler="azureQueueScaler",scalerIndex="1"} 311
keda_metrics_adapter_scaler_metrics_value{metric="s1-azure-queue-redacted-03",namespace="redacted03",scaledObject="redacted03",scaler="azureQueueScaler",scalerIndex="1"} 2

[zroubalik]

Thanks for reporting, do you think you can try to investigate what is the cause of the problem?