Keda-operator timing out for large number of configmaps and secrets

[ckuduvalli]

Report

Currently, whenever the cluster size is large and when the number of configmaps and secrets are huge in number, the keda-operator is timing out with the below error from the logs:

E0826 13:19:55.312899 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 1005; INTERNAL_ERROR

When we debugged it, this call is being made internally from client-go code. Then we figured out that if we disableCompression in client-go, the calls to list configmaps or secrets will significantly reduce and we won't be seeing these timeouts.

Expected Behavior

Timeouts should not happen within keda operator for list calls

Actual Behavior

We are seeing timeouts and reconciliation does not happen in keda-operator whenever there are large number of configmaps or secrets in the cluster.

Steps to Reproduce the Problem

1. Create large number of secrets or configmaps in the cluster.
2. Restart keda-operator. We saw this error happening

Logs from KEDA operator

E0826 13:19:55.312899 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 1005; INTERNAL_ERROR

KEDA Version

2.8.1

Kubernetes Version

< 1.23

Platform

Other

Scaler Details

Cpu/memory

Anything else?

No response

[zroubalik]

Related k8s upstream issue: kubernetes/kubernetes#112296