When specifying/updating cloudStorage in the CRD, the cluster config is not updated

[patrickangeles]

Version & Environment

Redpanda version: 22.1.5

What went wrong?

In Kubernetes, using the operator, when you specify cloudStorage parameters to enable shadow indexing, the values do not make it into centralized config. (So they don't take effect.)

[nicolaferraro]

Just checking it on a test cluster. I see values are correctly injected into the central config.

Using the following snippet:

# ...
  cloudStorage:
    enabled: true
    accessKey: "anaccesskey"
    secretKeyRef:
      name: asecretkey
      namespace: default
    region: eu-west-1
    bucket: abucket
# ...

redpanda-controller-manager-7f65c4db78-nbrhq manager 2022-08-08T09:31:35.221Z	INFO	controllers.redpanda.Cluster	Applying patch to the cluster configuration	{"redpandacluster": "default/example", "patch": "+cloud_storage_access_key +cloud_storage_bucket +cloud_storage_enabled +cloud_storage_region +cloud_storage_secret_key"}

I can also find them using rpk:

$ kubectl exec example-0 -c redpanda -- rpk cluster config get cloud_storage_enabled
true

What's the effect that you see. Are you checking the values with rpk or you see that shadow indexing is not enabled?

In case rpk gives the right values and shadow index does not work, maybe the cluster needs a restart (there's an issue for which need for restart is not always catched, see #5835).

Another thing I noticed is that the cluster is undergoing a double restart, one for changing the pod spec to include a volume for indexing, another for enabling shadow indexing (the one that may fail to be done).