Running the operator in a different namespace than the runner

[saviogl]

I've deployed the operator in a different namespace (operations) from the actual Terraform resource (default). The idea was to manage centralized configuration like git-ssh-key and aws provider secret close to the operator instead of having to deploy these in all the namespaces where we would be managing Terraform resources.

This setup doesn't work as the terraform-runner Job can't reference the required resources:

- kind: ConfigMap
  name: known-hosts
- kind: Secret
  name: git-ssh-key
- kind: Secret
  name: terraform-aws (provider cred)

which live outside of its own namespace (default). This issue is expected as K8s doesn't allow cross-namespace secret reference, but it means that currently we'd have to deploy the operator in all targeted namespaces (if this is the case the operator should document this).

Did I miss some configuration to enable this setup? If not, is worth thinking through a way of achieving it?

If the runner runs within the operator namespace instead this setup could work. We could just use K8s Events and Status field on the Terraform CRD to notify progress

[IbraheemAlSaady]

@saviogl first of all, thank you for starting a discussion here and being active on using the Operator :)

The operator is just a Kubernetes controller, it watches the Terraform object no matter which namespace it's deployed in. That means you only need one controller in your cluster

For your setup to work, you need the ConfigMaps and Secrets to be deployed in the same namespace where your Terraform object is created, in your case, its default

Reason for that is Terraform objects create Kubernetes jobs to execute the Terraform flow, if secrets & config maps are present where the job is, it wouldn't be an issue

[saviogl]

Hi @IbraheemAlSaady thanks for the quick reply! I'm fairly familiar with the operator pattern, we run quite a few of them at work.

Appreciate the comment but it feels like you just re-stated what I had written above 😄. My purpose with this thread were twofold:

1. See if there was some configuration in the controller to allow that particular deployment setup - I see there's none
2. Discuss whether or not it is worth to consider enhancing the operator to cater to that deployment pattern

With the way the operator and the runner currently runs, in a setup where resources are deployed to multiple namespaces (one per team, area, environment), platform is required to bundle the operator (controller) into all of the namespaces. Though this can work well with some automation, I wonder if enabling a simpler dependency setup wouldn't be of interest.

i.e. Running the Terraform flow within the Operator namespace

[IbraheemAlSaady]

@saviogl As I understand, the case is not to duplicate secrets/configmaps across namespaces, rather, a centerlized place for these secrets/configmaps with the ability for Terraform objects/runner to consume them even if they're in different namespaces

If that is the case, I would argue that this is outside the scope of the Terraform Operator. If we decide to extend the functionality to support cloning/referencing secrets/configmaps from other namespaces, the project would get way more complex. Also, it defeats the responsibility of only handling Terraform workflows since it needs to manage secrets/configmaps

Distributing secrets/configmaps across namespaces can be done via another operator, something similar to this project here. As you said, it can also be handled with some automation to push these configurations to the desired list of namespaces

In your case, I would recommend that even if different teams/areas/environments have the same config, duplicating these configurations and secrets in their own namespace is not a bad thing. They can have full control of their flows and configuration