Containers in a kind cluster are not being reported in Datadog

[arapulido]

Describe what happened:

I have created a 3-node kind cluster running 1.18 and I am running datadog-operator v0.2.1 on it.

I deploy the node agent using the following configuration:

apiVersion: datadoghq.com/v1alpha1
kind: DatadogAgent
metadata:
  name: datadog-agent
  namespace: datadog
spec:
  credentials:
    apiKeyExistingSecret: secretapi
    appKeyExistingSecret: secretapp
  agent:
    image:
      name: "datadog/agent:latest"
    config:
      logLevel: "DEBUG"
      leaderElection: true
      tolerations:
      - operator: Exists
      criSocket:
        criSocketPath: /var/run/containerd/containerd.sock
        useCriSocketVolume: true
      env:
      - name: DD_KUBELET_TLS_VERIFY
        value: "false"
      - name: DD_KUBERNETES_KUBELET_HOST
        valueFrom:
          fieldRef:
            fieldPath: status.hostIP
    process:
      enabled: true

Describe what you expected:

I expect to get container information in Datadog. I get this instead:

I get process information correctly:

Steps to reproduce the issue:

1. Create a 3 node kind cluster:

kind create cluster --name datadog-operator  --config cluster-config.yaml

cluster-config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.18.0
- role: worker
  image: kindest/node:v1.18.0
- role: worker
  image: kindest/node:v1.18.0

2. Deploy datadog operator v.0.2.1 using Helm 3
3. Apply the agent configuration described in the issue

[arapulido]

agent status and process-agent -check container outputs:

process-container-status.txt
status.txt

[L3n41c]

Indeed, good catch @arapulido !

In fact, what happened was that some environment variables were defined only for the core agent container whereas they were needed by the process agent as well.

I did a PR for the environment variables that are automatically managed by the operator (DD_CRI_SOCKET_PATH and DOCKER_HOST to be more precise) but the same issue applies to the user defined DD_KUBELET_TLS_VERIFY environment variable.

Environment variables defined in spec.agent.config.env are applied only to the core agent.
So, in my PR, I added a new parameter spec.agent.env for environment variables that have to be applied to all agents.

So, once you’ll have upgraded to a version of the operator that has my change, you’ll need to move the definition of DD_KUBELET_TLS_VERIFY from spec.agent.config.env to spec.agent.env.

[arapulido]

@L3n41c Thanks! And thanks so much for taking the time to explain the root cause and the fix. Really appreciated! ❤️