[kubelet] port cadvisor metric collection from legacy kubernetes check #1339

xvello · 2018-04-04T15:22:32Z

What does this PR do?

Port the cadvisor collection logic from agent5's kubernetes check to agent6. This PR:

adds the ContainerFilter helper class to consume the new agent interface. Only used for cadvisor mode for now, prometheus mode will be patched in another PR
factors-out common parts to a common.py file
copy agent5 cadvisor logic to a CadvisorScraper helper class for clearer separation
update the cadvisor logic to support agent6 facilities (tagger, filter)
update the cadvisor logic to report network metrics at the pod cardinality for consistency with prometheus mode (change from agent5): see comment lower
add the missing disk metric in the kubernetes.csv metadata file

Motivation

Support legacy k8s clusters

Testing Guidelines

Pushed the datadog/agent-dev:xvello-test1-3 and datadog/agent-dev:xvello-test1-3-jmx images for testing

Cadvisor mode can be triggered with the following confd configmap:

  confd:
    kubelet.yaml: |-
      init_config:
      instances:
        - cadvisor_port: 4194

Versioning

Bumped the check version in manifest.json
Bumped the check version in datadog_checks/{integration}/__init__.py
Updated CHANGELOG.md. Please use Unreleased as the date in the title
for the new section.
If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo

Additional Notes

Anything else we should know when reviewing?

xvello · 2018-04-12T12:49:21Z

As autodiscovering prometeus/cadvisor might be pretty tricky, we aggreed on going with manual configuration. This can be done by overriding the kubelet.d/conf.yaml.default by a custom kubelet.yaml. For example, with the helm chart:

  confd:
    kubelet.yaml: |-
      init_config:
      instances:
        - cadvisor_port: 4194

l0k0ms · 2018-04-12T12:51:44Z

README.md

@@ -119,6 +119,8 @@ the new testing approach:

 For checks that are not listed here, please refer to [Legacy development Setup](docs/dev/legacy.md).

+If you updated the test requirements for a check, you will need to run `tox --recreate` for changes to be effective.


r/you will need to run `tox --recreate` for changes/run `tox --recreate` for changes

l0k0ms · 2018-04-12T12:53:10Z

kubelet/conf.yaml.example

+    ###
+    ### Metric collection for legacy (< 1.7.6) clusters via the kubelet's
+    ### cadvisor port.
+    ### This port is closed by default on k8s 1.7 and OpenShift, make sure


r/make sure you enable it/enable it

l0k0ms · 2018-04-12T12:58:00Z

I left some minor nit phrasing comment to follow the contributing guidelines: https://github.com/DataDog/documentation/blob/master/CONTRIBUTING.md

It seems that we are collecting a new metric:

kubernetes.diskio.io_service_bytes.stats.total

should we update https://github.com/DataDog/integrations-core/blob/master/kubelet/metadata.csv or https://github.com/DataDog/integrations-core/blob/master/kubernetes/metadata.csv accordingly?

xvello · 2018-04-12T13:28:38Z

Cadvisor exposes network metrics per container, but as there's one network namespace per pod, the new endpoint exposes them per pod. The agent6 cadvisor mode will align with prometheus mode (that will have correct host sums), and break compat with agent5.

Agent5, sending one gauge per container in the pod, only pause container has a non-zero value

 (u'kubernetes.network.tx_bytes',
  1523537701,
  0.0,
  {'hostname': u'gke-xvello-19-default-pool-be004370-pjvz.c.datadog-sandbox.internal',
   'tags': [u'image_tag:1.14.8',
            u'image_name:asia.gcr.io/google_containers/k8s-dns-sidecar-amd64',
            'agent:5',
            u'kube_k8s-app:kube-dns',
            u'kube_namespace:kube-system',
            u'kube_container_name:sidecar',
            u'kube_replica_set:kube-dns-5c5884448b',
            u'pod_name:kube-dns-5c5884448b-z4d79',
            u'kube_replication_controller:kube-dns-5c5884448b',
            u'container_name:k8s_sidecar_kube-dns-5c5884448b-z4d79_kube-system_4874125c-3363-11e8-a97b-42010a840074_1',
            u'kube_pod-template-hash:1714400046',
            u'kube_deployment:kube-dns',

Agent6 + prometeus: correctly tagging by pod tags only

      "metric": "kubernetes.network.tx_bytes",
      "points": [
        [
          1523537810,
          0
        ]
      ],
      "tags": [
        "kube_deployment:kube-dns",
        "kube_namespace:kube-system",
        "kube_replica_set:kube-dns-5c5884448b",
        "pod_name:kube-dns-5c5884448b-z4d79"
      ],
      "host": "gke-xvello-19-default-pool-be004370-pjvz.c.datadog-sandbox.internal",
      "type": "gauge",
      "interval": 0,

Agent6 + cadvisor : move the network metric to pod level, mirroring prometeus mode

      "metric": "kubernetes.network.tx_bytes",
      "points": [
        [
          1523546563,
          0
        ]
      ],
      "tags": [
        "agent:6",
        "kube_daemon_set:fluentd-gcp-v2.0.10",
        "kube_namespace:kube-system",
        "pod_name:fluentd-gcp-v2.0.10-csz7w"
      ],
      "host": "gke-xvello-19-default-pool-be004370-pjvz.c.datadog-sandbox.internal",
      "type": "gauge",
      "interval": 0,
      "source_type_name": "System"
    },

JulienBalestra · 2018-04-13T12:50:41Z

kubelet/datadog_checks/kubelet/cadvisor.py

+        self._update_metrics(instance)
+
+    def _update_metrics(self, instance):
+        def parse_quantity(s):


Why do you define this function in this method ?
This function isn't used.

dead-code copy-pasted from agent5, removing

JulienBalestra · 2018-04-13T12:54:50Z

kubelet/datadog_checks/kubelet/cadvisor.py

+            except Exception as e:
+                self.log.error("Unable to collect metrics for container: {0} ({1})".format(c_id, e))
+
+    def _publish_raw_metrics(self, metric, dat, tags, is_pod, depth=0):


Is it possible to have a docstrings for this method ?
Like:

def _publish_raw_metrics(self, metric, dat, tags, is_pod, depth=0): """ Blahblah metric: type dat: type ... """

JulienBalestra · 2018-04-13T13:16:23Z

kubelet/datadog_checks/kubelet/cadvisor.py

+LEGACY_CADVISOR_METRICS_PATH = '/api/v1.3/subcontainers/'
+
+
+class CadvisorScraper():


Why not using the new style Python 2.7 class:

class CadvisorScraper(object):

WDYT ?

JulienBalestra · 2018-04-13T13:18:53Z

kubelet/datadog_checks/kubelet/cadvisor.py

+        return url
+
+    def retrieve_cadvisor_metrics(self, timeout=10):
+        return requests.get(self.cadvisor_legacy_url, timeout=timeout).json()


Is self.cadvisor_legacy_url attribute defined in this class ?

JulienBalestra · 2018-04-13T13:19:47Z

kubelet/datadog_checks/kubelet/cadvisor.py

+        metrics = self.retrieve_cadvisor_metrics()
+
+        if not metrics:
+            raise Exception('No metrics retrieved cmd=%s' % self.metrics_cmd)


Is self.metrics_cmd attribute defined in this class ?

JulienBalestra · 2018-04-13T13:21:15Z

kubelet/datadog_checks/kubelet/cadvisor.py

+            try:
+                self._update_container_metrics(instance, subcontainer)
+            except Exception as e:
+                self.log.error("Unable to collect metrics for container: {0} ({1})".format(c_id, e))


self.log isn't defined, this could be a global from (IIRC):

logger = logging.getLogger(__name__)

this class is a mixin intended to be used inside an AgentCheck class, so it'll use the agentcheck's self.log as it uses its self.gauge. Adding a docstring

JulienBalestra · 2018-04-13T13:22:18Z

kubelet/datadog_checks/kubelet/cadvisor.py

+            self._publish_raw_metrics(metric, dat[-1], tags, is_pod, depth + 1)
+
+    def _update_container_metrics(self, instance, subcontainer):
+        tags = []


tags definition isn't needed as you are redefining it in each condition below.

JulienBalestra · 2018-04-13T13:23:08Z

kubelet/datadog_checks/kubelet/cadvisor.py

+        # Let's see who we have here
+        if is_pod:
+            tags = tags_for_pod(pod_uid, True)
+        elif (in_static_pod and k_container_name):


nit: the parenthesis are redundant

JulienBalestra · 2018-04-13T13:23:33Z

kubelet/datadog_checks/kubelet/cadvisor.py

+            tags += tags_for_pod(pod_uid, True)
+            tags.append("kube_container_name:%s" % k_container_name)
+        else:  # Standard container
+            if self.container_filter.is_excluded(cid):


Same for self.container_filter, what's this scope ?

JulienBalestra · 2018-04-13T13:25:34Z

kubelet/datadog_checks/kubelet/common.py

+        return False
+
+
+class ContainerFilter:


Old style class, see the other mention.

JulienBalestra · 2018-04-16T12:38:29Z

kubelet/datadog_checks/kubelet/cadvisor.py

+
+        self._update_metrics(instance, cadvisor_url, pod_list, container_filter)
+
+    def _retrieve_cadvisor_metrics(self, cadvisor_url, timeout=10):


This method doesn't use the instance, can it be static ?

JulienBalestra · 2018-04-16T12:39:40Z

kubelet/datadog_checks/kubelet/cadvisor.py

+        """
+        Recusively parses and submit metrics for a given entity, until
+        reaching self.max_depth.
+        Nested metric names are flattened: memory/usage -> memory.usage


nit: I really enjoy docstrings with the type of each parameter

JulienBalestra · 2018-04-16T12:41:50Z

kubelet/datadog_checks/kubelet/cadvisor.py

+                self._publish_raw_metrics(metric + '.%s' % k.lower(), v, tags, is_pod, depth + 1)
+
+        elif isinstance(dat, list):
+            self._publish_raw_metrics(metric, dat[-1], tags, is_pod, depth + 1)


Could be useful to catch a potential else here ?

else would only be a pass. I'm not sure we should log it

JulienBalestra · 2018-04-16T12:43:37Z

kubelet/datadog_checks/kubelet/cadvisor.py

+        is_pod = False
+        in_static_pod = False
+        cid = subcontainer.get('id')
+        pod_uid = subcontainer.get('labels', []).get('io.kubernetes.pod.uid')


Can the default be a dict instead of a list ?
Especially with .get('io.kubernetes.pod.uid') over it (list doesn't support get on it)

good catch, ouch

JulienBalestra · 2018-04-16T12:45:49Z

kubelet/datadog_checks/kubelet/cadvisor.py

+        in_static_pod = False
+        cid = subcontainer.get('id')
+        pod_uid = subcontainer.get('labels', []).get('io.kubernetes.pod.uid')
+        k_container_name = subcontainer.get('labels', []).get('io.kubernetes.container.name')


JulienBalestra · 2018-04-16T12:50:06Z

kubelet/datadog_checks/kubelet/cadvisor.py

+            return
+        tags = list(set(tags + instance.get('tags', [])))
+
+        stats = subcontainer['stats'][-1]  # take the latest


Are we sure of the existence of stats and a len > 0 ?
Does it make sense to add a try except here ?

The exception will be caught in the parent _update_metrics

JulienBalestra · 2018-04-16T12:51:46Z

kubelet/datadog_checks/kubelet/cadvisor.py

+        stats = subcontainer['stats'][-1]  # take the latest
+        self._publish_raw_metrics(NAMESPACE, stats, tags, is_pod)
+
+        if is_pod is False and subcontainer.get("spec", {}).get("has_filesystem") and stats.get('filesystem', []) != []:


nit: what about doing this instead:

if is_pod is False and subcontainer.get("spec", {}).get("has_filesystem") and stats.get('filesystem'):

It doesn't create two empty lists to just compare if it's not None.

JulienBalestra · 2018-04-16T12:52:47Z

kubelet/datadog_checks/kubelet/common.py

+    return get_tags('docker://%s' % cid, cardinality)
+
+
+def get_pod_by_uid(uid, podlist):


Really like that kind of docstrings 👍

JulienBalestra · 2018-04-16T12:54:09Z

kubelet/datadog_checks/kubelet/common.py

+    def __init__(self, podlist):
+        self.containers = {}
+
+        for pod in podlist.get('items') or []:


Can we replace this by:

for pod in podlist.get('items', []):

Now that we have

if self.pod_list.get("items") is None: # Sanitize input: if no pod are running, 'items' is a NoneObject self.pod_list['items'] = []

in check() we can. Before, we had a items key with a None value, that made the iteration fail when no pod was running

- adds the ContainerFilter helper class to consume the new agent interface. Only used for cadvisor mode for now - factors-out common parts to a common.py file - copy agent5 cadvisor logic to a CadvisorScraper helper class for separation - update the cadvisor logic to support agent6 facilities (tagger, filter) - update the cadvisor logic to report network metrics at the pod cardinality for consistency with prometheus mode (change from agent5): see comment lower - add the missing disk metric in the kubernetes.csv metadata file

xvello · 2018-04-16T17:05:09Z

Final testing with datadog/agent-dev:xvello-test1-3 image show metrics are equivalent to agent5, and CPU usage is lower (~5% average) thant agent5 (~9% average)

xvello · 2018-04-17T09:29:36Z

@l0k0ms can we sync about this PR and DataDog/datadog-agent#1550 ? Do you see other docs to update?

xvello force-pushed the xvello/kubelet-cadvisor branch 2 times, most recently from 0243101 to b908076 Compare April 4, 2018 15:35

masci added the do-not-merge/WIP label Apr 4, 2018

xvello force-pushed the xvello/kubelet-cadvisor branch from b951a17 to 69871ee Compare April 9, 2018 13:50

xvello mentioned this pull request Apr 10, 2018

[legacy] add kubernetes.yml -> kubelet.yaml import logic DataDog/datadog-agent#1550

Merged

xvello added this to the 6.2 milestone Apr 10, 2018

xvello added containers kind/feature-request labels Apr 10, 2018

xvello mentioned this pull request Apr 11, 2018

The units of kubernetes.cpu.usage.total has changed from agent5 DataDog/datadog-agent#1552

Open

xvello requested a review from l0k0ms April 12, 2018 12:47

l0k0ms reviewed Apr 12, 2018

View reviewed changes

xvello force-pushed the xvello/kubelet-cadvisor branch from b7c6587 to a5ca319 Compare April 12, 2018 12:53

xvello force-pushed the xvello/kubelet-cadvisor branch 4 times, most recently from 51ab121 to e2c9487 Compare April 13, 2018 12:05

xvello requested a review from a team April 13, 2018 12:43

xvello removed the do-not-merge/WIP label Apr 13, 2018

xvello changed the title ~~[WIP] port cadvisor metric collection from legacy kubernetes check~~ [kubelet] port cadvisor metric collection from legacy kubernetes check Apr 13, 2018

JulienBalestra requested changes Apr 13, 2018

View reviewed changes

JulienBalestra reviewed Apr 16, 2018

View reviewed changes

xvello force-pushed the xvello/kubelet-cadvisor branch from f7b29c2 to 95a3ebc Compare April 16, 2018 14:22

xvello force-pushed the xvello/kubelet-cadvisor branch from 95a3ebc to d1a7593 Compare April 16, 2018 14:27

update README

463a8f9

xvello force-pushed the xvello/kubelet-cadvisor branch from 23acc34 to 463a8f9 Compare April 16, 2018 15:39

add missing test fixture

33e9c5c

JulienBalestra previously approved these changes Apr 17, 2018

View reviewed changes

update kubelet's metadata.csv

8f3fd95

xvello dismissed JulienBalestra’s stale review via 8f3fd95 April 17, 2018 09:32

xvello merged commit efd68a0 into master Apr 17, 2018

xvello deleted the xvello/kubelet-cadvisor branch April 17, 2018 10:42

xvello mentioned this pull request Apr 17, 2018

Kubernetes Daemonset manifest does not work on clusters < 1.7 DataDog/datadog-agent#1465

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[kubelet] port cadvisor metric collection from legacy kubernetes check #1339

[kubelet] port cadvisor metric collection from legacy kubernetes check #1339

xvello commented Apr 4, 2018 •

edited

Loading

xvello commented Apr 12, 2018

l0k0ms Apr 12, 2018 •

edited

Loading

l0k0ms Apr 12, 2018

l0k0ms commented Apr 12, 2018

xvello commented Apr 12, 2018 •

edited

Loading

JulienBalestra Apr 13, 2018

xvello Apr 13, 2018

JulienBalestra Apr 13, 2018

JulienBalestra Apr 13, 2018

JulienBalestra Apr 13, 2018

JulienBalestra Apr 13, 2018

JulienBalestra Apr 13, 2018

xvello Apr 13, 2018

JulienBalestra Apr 13, 2018

JulienBalestra Apr 13, 2018

JulienBalestra Apr 13, 2018

JulienBalestra Apr 13, 2018

JulienBalestra Apr 16, 2018

JulienBalestra Apr 16, 2018 •

edited

Loading

JulienBalestra Apr 16, 2018

xvello Apr 16, 2018

JulienBalestra Apr 16, 2018

xvello Apr 16, 2018

JulienBalestra Apr 16, 2018

JulienBalestra Apr 16, 2018

xvello Apr 16, 2018

JulienBalestra Apr 16, 2018

JulienBalestra Apr 16, 2018

JulienBalestra Apr 16, 2018

xvello Apr 16, 2018 •

edited

Loading

xvello commented Apr 16, 2018

xvello commented Apr 17, 2018

		@@ -119,6 +119,8 @@ the new testing approach:

		For checks that are not listed here, please refer to [Legacy development Setup](docs/dev/legacy.md).

		If you updated the test requirements for a check, you will need to run `tox --recreate` for changes to be effective.

		LEGACY_CADVISOR_METRICS_PATH = '/api/v1.3/subcontainers/'


		class CadvisorScraper():


		self._update_metrics(instance, cadvisor_url, pod_list, container_filter)

		def _retrieve_cadvisor_metrics(self, cadvisor_url, timeout=10):

		return get_tags('docker://%s' % cid, cardinality)


		def get_pod_by_uid(uid, podlist):

[kubelet] port cadvisor metric collection from legacy kubernetes check #1339

[kubelet] port cadvisor metric collection from legacy kubernetes check #1339

Conversation

xvello commented Apr 4, 2018 • edited Loading

What does this PR do?

Motivation

Testing Guidelines

Versioning

Additional Notes

xvello commented Apr 12, 2018

l0k0ms Apr 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

l0k0ms commented Apr 12, 2018

xvello commented Apr 12, 2018 • edited Loading

Agent5, sending one gauge per container in the pod, only pause container has a non-zero value

Agent6 + prometeus: correctly tagging by pod tags only

Agent6 + cadvisor : move the network metric to pod level, mirroring prometeus mode

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JulienBalestra Apr 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xvello Apr 16, 2018 • edited Loading

Choose a reason for hiding this comment

xvello commented Apr 16, 2018

xvello commented Apr 17, 2018

xvello commented Apr 4, 2018 •

edited

Loading

l0k0ms Apr 12, 2018 •

edited

Loading

xvello commented Apr 12, 2018 •

edited

Loading

JulienBalestra Apr 16, 2018 •

edited

Loading

xvello Apr 16, 2018 •

edited

Loading