Add cassandra_nodetool check #511

zippolyte · 2017-06-29T22:47:45Z

What does this PR do?

Add a python check for cassandra.

Motivation

It was not possible to get this information via the JMXCheck.

Versioning

Bumped the version check in manifest.json
Updated CHANGELOG.md

Additional Notes

This check uses the cassandra python driver. Docs here.
The driver tries to connect to a cassandra node, and if it succeed, it discovers automatically the other nodes of the cluster. It then tries to connect to those other nodes, and if it can't, it marks those nodes as down.
The check uses this flag to count the number of replicas that are down for a keyspace.

It now uses the nodetool utility that makes calls to JMX functions under the hood. Specifically, it uses the status command, documented here.

The check sends the cumulated percentage of data owned by each UP node of the cluster for a given keyspace, as well as the replication factor for this keyspace.

I removed the service check from The Last Pickle because it required the cassandra driver dependency that weighs 32MB unpackaged and 13MB packaged.
We'll have to discuss if it's worth keeping.

irabinovitch · 2017-07-01T00:58:03Z

@zippolyte Should this also include the service check provided by the TLP team?

zippolyte · 2017-07-03T21:23:00Z

I don't know, maybe it should be removed since we already have a service check with the jmxcheck ? I left it on purpose since it doesn't check for exactly the same connection (connection to the jmx server vs the actual cassandra).

…ns-core into hippo/cassandra_check

irabinovitch · 2017-07-26T22:00:26Z

@zippolyte @masci next steps on this and the dashboards?

masci · 2017-07-27T10:05:26Z

cassandra_check/README.md

@@ -0,0 +1,29 @@
+# Cassandra Check


We should change the name of this check, I think having "Cassandra" and "Cassandra Check" would be confusing.
At this point why not having a "Cassandra Nodetool" check? This would be explicit in what it does, plus we could support more nodetool commands other than status in the future with the same check.
@irabinovitch any thoughts?

masci · 2017-07-27T10:19:51Z

cassandra_check/check.py

+class CassandraCheck(AgentCheck):
+
+    datacenter_name_re = re.compile('^Datacenter: (.*)')
+    host_status_re = re.compile('^(?P<status>[UD])[NLJM].* (?P<owns>(\d+\.\d+%)|\?).*')


since we're here, why not parsing all the info nodetool returns?
Specifically:

Status (gauge)

State (tag)

Address (tag)

Load (gauge)

Tokens (we can probably ignore, this param is not easy to change on a cluster)

Owns (gauge)

Host ID (tag)

Rack (tag, this would be the availability zone on EC2)

Should we really send the state as a tag ? Appart from the state normal, these are temporary states.
For the status should this be a gauge (1 for up, 0 for down) ? or a service check ? both ? In the check they wrote, TLP sent a gauge so they could use it to fill a host map and see down host at a glance, so I can see the use of sending a gauge here.

masci · 2017-07-27T10:21:39Z

cassandra_check/conf.yaml.example

+
+instances:
+  # Configuration options:
+  # nodetool: a command or path to nodetool (e.g. /usr/bin/nodetool or docker exec container nodetool)


can we move this to init_config?

masci · 2017-07-27T10:28:31Z

cassandra_check/conf.yaml.example

@@ -0,0 +1,16 @@
+init_config:


Can you fix the format of the config file? Comments should go before actual options and config values can be part of the commented option, for example:

# command or path to nodetool (e.g. /usr/bin/nodetool or docker exec container nodetool) # nodetool: /usr/bin/nodetool # the list of keyspaces to monitor # - keyspaces: []

masci · 2017-07-27T10:30:54Z

cassandra_check/metadata.csv

@@ -0,0 +1,3 @@
+metric_name,metric_type,interval,unit_name,per_unit_name,description,orientation,integration,short_name
+cassandra.replication_availability,gauge,,,,Percentage of data available per keyspace times replication factor,+1,cassandra_check,available data


we could use the name of the nodetool command as a prefix for future proofing, like:
cassandra.status.replication_availability or even cassandra.nodetool.status.replication_availability

masci · 2017-07-27T10:33:01Z

@irabinovitch I left some comments, can you have a look?

masci

Let's fix the metadata catalog and we're good to go! 👍

masci · 2017-07-28T13:04:27Z

cassandra_nodetool/metadata.csv

@@ -0,0 +1,3 @@
+metric_name,metric_type,interval,unit_name,per_unit_name,description,orientation,integration,short_name
+cassandra.nodetool.status.replication_availability,gauge,,,,Percentage of data available per keyspace times replication factor,+1,cassandra_nodetool,available data
+cassandra.nodetool.status.replication_factor,gauge,,,,Replication factor per keyspace,0,cassandra_nodetool,replication factor


Can you add the new metrics to the catalog?

masci

One comment about docs, then we can 🚢

masci · 2017-07-31T16:05:53Z

cassandra_nodetool/README.md

+        -----------
+          - instance #0 [OK]
+          - Collected 39 metrics, 0 events & 7 service checks
+


Can you mention the 'cassandra.nodetool.node_up service check?

…ns-core into hippo/cassandra_check

masci

Let's ship it! 🚢

Also remove process.conf for Agent 6

[cassandra_check] Add cassandra_check

ac0045b

zippolyte force-pushed the hippo/cassandra_check branch from 426abd8 to ac0045b Compare June 29, 2017 23:14

zippolyte and others added 10 commits July 5, 2017 10:40

Rework tests

49395f4

disable pylint checks

639923f

Some linting

e2c69b6

Merge branch 'master' into hippo/cassandra_check

b87ab6a

Rework check. Use nodetool instead of python driver

edd33ac

Merge branch 'hippo/cassandra_check' of github.com:DataDog/integratio…

8df8f16

…ns-core into hippo/cassandra_check

Merge branch master into hippo/cassandra_check

a78758b

Add integration test

1c634a8

Fix tests by allowing to specify a docker command for nodetool

cb28feb

Fix mock test

5fe6264

masci suggested changes Jul 27, 2017

View reviewed changes

masci self-assigned this Jul 27, 2017

Address review comments

5110f7f

masci suggested changes Jul 28, 2017

View reviewed changes

masci added this to the 5.17 milestone Jul 28, 2017

More addressing

db4aa03

zippolyte force-pushed the hippo/cassandra_check branch from 4a15ce6 to db4aa03 Compare July 28, 2017 18:55

Merge branch 'master' into hippo/cassandra_check

50fe2cd

masci suggested changes Jul 31, 2017

View reviewed changes

zippolyte added 3 commits July 31, 2017 13:37

Add service check section in readme

4ac71c7

Merge branch 'hippo/cassandra_check' of github.com:DataDog/integratio…

1031262

…ns-core into hippo/cassandra_check

Continue loop if error on calling nodetool

9e53421

masci approved these changes Aug 7, 2017

View reviewed changes

zippolyte merged commit 12abb12 into master Aug 7, 2017

zippolyte deleted the hippo/cassandra_check branch August 7, 2017 15:06

ofek added integration/cassandra_nodetool changelog/Added labels Oct 13, 2018

ofek changed the title ~~[cassandra_check] Add cassandra_check~~ Add cassandra_nodetool check Oct 13, 2018

gml3ff pushed a commit that referenced this pull request May 14, 2020

Support process-agent configuration in datadog.yaml (#511)

884dd84

Also remove process.conf for Agent 6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cassandra_nodetool check #511

Add cassandra_nodetool check #511

zippolyte commented Jun 29, 2017 •

edited by masci

Loading

irabinovitch commented Jul 1, 2017

zippolyte commented Jul 3, 2017

irabinovitch commented Jul 26, 2017

masci Jul 27, 2017 •

edited

Loading

masci Jul 27, 2017

zippolyte Jul 27, 2017

masci Jul 27, 2017

masci Jul 27, 2017

masci Jul 27, 2017

masci commented Jul 27, 2017

masci left a comment

masci Jul 28, 2017

masci left a comment

masci Jul 31, 2017

masci left a comment

		@@ -0,0 +1,3 @@
		metric_name,metric_type,interval,unit_name,per_unit_name,description,orientation,integration,short_name
		cassandra.replication_availability,gauge,,,,Percentage of data available per keyspace times replication factor,+1,cassandra_check,available data

Add cassandra_nodetool check #511

Add cassandra_nodetool check #511

Conversation

zippolyte commented Jun 29, 2017 • edited by masci Loading

What does this PR do?

Motivation

Versioning

Additional Notes

irabinovitch commented Jul 1, 2017

zippolyte commented Jul 3, 2017

irabinovitch commented Jul 26, 2017

masci Jul 27, 2017 • edited Loading

Choose a reason for hiding this comment

masci Jul 27, 2017

Choose a reason for hiding this comment

zippolyte Jul 27, 2017

Choose a reason for hiding this comment

masci Jul 27, 2017

Choose a reason for hiding this comment

masci Jul 27, 2017

Choose a reason for hiding this comment

masci Jul 27, 2017

Choose a reason for hiding this comment

masci commented Jul 27, 2017

masci left a comment

Choose a reason for hiding this comment

masci Jul 28, 2017

Choose a reason for hiding this comment

masci left a comment

Choose a reason for hiding this comment

masci Jul 31, 2017

Choose a reason for hiding this comment

masci left a comment

Choose a reason for hiding this comment

zippolyte commented Jun 29, 2017 •

edited by masci

Loading

masci Jul 27, 2017 •

edited

Loading