Report metrics from custom-plugin-monitor #315

xueweiz · 2019-07-22T23:51:53Z

Add a new field metricsReporting (true by default) in the custom-plugin-monitor config file.
By setting the metricsReporting field to true, custom-plugin-monitor will report metrics when temporary/permanent problems happen.

The behavior was specified in the NPD metrics design doc here.

For example, when I make the custom plugin script to report problem (by manually editing the script to force it return error), I will see:

$ curl http://localhost:20257/metrics
# HELP problem_counter Number of times a specific type of problem have occurred.
# TYPE problem_counter counter
problem_counter{reason="NTPIsDown"} 5
# HELP problem_gauge Whether a specific type of problem is affecting the node or not.
# TYPE problem_gauge gauge
problem_gauge{reason="NTPIsDown",type="NTPProblem"} 1

This PR maintains backward compatibility on existing custom-plugin-monitor configs.

This PR is part of #284:

[ x ] Implement feature in System Log Monitor and Custom Plugin Monitor, to allow them to report metrics.

Most of the heavy lifting has been done in #300 .

I tried to update the README for custom-plugin-monitor, but seems that's not necessary:

The README for custom-plugin-monitor simply points to its design doc.
In the design doc, all common configuration flags (e.g. rules, conditions) were omitted.

Since the metricsReporting flag is now a common config flag used by both system-log-monitor and custom-plugin-monitor, I don't think it's necessary to add it to the custom-plugin-monitor design doc.

k8s-ci-robot · 2019-07-22T23:52:00Z

Hi @xueweiz. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

xueweiz · 2019-07-22T23:52:19Z

/assign @wangzhen127
/cc @Random-Liu

xueweiz · 2019-07-22T23:56:06Z

/cc @andyxning
Hi Ning, could you also help take a look of this PR? Thanks! Since you are the original author of this plugin, I want to hear your opinion.

wangzhen127 · 2019-07-23T20:08:34Z

/ok-to-test

xueweiz · 2019-07-23T20:12:52Z

My bad. I'll fix the test.
Also, since Lantao is back now:
/assign @Random-Liu

xueweiz · 2019-07-24T23:22:24Z

@Random-Liu Hi Lantao, sorry for the delay. I just fixed the tests. Could you help take a look? Thanks!

Random-Liu

LGTM with one comment.

Random-Liu · 2019-07-25T05:54:25Z

pkg/custompluginmonitor/custom_plugin_monitor.go

+							glog.Errorf("Failed to update problem gauge metrics for problem %q, reason %q: %v",
+								result.Rule.Condition, result.Rule.Reason, err)
+						}
+					}
 				}



Generate counter and gauge at the end of the function based on the final events and condition.

The logic is too complex, the code makes it even harder to maintain.

And let's do the same for system log monitor.

Ah this is a very good idea! Thanks for noticing that1
Done.

Random-Liu · 2019-07-25T18:18:45Z

pkg/custompluginmonitor/custom_plugin_monitor.go

+					event.Reason, err)
+			}
+		}
+		for _, condition := range metricChangingConditions {


Can't we just go through condition list, and set problem gauge for all conditions correspondingly?

That seems more reliable, basically a periodic gauge sync. Will that cause any performance issue?

Oh just realized that in custom-plugin-monitor, the generateStatus() is only called periodically. Then I think it's fine. Changing the code.

(in system-log-monitor, the generateStatus() is called for each matched Rule, which is much more frequent. So I'd prefer to keep the changedConditions logic in there.

Done. Thanks!

SG. The logic in system-log-monitor is simple, so I'm also fine with adding smarter logic there. :)

Random-Liu · 2019-07-25T18:46:34Z

/lgtm

k8s-ci-robot · 2019-07-25T18:46:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Random-Liu, xueweiz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Random-Liu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 22, 2019

k8s-ci-robot requested review from Random-Liu and wangzhen127 July 22, 2019 23:52

k8s-ci-robot assigned wangzhen127 Jul 22, 2019

k8s-ci-robot requested a review from andyxning July 22, 2019 23:56

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 23, 2019

k8s-ci-robot assigned Random-Liu Jul 23, 2019

This was referenced Jul 24, 2019

Implement NPD metrics mode #284

Closed

Prometheus metrics endpoint #259

Closed

xueweiz force-pushed the metrics branch from 925a622 to b1d33da Compare July 24, 2019 23:21

Random-Liu reviewed Jul 25, 2019

View reviewed changes

xueweiz force-pushed the metrics branch from b1d33da to 1b02962 Compare July 25, 2019 08:11

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 25, 2019

xueweiz force-pushed the metrics branch from 1b02962 to 5b4f004 Compare July 25, 2019 08:23

Random-Liu reviewed Jul 25, 2019

View reviewed changes

Report metrics from custom-plugin-monitor

94af7de

xueweiz force-pushed the metrics branch from 5b4f004 to 94af7de Compare July 25, 2019 18:28

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 25, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 25, 2019

k8s-ci-robot merged commit 599ca53 into kubernetes:master Jul 25, 2019

xueweiz mentioned this pull request Jul 29, 2019

Bump NPD version to v0.7.1 for GCI kubernetes/kubernetes#80726

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Report metrics from custom-plugin-monitor #315

Report metrics from custom-plugin-monitor #315

xueweiz commented Jul 22, 2019 •

edited

Loading

k8s-ci-robot commented Jul 22, 2019

xueweiz commented Jul 22, 2019

xueweiz commented Jul 22, 2019

wangzhen127 commented Jul 23, 2019

xueweiz commented Jul 23, 2019

xueweiz commented Jul 24, 2019

Random-Liu left a comment

Random-Liu Jul 25, 2019

Random-Liu Jul 25, 2019

xueweiz Jul 25, 2019

Random-Liu Jul 25, 2019 •

edited

Loading

xueweiz Jul 25, 2019

xueweiz Jul 25, 2019

Random-Liu Jul 25, 2019

Random-Liu commented Jul 25, 2019

k8s-ci-robot commented Jul 25, 2019

Report metrics from custom-plugin-monitor #315

Report metrics from custom-plugin-monitor #315

Conversation

xueweiz commented Jul 22, 2019 • edited Loading

k8s-ci-robot commented Jul 22, 2019

xueweiz commented Jul 22, 2019

xueweiz commented Jul 22, 2019

wangzhen127 commented Jul 23, 2019

xueweiz commented Jul 23, 2019

xueweiz commented Jul 24, 2019

Random-Liu left a comment

Choose a reason for hiding this comment

Random-Liu Jul 25, 2019

Choose a reason for hiding this comment

Random-Liu Jul 25, 2019

Choose a reason for hiding this comment

xueweiz Jul 25, 2019

Choose a reason for hiding this comment

Random-Liu Jul 25, 2019 • edited Loading

Choose a reason for hiding this comment

xueweiz Jul 25, 2019

Choose a reason for hiding this comment

xueweiz Jul 25, 2019

Choose a reason for hiding this comment

Random-Liu Jul 25, 2019

Choose a reason for hiding this comment

Random-Liu commented Jul 25, 2019

k8s-ci-robot commented Jul 25, 2019

xueweiz commented Jul 22, 2019 •

edited

Loading

Random-Liu Jul 25, 2019 •

edited

Loading