How to set up alerting?

[wsw70]

I wanted to set up an alert triggered by a failed monitor. Thinking I understood the docs correctly I set up

monitor.ini (cropped down to monitoring and alerting):

# general
[monitor]
interval=60
remote=1
remote_port=9999
key=XX

[reporting]
loggers=db,file,status,json_full
alerters=ntfy.sh,ntfy.swtk.eu

; (...)

# alerters
[ntfy.swtk.eu]
type=ntfy
priority=default
token=X
server=https://ntfy.swtk.eu
topic=simplemonitor
icon_prefix=true
groups=ntfy.swtk.eu

[ntfy.sh]
type=ntfy
priority=default
token=X
server=https://ntfy.swtk.eu
topic=simplemonitor
icon_prefix=true
groups=ntfy.sh

My understanding of the above was: the alerters that are defined are ntfy.sh and ntfy.swtk.eu, and there are relevant [<section with the alterter>] for them further down.

I then attached monitors to the alerts in monitors.ini:

[defaults]
tolerance=2
group=ntfy.swtk.eu

; via TCP
[PING router]
type=tcp
host=ext.swtk.eu
; this port is available
port=X
; external notified because mine is off if this is down
group=ntfy.sh

[APP → pihole]
type=http
url=https://pihole.swtk.eu?simplemonitor=X
depend=APP → authelia

# tests
[test-fail-ntfy.sh]
type=fail
group=ntfy.sh

[test-fail-ntfy.swtk.eu]
type=fail
group=ntfy.swtk.eu

My understanding from this minimal monitors.ini is that

• by default failed monitors are alerted via ntfy.swtk.eu
• example: [APP → pihole]
• some monitors are explicitly managed via the ntfy.sh alerter
• example: [PING router]

After running ./simplemonitor/.venv/bin/simplemonitor -v -1 -d I have my monitors triggered correctly, but the debug also shows

2024-07-03 15:57:21    DEBUG (simplemonitor.alerter-ntfy) not alerting for test-fail-ntfy.sh: group mismatch (monitor: ntfy.sh; alerter: ['ntfy.swtk.eu'])
2024-07-03 15:57:21    DEBUG (simplemonitor.alerter-ntfy) not alerting for PING router: group mismatch (monitor: ntfy.sh; alerter: ['ntfy.swtk.eu'])
2024-07-03 15:57:21    DEBUG (simplemonitor.alerter-ntfy) not alerting for DSK → pihole /: group mismatch (monitor: ntfy.swtk.eu; alerter: ['ntfy.sh'])

I have a hard time parsing the reason for the failures (the message after "group mismatch") in the context of my configuration

[wsw70]

It looks like group and groups are flipped: when a monitor wants to use ntfy.sh the error is that ntfy.swtk.eu is available, and vice-versa.

[jamesoff]

The logging isn't great here I think as it omits the name of the alerter, so the only way to tell the alerters apart in the log output is by their groups.

The alerting loop just runs each alerter in turn, and tries to alert each failed monitor with them.

What the log lines you have there are saying is that the first two monitors (in the log) are set to be in the ntfy.sh group, and the third is in the ntfy.swtk.eu group, which I think is right from your config snippet. The alerter: bit at the end of the line gives the groups the alerter is a member of, and the message appears when the former is not in the latter, so I think the logic is working right. (Although the format of the message isn't great, I agree :)

The group logic is the first check in the should_alert() code which is why you see those messages, but I see your tolerance is set to 2, so for the monitor/alerter pairs which would alert, if this is the first loop they've failed on, the alerts are skipped since the fail count is 1 and that's less than the tolerance.

Hope that makes sense and is answering the question — work and non-work have been occupying my brain a lot recently but I wanted to respond promptly!

[wsw70]

ahhhhhhhhhhhhh 😳😳😳🫥

You mean that the alerting does work, the debug shows what was missed in the alerting matrix due to the logic!!!

And I just had the eureka moment when I realized that I have a tolerance of 2, which means that with a -1 in the command line the actual alerting message is never sent!

I commented out the tolerance and it works!

THANK YOU. I spent a part of last night browsing through the code to try to understand this and somewhere in the back of my mind I had the "wtf is this a debug-only message when the alerting fails??" (but I did not pay attention enough).

[jamesoff]

No worries, and sorry it wasted a bunch of your time. The debug logging around the alerting logic is mostly stuff I sprinkled in when I was (re)writing it and trying to get everything straight, so it's not completely comprehensive about the decisions being made. I guess if there were more things it logged as it went through the logic it would have helped? I can add that.

[wsw70]

I guess if there were more things it logged as it went through the logic it would have helped?

No, no worries. The fact that this is a debug message should have made it clear to me right away (an alerter that is misconfigured and does not trigger would be at least a warning, certainly not hidden in debug logs).

Thanks a lot for your answer, when I read it the first time I was "yes, fine so how should I configure it? :)" and it is only then that it it me that you were essentially telling me "yeah, looks good" :).

Everything is fine. Thanks a lot. Have a great summer.