Retry Policy

[blalop]

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

In some cases, Teams won't be able to publish the Prometheus alerts and due to no retry policy implemented in prom2teams, those alerts will be missed.

There are two different scenarios to be considered:

• We've sent a payload that is longer that what Teams can accept (see their docs). This should be handled inside prom2teams.
• Teams is not available. Two responses can be expected:
  • Status code 200. Exception Microsoft.Griffin.Connectors.Store.ExchangeTooManySessionsException in body.
  • Status code 503.

Steps to Reproduce

First scenario can be easily reproduced sending a long payload. The other one depends on MS and so is not easily reproducible.

Expected behavior: Retry the alarm sending so no alarms are lost in MS Teams

Actual behavior: Some alarms are missed

Reproduces how often: Enough to be annoying

Versions

All.

Additional Information

None.