Support all event fields in the Pagerduty action.

[tobio]

Currently the Pagerduty action doesn't support setting the custom_details, images or links event fields. These fields should be configurable and templated.

cc: @pmuellr

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[alexfrancoeur]

@mikecote @arisonl I've been on a call recently where custom_details would provide a lot of additional value to the PagerDuty integration. I realize the that there is a PR linked to this issue that was stalled last year. While fully understanding the large amount of competing priorities on the alerting team, what would the work effort be to enhance this action to include the custom_details field? Is it relatively small? As I understand it, summary can only provide so much information and is limited to 1024 characters.

With a PR already in place, it looked like there were some unanswered questions and a need for tests. Looking at some of the comments, there are questions around static and dynamic content. I do think we'd probably want to include context variables in this field in particular.

[pmuellr]

I don't think there's a huge problem with the custom_details field, beyond the UI for it. I think it would be like the headers field in the webhook connector.

links doesn't look bad, as the data it is, but it's not clear how it would be used, since it has both the URL and link text. Maybe that's a PagerDuty thing? I would have expected just urls, not link text.

images is the one which is probably the most challenging, because I think what we'd want to do is to somehow have Kibana generate images, and then populate those fields with urls to those images Kibana generated, that got uploaded ... somewhere. We've looked into how this would work with Slack - we can upload the images separately, and then get links back to them from the submission responses. If PagerDuty has something similar - a way to upload images for later reference in PagerDuty events, then it would follow the same flow (whatever that may be, since we haven't done it yet).

I did add a comment to the PR that the schema definition for the params needs some slight tweaking.

I do think we'd probably want to include context variables in this field in particular.

The parameters are always available as a mustache template variable, alertParams. The question will be, how easy will it be to get the data out. Eg, if custom_details is {a: b, c: d}, you'd access the values via alertParams.custom_details.a - and of course every alert could have different properties, so every mustache template for every alert may end up referencing different properties. I suspect there are some "common" values here, like description or such, so maybe that's not a big deal.

[alexfrancoeur]

++ I think custom_details and links as optional fields would only make our integration more useful. I completely agree that images is a much larger and complex issue. If the team agrees, maybe we augment the PD action with custom_details and links and open a separate issue for images?

I also agree that there are probably some challenges to getting the data out, but is that any different than other context variables? I feel like most of them are rather alert type specific with some common values. Not that that's ideal, and as we unify rules, maybe that gets better.

[pmuellr]

I also agree that there are probably some challenges to getting the data out, but is that any different than other context variables? I feel like most of them are rather alert type specific with some common values. Not that that's ideal, and as we unify rules, maybe that gets better.

Ya, sorry, I misunderstood the original question, shouldn't be a problem adding context variables to the custom details properties - or we need to make that possible anyway, since this function would likely be worthless without it.

I completely agree that images is a much larger and complex issue. If the team agrees, maybe we augment the PD action with custom_details and links and open a separate issue for images?

Ya, makes sense to split custom_details and links from images.

[arisonl]

Please note that the inclusion of custom_details has come up again in conversations with users, as an incremental improvement that will add a great lot of value in very important use cases cc @gmmorris @mikecote @pmuellr @paulewing @cnasikas @tobio

[sparrowt]

EDIT: I realised since posting this that, being on the Kibana repo, perhaps this issue is actually referring to the Kibana Alerts connector 'PagerDuty' functionality (https://www.elastic.co/guide/en/kibana/current/pagerduty-action-type.html) rather than the Elastic watcher pagerduty action (https://www.elastic.co/guide/en/elasticsearch/reference/8.4/actions-pagerduty.html)?

If so apologies for barking up the wrong tree!

---

Original comment:

It's worth noting that you can currently send images and links however to do so you have to use the contexts attribute on your watcher action (https://www.elastic.co/guide/en/elasticsearch/reference/8.4/actions-pagerduty.html#pagerduty-action-attributes).

The confusion lies in the fact that:

• these attributes were named after the corresponding fields in the PagerDuty "Events API v1"
• since then Elastic switched to sending via the PagerDuty "Events API v2" (Watcher: migrate PagerDuty v1 events API to v2 API elasticsearch#32285)
• however the pagerduty watcher action still uses the original attribute names, which are then translated behind the scenes into the new v2 API fields

So to set images and links, simply supply something like this:

"contexts": [
  {
    "type": "link",
    "href": "..."
  },
  {
    "type": "image",
    "src": "..."
  }
]

Then of the three v2 API fields mentioned in this ticket originally, only custom_details remains - however if you set the pagerduty watcher action attribute attach_payload to True, then custom_details gets filled in with the payload from your watcher, which you can control to some extent as you define your watcher.

In other words, all 3 of these fields can be set, just not using the same strings for the watcher attributes, as are used in the underlying PagerDuty API.

[ymao1]

cc @shanisagiv1

[mikecote]

cc @elastic/response-ops-cases in case this is still a request for cases

[lucasmoore]

Is there something we can do to help get this prioritized? We (Platform SRE), along with other teams working on Serverless, are making heavy use of Kibana Alerting Rules and PagerDuty actions. We would greatly benefit from being able to include things like links to runbooks and dashboards in a PagerDuty action field. It's a significant improvement to the on-call experience for engineers to have links to the relevant information present in the alert rather than having to look for it separately.

cc @rbrunan @elkargig

[cnasikas]

Hey @lucasmoore! It is a high-priority issue for us and we will start working on it soon. As images are complicated we can reduce the scope and start with supporting custom_details and links. Is that ok with you?

[lucasmoore]

Hey @lucasmoore! It is a high-priority issue for us and we will start working on it soon. As images are complicated we can reduce the scope and start with supporting custom_details and links. Is that ok with you?

Yes, that sounds good to me. Thanks!

[cnasikas]

Related: #76910

[cnasikas]

PR #170459 added support for links and custom_details in the connector's PagerDuty API. Issue for the UI: #170048.

[cnasikas]

PR #171748 added support for the UI. If needed I think is better to open a separate issue for images.