[Alerting] Provide services to set context for recovered alerts

[ymao1]

POC here: https://github.com/elastic/kibana/pull/119080/files

As a framework, we want to provide rule executors the ability to specify context variables for recovered alerts. Typically, recovered alerts are determined by the framework and provide no context. We've seen an uptick of user requests wanting context information in their recovery notifications and we want to enable rule executors to do this.

We will be providing two services to the executors:

• getRecoveredAlertIds - Provides executors with a list of alert IDs for which the framework will schedule recovery actions.
• setRecoveryContext - Allows executors to specify recovery context for a specified alert id.

Ideally, we would like to rule type executors to make use of these service functions after they are done creating alerts and scheduling actions for these alerts. Although we cannot strictly enforce this at a framework level, we can strongly suggest this code flow by adding a .done() function to the existing alertInstanceFactory service. Only when this .done() function is called will the recovery service functions be available for use. For implementation details, consult the POC.

Finally, in order to try to reduce confusion in the UI before all rule types have had a chance to implement specifying recovery context, we want to add a flag to the rule type during registration where the rule type author can opt-in to defining recovery context. Opting in will allow them access to the proposed new service functions and opting in will surface the context variables in the list of action variables for the Recovered action group for this rule type. For implementation details, consult the POC.

Note: When implementing, we might want to consider making the calls for scheduling actions for active alerts and setting context for recovered alerts more consistent. A suggestion that has been floated is to change scheduleActions into two steps:
alert.scheduleAction().setContext() and have the recovery service act like alert.setContext(). Not a hard requirement, just something to consider.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)