[RAM] Bulk update on rules

[XavierM]

This work will be able to allow our solution to create bulk update on Rule with the bulk api from our saved object. Below, I am going to describe step by step how we want this bulk update to work but before that we want to make sure that will have our new bulk_update using our retryIfConflicts like we do for all our mutations.

Our bulk_update interface should looks like that:

export interface BulkUpdateOptions<Params extends AlertTypeParams> {
 filter: KueryNode;
 data: {
   name?: string;
   tags?: string[];
   schedule?: IntervalSchedule;
   actions?: NormalizedAlertAction[];
   params?: Params;
   throttle?: string | null;
   notifyWhen?: AlertNotifyWhenType | null;
 };
}

Step by Step:

1. Kibana (Rule Client) --> Kibana (Saved Object): Get all the rule type id by using a terms aggs with the filter (caveat if we think that we are going to have more than 1000 unique terms, we will have to use an aggs but I do think that we should be ok with terms for now)
2. Kibana (Rule Client) --> Kibana (Security): Check if user is authorized to access all the rule type id from step 1 by using p-map
3. Kibana (Rule Client) --> Kibana (Licensing): Check license for all the rule type id coming from the aggs by using p-map
4. Kibana (Rule Client) --> Kibana (Saved Object): At this point we will use the SOC createPointInTimeFinder to build our bulk update objects SavedObjectsBulkUpdateObject

• Let’s build the Array<SavedObjectsBulkUpdateObject> at this point we should use p-map to limit the concurrency request to do this different steps below
  1. Kibana (Rule Client) --> Kibana (Rule Client): Validate rule's params if we get an error here we throw an error and we stop completely the bulk update Game over
  2. Kibana (Rule Client) --> Kibana (Rule Client): Validate rule's actions if we get an error here we throw an error and we stop completely the bulk update Game over
  3. Kibana (Rule Client) --> Kibana (Action Client): extract references + normalize params and actions (we should be smarter here and realize if we need to update or not the actions)
  4. Kibana (Rule Client)-->>Kibana (Security): generate/create API key

5. Kibana (Rule Client) --> Kibana (Saved Object): bulk update rules
6. Kibana (Rule Client) --> Kibana (Saved Object): use the bulkCreate of SOC to create all the apiKeyId in api_key_pending_invalidation's Saved object if rule as apiKey attributes
7. Kibana (Rule Client) --> Kibana (Task Manager): update schedule task id if needed it -> Caveat here to have this working perfectly we will need to create a bulk update on the task manager (@mikecote)

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[mikecote]

Kibana (Rule Client) --> Kibana (Task Manager): update schedule task id if needed it -> Caveat here to have this working perfectly we will need to create a bulk update on the task manager (@mikecote)

I created #124850. Please let us know when you would like this completed and we'll plan accordingly 🙏

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[vitaliidm]

@XavierM
Few observations:

1. Proposed interface doesn't allow to remove/add single array value from bunch of rules, only rewrite entire property.
   Our use cases: user can add/remove specified value from list of rules. For example tag or index pattern.
   Let's look into this on example of adding tag:

   • user types only one tag to be added to all selected rules.
   • currently we fetch all rules on BE, and then traverse through them adding to tags property new tag
   • update rules

     export interface BulkUpdateOptions<Params extends AlertTypeParams> {
      filter: KueryNode;
      data: {
        name?: string;
        tags?: string[];
        schedule?: IntervalSchedule;
        actions?: NormalizedAlertAction[];
        params?: Params;
        throttle?: string | null;
        notifyWhen?: AlertNotifyWhenType | null;
      };
     }

   With this interface, tags property will rewrite existing tags property in all rules.
   So, I was thinking about few ways how to approach this:

   1. Pass into bulkUpdate modifier callback, that will receive rule, modifies it and modified rule can be then validated and saved

   export interface BulkUpdateOptions<Params extends AlertTypeParams> {
    filter: KueryNode;
    modifier: (rule: Rule) => Rule,
   }

   2. Move current bulk edit logic inside bulkUpdate method.

   export interface BulkUpdateOptions<Params extends AlertTypeParams> {
    filter: KueryNode;
    actions: Array<{ type: add_tags | delete_tags| set_tags, value: string[]}>[]
   }

2. Bulk updated list of rule ids. Apart from filter query, we need to bulk update list of rules by ids.
It probably can be done by introducing ids parameter

   export interface BulkUpdateOptions<Params extends AlertTypeParams> {
    filter?: KueryNode;
    ids?: string[]
   }

   The difference in underlying implementation of bulkUpdate will be only on step 1

   1 Kibana (Rule Client) --> Kibana (Saved Object): Get all the rule type id by using a terms aggs with the filter (caveat if we think that we are going to have more than 1000 unique terms, we will have to use an aggs but I do think that we should be ok with terms for now)

   We will need to use something similar to bulkGet.
   But I think this part can be implemented later

3. In rulesClient.find method, used search parameter has string type. Should we also use filter as string for bulkUpdate?
   https://github.com/elastic/kibana/blob/main/x-pack/plugins/alerting/server/rules_client/rules_client.ts#L143

4. Check for rule authorizing



   2 Kibana (Rule Client) --> Kibana (Security): Check if user is authorized to access all the rule type id from step 1 by using p-map

   Is this a check mention above? 

   https://github.com/elastic/kibana/blob/main/x-pack/plugins/alerting/server/rules_client/rules_client.ts#L91
6

5. 3 Kibana (Rule Client) --> Kibana (Licensing): Check license for all the rule type id coming from the aggs by using p-map

   Looks like this check is synchronous, so we can skip p-map
   https://github.com/elastic/kibana/blob/main/x-pack/plugins/alerting/server/rules_client/rules_client.ts#L941

6. 4 Kibana (Rule Client) --> Kibana (Saved Object): At this point we will use the SOC createPointInTimeFinder to build our bulk update objects [SavedObjectsBulkUpdateObject] (https://github.com/elastic/kibana/blob/main/src/core/server/saved_objects/service/saved_objects_client.ts#L116-L131)

   This part is a bit unclear to me for now. We get all rules on step one, don't we? As I understand we want to use createPointInTimeFinder when we page through large data. But as we already have all date from step one - how are we going to use it on a stage of update?

[vitaliidm]

@XavierM

for question in item 2 (ids and KQL query by ids)

curl --location --request GET 'http://localhost:5601/kbn/api/detection_engine/rules/_find?page=1&sort_field=enabled&sort_order=desc&filter=alert._id: "alert:8e5a93a0-9320-11ec-9265-8b772383a08d"&per_page=100' \
--header 'kbn-xsrf: reporting' \
--header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \
--data-raw ''

response

{
    "message": "This key 'alert._id' does NOT exist in alert saved object index patterns: Bad Request",
    "status_code": 400
}

where search query is alert._id: "alert:8e5a93a0-9320-11ec-9265-8b772383a08d"

So it looks like, search by id is not supported. It located outside of _source, in comparison queries like alert.coreMigrationVersion work fine as coreMigrationVersion property located within _source object.

Should we consider introducing ids parameter? Or play with underlying savedObject client

[XavierM]

@vitaliidm thank you for proving me wrong, I created this PR to fix it https://github.com/elastic/kibana/pull/126933/files

[XavierM]

@vitaliidm and I met yesterday and talked about his observations. I do agree with observation 1#, I missed it and we should do something different. The data attributes will be just a rewrite for now until we get more requirements for the other attributes.

      export interface BulkUpdateOptions<Params extends AlertTypeParams> {
       filter: KueryNode;
       data: {
         name?: string;
         schedule?: IntervalSchedule;
         actions?: NormalizedAlertAction[];
         params?: Params;
         throttle?: string | null;
         notifyWhen?: AlertNotifyWhenType | null;
        },
       actions: Array<{ type: 'add' | 'delete' | 'set',  field: 'params.index' | 'tags', value: string[]}>[]
      }

I talked to my team about the validation and we do think that we can use the validation around rule's params and if we get an error here we do not add this rule in our Array<SavedObjectsBulkUpdateObject> and return an error to our user. Let's discuss more but I think that should be sufficient and we can create specific validation for our direct fields inside our Rules's saved object.

• 2 we are creating a PR to fix it
• 3 we will use the unsecureSavedObjectClient and the find allow us to use KueryNode
• 4 we clarify this situation by using the find aggs to get all the ruleTypeId associated to the filter and then we can check if the user have access to the rules associated to the filter
• 5 Correct!
• 6 been clarify with number 4

[vitaliidm]

Thanks @XavierM for coming up with the solution so quick regarding search by ids.

I have prepared the first working POC: https://github.com/elastic/kibana/pull/126904/files#diff-6736e143ede2dc06e825bddcdc23b4d088a6620805751db4eddc5900d586c4df

And while working on it, have a couple questions

1. In regard of aggregations and checking for authorizing
   In updateWithOCC autorizing is done not only with alertTypeId but with consumer property as well

         await this.authorization.ensureAuthorized({
           ruleTypeId: alertSavedObject.attributes.alertTypeId,
           consumer: alertSavedObject.attributes.consumer,
           operation: WriteOperations.Update,
           entity: AlertingAuthorizationEntity.Rule,
         });

   So, it looks like, we need to aggregate by multi terms here, which is not currently supported by aggregate method.
   For now, I have hardcoded consumer with siem value, as I can see all rules have only this value. But I wonder, are there cases when consumer property has another value?
   My implementation:

         await this.authorization.ensureAuthorized({
           ruleTypeId: key,
           // TODO: need multi aggregation for alertTypeId & consumer?
           consumer: 'siem',
           operation: WriteOperations.Update,
           entity: AlertingAuthorizationEntity.Rule,
         });

2. Looks like unsecuredSavedObjectsClient doesn't return apiKeys. So it's impossible to invalidate them.

       const rulesFinder = await this.unsecuredSavedObjectsClient.createPointInTimeFinder<RawRule>({

   Are there any other ways we can retrieve apiKeys with find API?

3. I had a second thought regarding validation:
   Currently we do different validations when we bulk edit rules:

   • if rule is immutable, we throw error
   • if rule is machine learning and we try to update index patterns, we throw error
   • if we update rule's index pattern with empty array, we throw error

   All these validations are done before calling rulesClient.update method. So, rulesClient allows to update rules with the
   properties which are not valid from our detections rules perspective.
   Hence, I'm wondering what is a valid rule from rulesClient perspective? I can see there are some validations
   done(scheduling, actions for example), but looks like not all. For example, current update method allows to mutate
   prebuilt rule, which should be immutable. That's why we do this additional validation.
   Possible solutions to this:

   • we can implement custom validation callback, that can be passed into bulkUpdate. We discussed that during our call
   • we can do all validations within rulesClient.bulkUpdate method. But we would need to handle there cases I described above

4. Kibana (Rule Client) --> Kibana (Rule Client): Validate rule's params if we get an error here we throw an error and we stop completely the bulk update Game over
   Kibana (Rule Client) --> Kibana (Rule Client): Validate rule's actions if we get an error here we throw an error and we stop completely the bulk update Game over

Instead of 'game over', why can't we move this rule into errors array and proceed with updating the rest of rules?
Then response would look something like this:

 { rules: Rule[], errors: Error[] }

5. actions also need to be move to bulkUpdate actions like:

export type BulkUpdateAction =
| {
   action: 'add' | 'delete' | 'set';
   field: 'tags';
   value: string[];
 }
| {
   action: 'add' | 'delete' | 'set';
   field: 'params.index';
   value: string[];
 }
| {
   action: 'add' | 'set';
   field: 'actions';
   value: NormalizedAlertAction[];
 };

as one of the requirements to add action to existed list of actions in rules or rewrite existing actions

[banderror]

@XavierM Ok, sorry for being late here, but I agree with all the concerns shared by @vitaliidm in #124715 (comment)

Let me reiterate over the main points.

1. We need to be able to match rules that we need to update either by query or by a list of rule ids (SO ids). So the proposed interface of the bulk update options is not sufficient:

   filter: KueryNode;

   Without this ability we won't be able to use the new method in our bulk actions endpoint.

2. This is important: we need to be able to update each rule individually with its own parameter/field values. Example:
   Not this: update rule 1, 2 and 3 and set tags = ['foo', 'bar'] for each of them.
   But this: update rule 1 and set tags to ['foo'], update rule 2 and set tags to ['bar'], update rule 3 and set tags to [].

   This is why the proposed interface is not sufficient, because it will only allow to update N rules with the same parameter values (e.g. the same name, the same set of tags, etc):

     export interface BulkUpdateOptions<Params extends AlertTypeParams> {
      filter: KueryNode;
      data: {
        name?: string;
        tags?: string[];
        schedule?: IntervalSchedule;
        actions?: NormalizedAlertAction[];
        params?: Params;
        throttle?: string | null;
        notifyWhen?: AlertNotifyWhenType | null;
      };
     }

3. Performance optimizations that we'd like to have:

   • Rules to be updated should be fetched in bulk in a single request to Elasticsearch (instead of N requests where N is the number of rules matching the query or the list of rule ids). Our original expectation was that this is going to be implemented and exposed as a separate bulkGet method in RulesClient as part of [RAM] Bulk get on rules #125652
   • Rules to be updated should be actually updated in a single request to Elasticsearch. Our original expectation was that this is going to be implemented and exposed as a separate bulkUpdate method in RulesClient as part of this ticket.
   • We expect the Framework/Core to use the ES bulk API for doing the above operations in a single request to ES.

If we decide to combine the bulk get-modify-update functionality in a single method of RulesClient (instead of separate bulk get and bulk update methods), I'd suggest to expose it as 2 separate bulkEditByFilter and bulkEditByIds (naming it "edit" because the semantics is different) methods with the following parameters:

type Rule<Params extends AlertTypeParams> = SanitizedAlert<Params>;

interface RulePropsModified<Params extends AlertTypeParams> {
  name?: string;
  tags?: string[];
  schedule?: IntervalSchedule;
  actions?: NormalizedAlertAction[];
  params?: Params;
  throttle?: string | null;
  notifyWhen?: RuleNotifyWhenType | null;
}

type RuleModifier<Params extends AlertTypeParams> = (rule: Rule<Params>) => RulePropsModified<Params>;

interface BulkEditByFilterArgs<Params extends AlertTypeParams> {
  filter: string; // KQL string
  modifier: RuleModifier<Params>;
}

interface BulkEditByIdsArgs<Params extends AlertTypeParams> {
  ids: string[]; // rule SO ids
  modifier: RuleModifier<Params>;
}

I don't think it'd be a good idea to expose something like this:

actions: Array<{ type: add_tags | delete_tags| set_tags, value: string[]}>[]

or this:

actions: Array<{ type: 'add' | 'delete' | 'set',  field: 'params.index' | 'tags', value: string[]}>[]

Bulk editing logic can contain something specific to our solution, so I'd keep it flexible by using a callback-based approach (like RuleModifier<Params>).

[XavierM]

@vitaliidm and @banderror and I had a good conversation today about validation and mutation of our attributes in the rule's saved object. I was able to bring back our idea to my team too. And it is pretty close to what we decided with each other, so you should not be surprise about our decision.

First of all, we think it will be good idea to add a new property in AlertTypeParamsValidator to allow rule type to manage their own validation around their params. We think validate is more here to validate the schema of the params object but validateMutatedParams will be there to do more logical validation for the need of the rule type creator.

export interface AlertTypeParamsValidator<Params extends AlertTypeParams> {
  validate: (object: unknown) => Params;
  validateMutatedParams?: (mutatedOject: unknown, origObject?: unknown) => Params;
}

I think we all agree that it is better to combine the bulk get-modify-update functionality in a single method of RulesClient, I think the main reason is to avoid multiple queries on the rules and also to validate the authorization of user on the rules. So We think that @banderror is right to call it BulkEdit.... I think we like to control the bulk update of our own attributes like actions, name, notifyWhen, schedule, tags, throttle but let you do what you need to do with params because you own this data. Let's focus on tags, actions and params.

type RuleParamsModifier<Params extends AlertTypeParams> = (params: Params) => Params;

type BulkEditActionRuleFields = keyof Pick<Alert, 'name' | 'actions' | 'schedule' | 'tags' | 'throttle' | 'notifyWhen' ;
interface BulkEditAction {
  type: 'add' | 'delete' | 'set';
  field: BulkEditActionRuleFields;
  value: Rule[BulkEditActionRuleFields];
}

interface BulkEditByFilterArgs<Params extends AlertTypeParams> {
  actions?: BulkEditAction[];
  filter: KueryNode;
  paramsModifier?: RuleParamsModifier<Params>;
}

[banderror]

I was checking our bulk actions endpoint implementation and noticed a few things that I think need to be discussed. Please refer to these files:

• kibana/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/perform_bulk_action_route.ts

  Lines 343 to 375 in 3713296

  	case BulkAction.edit:
  	bulkActionOutcome = await initPromisePool({
  	concurrency: MAX_RULES_TO_UPDATE_IN_PARALLEL,
  	items: rules,
  	executor: async (rule) => {
  	if (rule.params.immutable) {
  	throw new BadRequestError('Elastic rule can`t be edited');
  	}
  	throwAuthzError(await mlAuthz.validateRuleType(rule.params.type));
  	const editedRule = body[BulkAction.edit].reduce(
  	(acc, action) => applyBulkActionEditToRule(acc, action),
  	rule
  	);
  	const { tags, params: { timelineTitle, timelineId } = {} } = editedRule;
  	const index = 'index' in editedRule.params ? editedRule.params.index : undefined;
  	await patchRules({
  	rulesClient,
  	rule,
  	tags,
  	index,
  	timelineTitle,
  	timelineId,
  	});
  	return editedRule;
  	},
  	abortSignal: abortController.signal,
  	});
  	break;

• kibana/x-pack/plugins/security_solution/server/lib/detection_engine/rules/patch_rules.ts

  Lines 37 to 231 in bbb531a

  	export const patchRules = async ({
  	rulesClient,
  	author,
  	buildingBlockType,
  	description,
  	eventCategoryOverride,
  	falsePositives,
  	enabled,
  	query,
  	language,
  	license,
  	outputIndex,
  	savedId,
  	timelineId,
  	timelineTitle,
  	meta,
  	filters,
  	from,
  	index,
  	interval,
  	maxSignals,
  	riskScore,
  	riskScoreMapping,
  	ruleNameOverride,
  	rule,
  	name,
  	severity,
  	severityMapping,
  	tags,
  	threat,
  	threshold,
  	threatFilters,
  	threatIndex,
  	threatIndicatorPath,
  	threatQuery,
  	threatMapping,
  	threatLanguage,
  	concurrentSearches,
  	itemsPerSearch,
  	timestampOverride,
  	throttle,
  	to,
  	type,
  	references,
  	namespace,
  	note,
  	version,
  	exceptionsList,
  	anomalyThreshold,
  	machineLearningJobId,
  	actions,
  	}: PatchRulesOptions): Promise<PartialAlert<RuleParams> | null> => {
  	if (rule == null) {
  	return null;
  	}
  	const calculatedVersion = calculateVersion(rule.params.immutable, rule.params.version, {
  	author,
  	buildingBlockType,
  	description,
  	eventCategoryOverride,
  	falsePositives,
  	query,
  	language,
  	license,
  	outputIndex,
  	savedId,
  	timelineId,
  	timelineTitle,
  	meta,
  	filters,
  	from,
  	index,
  	interval,
  	maxSignals,
  	riskScore,
  	riskScoreMapping,
  	ruleNameOverride,
  	name,
  	severity,
  	severityMapping,
  	tags,
  	threat,
  	threshold,
  	threatFilters,
  	threatIndex,
  	threatIndicatorPath,
  	threatQuery,
  	threatMapping,
  	threatLanguage,
  	concurrentSearches,
  	itemsPerSearch,
  	timestampOverride,
  	to,
  	type,
  	references,
  	version,
  	namespace,
  	note,
  	exceptionsList,
  	anomalyThreshold,
  	machineLearningJobId,
  	});
  	const nextParams = defaults(
  	{
  	...rule.params,
  	},
  	{
  	author,
  	buildingBlockType,
  	description,
  	falsePositives,
  	from,
  	query,
  	language,
  	license,
  	outputIndex,
  	savedId,
  	timelineId,
  	timelineTitle,
  	meta,
  	filters,
  	index,
  	maxSignals,
  	riskScore,
  	riskScoreMapping,
  	ruleNameOverride,
  	severity,
  	severityMapping,
  	threat,
  	threshold: threshold ? normalizeThresholdObject(threshold) : undefined,
  	threatFilters,
  	threatIndex,
  	threatIndicatorPath,
  	threatQuery,
  	threatMapping,
  	threatLanguage,
  	concurrentSearches,
  	itemsPerSearch,
  	timestampOverride,
  	to,
  	type,
  	references,
  	namespace,
  	note,
  	version: calculatedVersion,
  	exceptionsList,
  	anomalyThreshold,
  	machineLearningJobId: machineLearningJobId
  	? normalizeMachineLearningJobIds(machineLearningJobId)
  	: undefined,
  	}
  	);
  	const newRule = {
  	tags: addTags(tags ?? rule.tags, rule.params.ruleId, rule.params.immutable),
  	name: calculateName({ updatedName: name, originalName: rule.name }),
  	schedule: {
  	interval: calculateInterval(interval, rule.schedule.interval),
  	},
  	params: removeUndefined(nextParams),
  	actions: actions?.map(transformRuleToAlertAction) ?? rule.actions,
  	throttle: throttle !== undefined ? transformToAlertThrottle(throttle) : rule.throttle,
  	notifyWhen: throttle !== undefined ? transformToNotifyWhen(throttle) : rule.notifyWhen,
  	};
  	const [validated, errors] = validate(newRule, internalRuleUpdate);
  	if (errors != null || validated === null) {
  	throw new PatchError(`Applying patch would create invalid rule: ${errors}`, 400);
  	}
  	const update = await rulesClient.update({
  	id: rule.id,
  	data: validated,
  	});
  	if (throttle !== undefined) {
  	await maybeMute({ rulesClient, muteAll: rule.muteAll, throttle, id: update.id });
  	}
  	if (rule.enabled && enabled === false) {
  	await rulesClient.disable({ id: rule.id });
  	} else if (!rule.enabled && enabled === true) {
  	await rulesClient.enable({ id: rule.id });
  	} else {
  	// enabled is null or undefined and we do not touch the rule
  	}
  	if (enabled != null) {
  	return { ...update, enabled };
  	} else {
  	return update;
  	}
  	};

You can notice that there are some actions we do before and after calling the rulesClient.update in the implementation of bulk editing:

1. [Before][Async] ML auth
2. [Before][Sync] Calculating next rule version (params.version)
3. [Before][Sync] Calculating next params, name, tags, actions, schedule
4. [Before][Sync] Validating rulesClient.update parameters
5. [After][Async] Maybe muting/unmuting the rule
6. [After][Async] Maybe enabling/disabling the rule

How are we going to support all of that using the new RulesClient.bulkEdit method? For each of these items, we will need to either provide some support/API in the method, or reimplement it in a different way. Some options off the top of my head:

1. [Before][Async] ML auth
   • provide an async pre-update hook in RulesClient.bulkEdit
   • make paramsModifier async
   • perform this validation outside of RulesClient.bulkEdit - in this case we'd need to duplicate fetching the rules or do it with an aggregation query returning all rule types matching the filter/ids - both seem to be prone to race conditions
2. [Before][Sync] Calculating next rule version (params.version)
   • seems like could be done in the paramsModifier
3. [Before][Sync] Calculating next params, name, tags, actions, schedule
   • seems like could be done in the paramsModifier and actions
4. [Before][Sync] Validating rulesClient.update parameters
   • validateMutatedParams?
5. [After][Async] Maybe muting/unmuting the rule
   • We don't need it for bulk editing right now, but it's possible we'll need it for other bulk editing types, bulk actions, or bulk endpoints in the future.
6. [After][Async] Maybe enabling/disabling the rule
   • We don't need it for bulk editing right now, but it's possible we'll need it for other bulk editing types, bulk actions, or bulk endpoints in the future.

@vitaliidm do you have an idea how we're going to handle each and would we need any specific API like async hooks in the bulkEdit method?

[banderror]

@vitaliidm @XavierM Do we have a proposal or a draft of the bulk edit method result? Since it's gonna be used in the existing bulk actions endpoint with a specific existing response structure, it will need to return all the data required to build the response.

Here's where the response is being built:

kibana/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/perform_bulk_action_route.ts

Lines 86 to 135 in 3713296

	const buildBulkResponse = (
	response: KibanaResponseFactory,
	fetchRulesOutcome: PromisePoolOutcome<string, RuleAlertType>,
	bulkActionOutcome: PromisePoolOutcome<RuleAlertType, RuleAlertType | null>
	) => {
	const errors = [...fetchRulesOutcome.errors, ...bulkActionOutcome.errors];
	const summary = {
	failed: errors.length,
	succeeded: bulkActionOutcome.results.length,
	total: bulkActionOutcome.results.length + errors.length,
	};
	const results = {
	updated: bulkActionOutcome.results
	.filter(({ item, result }) => item.id === result?.id)
	.map(({ result }) => result && internalRuleToAPIResponse(result)),
	created: bulkActionOutcome.results
	.filter(({ item, result }) => result != null && result.id !== item.id)
	.map(({ result }) => result && internalRuleToAPIResponse(result)),
	deleted: bulkActionOutcome.results
	.filter(({ result }) => result == null)
	.map(({ item }) => internalRuleToAPIResponse(item)),
	};
	if (errors.length > 0) {
	return response.custom({
	headers: { 'content-type': 'application/json' },
	body: Buffer.from(
	JSON.stringify({
	message: summary.succeeded > 0 ? 'Bulk edit partially failed' : 'Bulk edit failed',
	status_code: 500,
	attributes: {
	errors: normalizeErrorResponse(errors),
	results,
	summary,
	},
	})
	),
	statusCode: 500,
	});
	}
	return response.ok({
	body: {
	success: true,
	rules_count: summary.total,
	attributes: { results, summary },
	},
	});
	};

[banderror]

@XavierM so regarding the filter (filter: KueryNode;),

• Why it's KueryNode and not string and what are the benefits of that?
• How are we going to implement filtering by a list of rule ids?

[vitaliidm]

@vitaliidm @XavierM Do we have a proposal or a draft of the bulk edit method result? Since it's gonna be used in the existing bulk actions endpoint with a specific existing response structure, it will need to return all the data required to build the response.

this is a draft POC PR. After rulesClient.bulkEdit method is implemented - it will be placed instead of current implementation in a separate PR

@XavierM so regarding the filter (filter: KueryNode;),
Why it's KueryNode and not string and what are the benefits of that?
How are we going to implement filtering by a list of rule ids?

1. we will support both string and KueryNode. PR will be updated
2. we use filter property for this, by transforming list of ids into query(from PR):

query = body.ids.map((id) => `alert.id:"alert:${id}"`).join(' OR ');

[Before][Async] ML auth

Would make sense to do this check in async paramsModifier. in this case we need to make paramsModifier async.

[Before][Sync] Calculating next rule version (params.version)
seems like could be done in the paramsModifier

agree

[Before][Sync] Calculating next params, name, tags, actions, schedule
seems like could be done in the paramsModifier and actions

Looks like, it mostly patch specific logic (calculating params using defaults) And. most of properties that changed here should be handled within actions, within bulkEdit(update and validation)

[Before][Sync] Validating rulesClient.update parameters
validateMutatedParams?

I can see, it validates not only params(which can be validated in validateMutatedParams but other properties:

 name,
  tags,
  schedule: t.type({
    interval: t.string,
  }),
  actions: actionsCamel,
  throttle: throttleOrNull,
  notifyWhen,

As update of these properties will be done inside bulkEdit method, looks like we don't need to do additional validation, as it should be done within bulkEdit method. Unless, these properties can be deemed as valid for rulesClient, but invalid for us(securitySolution/detections). Can this happen?

@banderror , @XavierM

[banderror]

Had a chat with @vitaliidm on the above points, here's the summary.

Aligned on

[Before][Async] ML auth

We agreed that doing this ML auth would be best in the paramsModifier function. We will need to make it async to do that. This would be the simplest and actually the most performant implementation out of all the discussed options.

type RuleParamsModifier<Params extends AlertTypeParams> = async (params: Params) => Params;

So it would be expected that this function could do pre-validation and throw exceptions with validation messages.

[Before][Sync] Calculating next rule version (params.version)

We will just increment it in the paramsModifier function. We'll increment it always regardless of the endpoint attributes or whether the rule params object is going to be changed.

[Before][Sync] Calculating next params, name, tags, actions, schedule

• name, tags, actions, schedule: this calculation will be done internally in the bulk edit method
• params: this will be done in the paramsModifier callback in the bulk actions endpoint

[Before][Sync] Validating rulesClient.update parameters

The bulk edit method will validate the rule name, tags, etc itself. If we need any additional validation for our rule params object, we will be able to move it to validateMutatedParams.

Some other outstanding questions

@XavierM we have a few new questions to clarify and would appreciate any feedback from your side, if you have some time to review them:

export interface AlertTypeParamsValidator<Params extends AlertTypeParams> {
  validate: (object: unknown) => Params;
  validateMutatedParams?: (mutatedOject: unknown, origObject?: unknown) => Params;
}

• Why do we need a separate validateMutatedParams function? Can we put our "dynamic" params validation to the existing validate function? We'd need to add the second origObject?: unknown arg to it which would be a non-breaking change for existing rule types.
• Could validateMutatedParams accept whole rule objects instead of only params? This would make it more flexible and allow us to base our validation logic on fields like alertTypeId - if at some point we decide to remove params.type from our schema because it's redundant.
• Are validate and validateMutatedParams expected to be able to change the params object in addition to validating it? E.g. set default values. Technically their interfaces allow that. If this is not expected, should the return type be void or some validation result?

Regarding the filter (filter: KueryNode;), how are we going to implement filtering by a list of rule ids?
query = body.ids.map((id) => alert.id:"alert:${id}").join(' OR ');

• Can Response Ops guarantee that the rule SO type (alert) will never be migrated to a different value (e.g. rule) in the future? With the proposed implementation, it would break filtering by rule ids on the solution side.
• Wouldn't it be more developer-friendly to encapsulate building such a filter and expose a separate method that would accept ids: string[]?

[vitaliidm]

@banderror , @XavierM

Can Response Ops guarantee that the rule SO type (alert) will never be migrated to a different value (e.g. rule) in the future? With the proposed implementation, it would break filtering by rule ids on the solution side.
Wouldn't it be more developer-friendly to encapsulate building such a filter and expose a separate method that would accept ids: string[]?

We agreed on introducing ids parameter in bulkEdit method

Could validateMutatedParams accept whole rule objects instead of only params? This would make it more flexible and allow us to base our validation logic on fields like alertTypeId - if at some point we decide to remove params.type from our schema because it's redundant.

We can extend it to passing the whole rule object, rather than only params. But in this case validateMutatedParams should still return only params. As everything, outside params, is responsibility of alerting team.
But is there a valid use case right now to have access to the whole object? We can use the principle of the least privilege here and if there is a need in future - extend this interface, updating this method step by step.

Are validate and validateMutatedParams expected to be able to change the params object in addition to validating it? E.g. set default values. Technically their interfaces allow that. If this is not expected, should the return type be void or some validation result?

Yes, params can be changed, similarly to validate method. We can leave current signature that returns params, so it will be consistent with validate method.

[banderror]

@vitaliidm thanks for the update. Sounds good to me!

But is there a valid use case right now to have access to the whole object? We can use the principle of the least privilege here and if there is a need in future - extend this interface, updating this method step by step.

I'm not aware of any immediate use cases for this right now.

[vitaliidm]

@XavierM
I have discovered one limitation with current implementation of applying operation to tags field.

Currently, each rule on our side has so called internals tags

GET /.kibana/_doc/alert:9c4bbac0-b5cb-11ec-8f1e-adaa7d7d57e5

{
  "_index" : ".kibana_8.3.0_001",
  "_id" : "alert:9c4bbac0-b5cb-11ec-8f1e-adaa7d7d57e5",
  "_version" : 1,
  "_seq_no" : 2116,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "alert" : {
      "name" : "Web Application Suspicious Activity: sqlmap User Agent [Duplicate]",
      "tags" : [
        "Elastic",
        "APM",
        "__internal_rule_id:2e41245d-4dee-42f7-92da-0701e9567463",
        "__internal_immutable:false"
      ],
      "alertTypeId" : "siem.queryRule",
      "consumer" : "siem",
      "params" : {
        "author" : [
          "Elastic"
        ],
        "description" : "This is an example of how to detect an unwanted web client user agent. This search matches the user agent for sqlmap 1.3.11, which is a popular FOSS tool for testing web applications for SQL injection vulnerabilities.",
        "ruleId" : "2e41245d-4dee-42f7-92da-0701e9567463",
        "falsePositives" : [
          "This rule does not indicate that a SQL injection attack occurred, only that the `sqlmap` tool was used. Security scans and tests may result in these errors. If the source is not an authorized security tester, this is generally suspicious or malicious activity."
        ],
        "from" : "now-6m",
        "immutable" : false,
        "license" : "Elastic License v2",
        "outputIndex" : ".siem-signals-default",
        "maxSignals" : 100,
        "riskScore" : 47,
        "riskScoreMapping" : [ ],
        "severity" : "medium",
        "severityMapping" : [ ],
        "threat" : [ ],
        "timestampOverride" : "event.ingested",
        "to" : "now",
        "references" : [
          "http://sqlmap.org/"
        ],
        "version" : 7,
        "exceptionsList" : [ ],
        "index" : [
          "apm-*-transaction*",
          "traces-apm*"
        ],
        "query" : """user_agent.original:"sqlmap/1.3.11#stable (http://sqlmap.org)"
""",
        "language" : "kuery",
        "type" : "query"
      },
      "schedule" : {
        "interval" : "5m"
      },
      "enabled" : false,
      "actions" : [ ],
      "throttle" : null,
      "notifyWhen" : "onActiveAlert",
      "apiKeyOwner" : null,
      "apiKey" : null,
      "legacyId" : null,
      "createdBy" : "elastic",
      "updatedBy" : "elastic",
      "createdAt" : "2022-04-06T17:04:24.687Z",
      "updatedAt" : "2022-04-06T17:04:24.687Z",
      "snoozeEndTime" : null,
      "muteAll" : false,
      "mutedInstanceIds" : [ ],
      "executionStatus" : {
        "status" : "pending",
        "lastExecutionDate" : "2022-04-06T17:04:24.687Z",
        "error" : null,
        "warning" : null
      },
      "monitoring" : {
        "execution" : {
          "history" : [ ],
          "calculated_metrics" : {
            "success_ratio" : 0
          }
        }
      },
      "mapped_params" : {
        "risk_score" : 47,
        "severity" : "40-medium"
      },
      "meta" : {
        "versionApiKeyLastmodified" : "8.3.0"
      }
    },
    "type" : "alert",
    "references" : [ ],
    "namespaces" : [
      "default"
    ],
    "migrationVersion" : {
      "alert" : "8.2.0"
    },
    "coreMigrationVersion" : "8.3.0",
    "updated_at" : "2022-04-06T17:04:24.687Z"
  }
}

Here is tags:

      "tags" : [
        "Elastic",
        "APM",
        "__internal_rule_id:2e41245d-4dee-42f7-92da-0701e9567463",
        "__internal_immutable:false"
      ],

These tags cannot be rewritten by user and set internally by calling this method
https://github.com/elastic/kibana/tree/main/x-pack/plugins/security_solution/server/lib/detection_engine/rules/add_tags.ts, whenever we update rule.

With current implementation, such payload of bulkEdit would wipe out internal tags

{
    "ids": ["9c4bbac0-b5cb-11ec-8f1e-adaa7d7d57e5"],
    "operations":  [{
        "operation": "set",
        "field": "tags",
        "value": ["alerting-in-default"]
    }]
}

and field tags in rule would become tags: ["alerting-in-default"].

But expectation on our side would be, it to become:

     "tags" : [
        "alerting-in-defaul",
        "__internal_rule_id:2e41245d-4dee-42f7-92da-0701e9567463",
        "__internal_immutable:false"
      ],

How we can resolve this?

1. Do not remove tags with __internal prefix? Filter these tags within rulesClient.bulkEdit?
   This seems like securitySolution piece of logic and not shared with alerting.
2. Extend paramsModifier, so it can change tags as well and, not only rule params?
3. Maybe some other approach?

[banderror]

@vitaliidm Perhaps the best place for ensuring these internal tags are always there would be the validateMutatedParams function?

The presence of these internal tags is an invariant that must be always respected. Actually, there are two invariants:

• every rule must have a global rule_id and a corresponding __internal_rule_id tag containing it
• every rule must have an __internal_immutable tag reflecting the current value of rule.params.immutable

Since these are invariants and they do not depend on any external variables, only on the rule object itself, I think we could keep this normalization logic in validateMutatedParams. It would need to accept whole rule objects and return something more than just a rule params object.

[XavierM]

@banderror, @vitaliidm, I am thinking about it a little bit differently here, why we are not moving these internal tags in a different attributes like internal_tags and since you are our only user who are having internal tags, I think it will make sense to move this new attribute in params. Then we just need to write a migration for it and keep the code the way it is. What do you think?

[vitaliidm]

@XavierM , that would require to change our logic that uses internal tags across the app. For example filtering Elastic/custom rules and so on. I had a quick look, there are not that many places that uses it(few places for immutable tag, few for internal rule)

@banderror , any additional concerns you can think of, that might rise if try to move internal tags to a separate field within params?

[banderror]

@XavierM @vitaliidm I like the idea 👍 But I suspect we could even completely get rid of all those internal tags and wouldn't need to migrate them to params. internal_tags.

I don't have any historical context on why we have internal tags, but my guess is because we were not able to filter by params.immutable and params.rule_id in the past when params was mapped as object with disabled indexing => it wasn't filterable at all. Now, when it's mapped as a flattened field, we can adjust our app to filter by params.immutable and params.rule_id instead of the internal tags.

I'll try to figure out if that makes sense or there's another use case for internal tags that I'm missing.

[spong]

I don't have any historical context on why we have internal tags, but my guess is because we were not able to filter by params.immutable and params.rule_id in the past when params was mapped as object with disabled indexing => it wasn't filterable at all. Now, when it's mapped as a flattened field, we can adjust our app to filter by params.immutable and params.rule_id instead of the internal tags.

Yup! That's correct -- I created this issue (#124899) a little while back to remove our internal_tags implementation/hack completely. We should be able to move rule_id and immutable to searchable params and remove internal_tags 🙂

[banderror]

Awesome-awesome! So my vote would go for addressing #124899 first and then finalizing the bulkEdit method.

[vitaliidm]

Fantastic, thanks @XavierM , @spong and @banderror for clearing out situation with internal tags

[vitaliidm]

Implemented in #126904