[SavedObjects] Add aggregations support (#96292)

* step 1 to add aggs in the find function of saved object

* setp 2 - add specific unit test to aggs + fix bug found during integrations

* step 3 - add security api_integration arounds aggs

* fix types

* unit test added for aggs_utils

* add documentation

* fix docs

* review I

* doc

* try to fix test

* add the new property to the saved object globaltype

* fix types

* delete old files

* fix types + test api integration

* type fix + test

* Update src/core/server/saved_objects/types.ts

Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>

* review I

* change our validation to match discussion with Pierre and Rudolph

* Validate multiple items nested filter query through KueryNode

* remove unused import

* review + put back test

* migrate added tests to new TS file

* fix documentation

* fix license header

* move stuff

* duplicating test mappings

* rename some stuff

* move ALL the things

* cast to aggregation container

* update generated doc

* add deep nested validation

* rewrite the whole validation mechanism

* some cleanup

* minor cleanup

* update generated doc

* adapt telemetry client

* fix API integ tests

* fix doc

* TOTO-less

* remove xpack tests

* list supported / unsupported aggregations

* typo fix

* extract some validation function

* fix indent

* add some unit tests

* adapt FTR assertions

* update doc

* fix doc

* doc again

* cleanup test names

* improve tsdoc on validation functions

* perf nit

Co-authored-by: Xavier Mouligneau <189600+XavierM@users.noreply.github.com>
Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

docs/api/saved-objects/find.asciidoc

@@ -53,9 +53,14 @@ experimental[] Retrieve a paginated set of {kib} saved objects by various condit
   (Optional, object) Filters to objects that have a relationship with the type and ID combination.
 
 `filter`::
-  (Optional, string) The filter is a KQL string with the caveat that if you filter with an attribute from your type saved object.
-  It should look like that savedObjectType.attributes.title: "myTitle". However, If you used a direct attribute of a saved object like `updatedAt`,
-  you will have to define your filter like that savedObjectType.updatedAt > 2018-12-22.
+  (Optional, string) The filter is a KQL string with the caveat that if you filter with an attribute from your saved object type,
+  it should look like that: `savedObjectType.attributes.title: "myTitle"`. However, If you use a root attribute of a saved 
+  object such as `updated_at`, you will have to define your filter like that: `savedObjectType.updated_at > 2018-12-22`.
+
+`aggs`::
+  (Optional, string) **experimental** An aggregation structure, serialized as a string. The field format is similar to `filter`, meaning
+  that to use a saved object type attribute in the aggregation, the `savedObjectType.attributes.title`: "myTitle"` format
+  must be used. For root fields, the syntax is `savedObjectType.rootField`
 
 NOTE: As objects change in {kib}, the results on each page of the response also
 change. Use the find API for traditional paginated results, but avoid using it to export large amounts of data.

docs/development/core/public/kibana-plugin-core-public.savedobjectsclient.find.md

@@ -9,5 +9,5 @@ Search for objects
 <b>Signature:</b>
 
 ```typescript
-find: <T = unknown>(options: SavedObjectsFindOptions) => Promise<SavedObjectsFindResponsePublic<T>>;
+find: <T = unknown, A = unknown>(options: SavedObjectsFindOptions) => Promise<SavedObjectsFindResponsePublic<T, unknown>>;
 ```

docs/development/core/public/kibana-plugin-core-public.savedobjectsclient.md

@@ -24,7 +24,7 @@ The constructor for this class is marked as internal. Third-party code should no
 |  [bulkGet](./kibana-plugin-core-public.savedobjectsclient.bulkget.md) |  | <code>(objects?: Array&lt;{</code><br/><code>        id: string;</code><br/><code>        type: string;</code><br/><code>    }&gt;) =&gt; Promise&lt;SavedObjectsBatchResponse&lt;unknown&gt;&gt;</code> | Returns an array of objects by id |
 |  [create](./kibana-plugin-core-public.savedobjectsclient.create.md) |  | <code>&lt;T = unknown&gt;(type: string, attributes: T, options?: SavedObjectsCreateOptions) =&gt; Promise&lt;SimpleSavedObject&lt;T&gt;&gt;</code> | Persists an object |
 |  [delete](./kibana-plugin-core-public.savedobjectsclient.delete.md) |  | <code>(type: string, id: string, options?: SavedObjectsDeleteOptions &#124; undefined) =&gt; ReturnType&lt;SavedObjectsApi['delete']&gt;</code> | Deletes an object |
-|  [find](./kibana-plugin-core-public.savedobjectsclient.find.md) |  | <code>&lt;T = unknown&gt;(options: SavedObjectsFindOptions) =&gt; Promise&lt;SavedObjectsFindResponsePublic&lt;T&gt;&gt;</code> | Search for objects |
+|  [find](./kibana-plugin-core-public.savedobjectsclient.find.md) |  | <code>&lt;T = unknown, A = unknown&gt;(options: SavedObjectsFindOptions) =&gt; Promise&lt;SavedObjectsFindResponsePublic&lt;T, unknown&gt;&gt;</code> | Search for objects |
 |  [get](./kibana-plugin-core-public.savedobjectsclient.get.md) |  | <code>&lt;T = unknown&gt;(type: string, id: string) =&gt; Promise&lt;SimpleSavedObject&lt;T&gt;&gt;</code> | Fetches a single object |
 
 ## Methods

docs/development/core/public/kibana-plugin-core-public.savedobjectsfindresponsepublic.aggregations.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-public](./kibana-plugin-core-public.md) &gt; [SavedObjectsFindResponsePublic](./kibana-plugin-core-public.savedobjectsfindresponsepublic.md) &gt; [aggregations](./kibana-plugin-core-public.savedobjectsfindresponsepublic.aggregations.md)
+
+## SavedObjectsFindResponsePublic.aggregations property
+
+<b>Signature:</b>
+
+```typescript
+aggregations?: A;
+```

docs/development/core/public/kibana-plugin-core-public.savedobjectsfindresponsepublic.md

@@ -11,13 +11,14 @@ Return type of the Saved Objects `find()` method.
 <b>Signature:</b>
 
 ```typescript
-export interface SavedObjectsFindResponsePublic<T = unknown> extends SavedObjectsBatchResponse<T> 
+export interface SavedObjectsFindResponsePublic<T = unknown, A = unknown> extends SavedObjectsBatchResponse<T> 
 ```
 
 ## Properties
 
 |  Property | Type | Description |
 |  --- | --- | --- |
+|  [aggregations](./kibana-plugin-core-public.savedobjectsfindresponsepublic.aggregations.md) | <code>A</code> |  |
 |  [page](./kibana-plugin-core-public.savedobjectsfindresponsepublic.page.md) | <code>number</code> |  |
 |  [perPage](./kibana-plugin-core-public.savedobjectsfindresponsepublic.perpage.md) | <code>number</code> |  |
 |  [total](./kibana-plugin-core-public.savedobjectsfindresponsepublic.total.md) | <code>number</code> |  |

docs/development/core/server/kibana-plugin-core-server.savedobjectsclient.find.md

@@ -9,7 +9,7 @@ Find all SavedObjects matching the search query
 <b>Signature:</b>
 
 ```typescript
-find<T = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindResponse<T>>;
+find<T = unknown, A = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindResponse<T, A>>;
 ```
 
 ## Parameters
@@ -20,5 +20,5 @@ find<T = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindRes
 
 <b>Returns:</b>
 
-`Promise<SavedObjectsFindResponse<T>>`
+`Promise<SavedObjectsFindResponse<T, A>>`
 

docs/development/core/server/kibana-plugin-core-server.savedobjectsfindresponse.aggregations.md

@@ -0,0 +1,11 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [SavedObjectsFindResponse](./kibana-plugin-core-server.savedobjectsfindresponse.md) &gt; [aggregations](./kibana-plugin-core-server.savedobjectsfindresponse.aggregations.md)
+
+## SavedObjectsFindResponse.aggregations property
+
+<b>Signature:</b>
+
+```typescript
+aggregations?: A;
+```

docs/development/core/server/kibana-plugin-core-server.savedobjectsfindresponse.md

@@ -11,13 +11,14 @@ Return type of the Saved Objects `find()` method.
 <b>Signature:</b>
 
 ```typescript
-export interface SavedObjectsFindResponse<T = unknown> 
+export interface SavedObjectsFindResponse<T = unknown, A = unknown> 
 ```
 
 ## Properties
 
 |  Property | Type | Description |
 |  --- | --- | --- |
+|  [aggregations](./kibana-plugin-core-server.savedobjectsfindresponse.aggregations.md) | <code>A</code> |  |
 |  [page](./kibana-plugin-core-server.savedobjectsfindresponse.page.md) | <code>number</code> |  |
 |  [per\_page](./kibana-plugin-core-server.savedobjectsfindresponse.per_page.md) | <code>number</code> |  |
 |  [pit\_id](./kibana-plugin-core-server.savedobjectsfindresponse.pit_id.md) | <code>string</code> |  |

docs/development/core/server/kibana-plugin-core-server.savedobjectsrepository.find.md

@@ -7,7 +7,7 @@
 <b>Signature:</b>
 
 ```typescript
-find<T = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindResponse<T>>;
+find<T = unknown, A = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindResponse<T, A>>;
 ```
 
 ## Parameters
@@ -18,7 +18,7 @@ find<T = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindRes
 
 <b>Returns:</b>
 
-`Promise<SavedObjectsFindResponse<T>>`
+`Promise<SavedObjectsFindResponse<T, A>>`
 
 {<!-- -->promise<!-- -->} - { saved\_objects: \[{ id, type, version, attributes }<!-- -->\], total, per\_page, page }
 

docs/development/core/server/kibana-plugin-core-server.savedobjectsutils.createemptyfindresponse.md

@@ -9,5 +9,5 @@ Creates an empty response for a find operation. This is only intended to be used
 <b>Signature:</b>
 
 ```typescript
-static createEmptyFindResponse: <T>({ page, perPage, }: SavedObjectsFindOptions) => SavedObjectsFindResponse<T>;
+static createEmptyFindResponse: <T, A>({ page, perPage, }: SavedObjectsFindOptions) => SavedObjectsFindResponse<T, A>;
 ```

docs/development/core/server/kibana-plugin-core-server.savedobjectsutils.md

@@ -15,7 +15,7 @@ export declare class SavedObjectsUtils
 
 |  Property | Modifiers | Type | Description |
 |  --- | --- | --- | --- |
-|  [createEmptyFindResponse](./kibana-plugin-core-server.savedobjectsutils.createemptyfindresponse.md) | <code>static</code> | <code>&lt;T&gt;({ page, perPage, }: SavedObjectsFindOptions) =&gt; SavedObjectsFindResponse&lt;T&gt;</code> | Creates an empty response for a find operation. This is only intended to be used by saved objects client wrappers. |
+|  [createEmptyFindResponse](./kibana-plugin-core-server.savedobjectsutils.createemptyfindresponse.md) | <code>static</code> | <code>&lt;T, A&gt;({ page, perPage, }: SavedObjectsFindOptions) =&gt; SavedObjectsFindResponse&lt;T, A&gt;</code> | Creates an empty response for a find operation. This is only intended to be used by saved objects client wrappers. |
 |  [namespaceIdToString](./kibana-plugin-core-server.savedobjectsutils.namespaceidtostring.md) | <code>static</code> | <code>(namespace?: string &#124; undefined) =&gt; string</code> | Converts a given saved object namespace ID to its string representation. All namespace IDs have an identical string representation, with the exception of the <code>undefined</code> namespace ID (which has a namespace string of <code>'default'</code>). |
 |  [namespaceStringToId](./kibana-plugin-core-server.savedobjectsutils.namespacestringtoid.md) | <code>static</code> | <code>(namespace: string) =&gt; string &#124; undefined</code> | Converts a given saved object namespace string to its ID representation. All namespace strings have an identical ID representation, with the exception of the <code>'default'</code> namespace string (which has a namespace ID of <code>undefined</code>). |
 

src/core/public/public.api.md

@@ -1224,7 +1224,7 @@ export class SavedObjectsClient {
     // Warning: (ae-forgotten-export) The symbol "SavedObjectsClientContract" needs to be exported by the entry point index.d.ts
     delete: (type: string, id: string, options?: SavedObjectsDeleteOptions | undefined) => ReturnType<SavedObjectsClientContract_2['delete']>;
     // Warning: (ae-forgotten-export) The symbol "SavedObjectsFindOptions" needs to be exported by the entry point index.d.ts
-    find: <T = unknown>(options: SavedObjectsFindOptions_2) => Promise<SavedObjectsFindResponsePublic<T>>;
+    find: <T = unknown, A = unknown>(options: SavedObjectsFindOptions_2) => Promise<SavedObjectsFindResponsePublic<T, unknown>>;
     get: <T = unknown>(type: string, id: string) => Promise<SimpleSavedObject<T>>;
     update<T = unknown>(type: string, id: string, attributes: T, { version, migrationVersion, references }?: SavedObjectsUpdateOptions): Promise<SimpleSavedObject<T>>;
 }
@@ -1244,6 +1244,8 @@ export interface SavedObjectsCreateOptions {
 
 // @public (undocumented)
 export interface SavedObjectsFindOptions {
+    // @alpha
+    aggs?: Record<string, estypes.AggregationContainer>;
     defaultSearchOperator?: 'AND' | 'OR';
     fields?: string[];
     // Warning: (ae-forgotten-export) The symbol "KueryNode" needs to be exported by the entry point index.d.ts
@@ -1284,7 +1286,9 @@ export interface SavedObjectsFindOptionsReference {
 }
 
 // @public
-export interface SavedObjectsFindResponsePublic<T = unknown> extends SavedObjectsBatchResponse<T> {
+export interface SavedObjectsFindResponsePublic<T = unknown, A = unknown> extends SavedObjectsBatchResponse<T> {
+    // (undocumented)
+    aggregations?: A;
     // (undocumented)
     page: number;
     // (undocumented)

src/core/public/saved_objects/saved_objects_client.ts

@@ -103,7 +103,9 @@ export interface SavedObjectsDeleteOptions {
  *
  * @public
  */
-export interface SavedObjectsFindResponsePublic<T = unknown> extends SavedObjectsBatchResponse<T> {
+export interface SavedObjectsFindResponsePublic<T = unknown, A = unknown>
+  extends SavedObjectsBatchResponse<T> {
+  aggregations?: A;
   total: number;
   perPage: number;
   page: number;
@@ -310,7 +312,7 @@ export class SavedObjectsClient {
    * @property {object} [options.hasReference] - { type, id }
    * @returns A find result with objects matching the specified search.
    */
-  public find = <T = unknown>(
+  public find = <T = unknown, A = unknown>(
     options: SavedObjectsFindOptions
   ): Promise<SavedObjectsFindResponsePublic<T>> => {
     const path = this.getPath(['_find']);
@@ -326,6 +328,7 @@ export class SavedObjectsClient {
       sortField: 'sort_field',
       type: 'type',
       filter: 'filter',
+      aggs: 'aggs',
       namespaces: 'namespaces',
       preference: 'preference',
     };
@@ -342,13 +345,20 @@ export class SavedObjectsClient {
       query.has_reference = JSON.stringify(query.has_reference);
     }
 
+    // `aggs` is a structured object. we need to stringify it before sending it, as `fetch`
+    // is not doing it implicitly.
+    if (query.aggs) {
+      query.aggs = JSON.stringify(query.aggs);
+    }
+
     const request: ReturnType<SavedObjectsApi['find']> = this.savedObjectsFetch(path, {
       method: 'GET',
       query,
     });
     return request.then((resp) => {
       return renameKeys<SavedObjectsFindResponse, SavedObjectsFindResponsePublic>(
         {
+          aggregations: 'aggregations',
           saved_objects: 'savedObjects',
           total: 'total',
           per_page: 'perPage',

src/core/server/saved_objects/routes/find.ts

@@ -44,6 +44,7 @@ export const registerFindRoute = (router: IRouter, { coreUsageData }: RouteDepen
           has_reference_operator: searchOperatorSchema,
           fields: schema.maybe(schema.oneOf([schema.string(), schema.arrayOf(schema.string())])),
           filter: schema.maybe(schema.string()),
+          aggs: schema.maybe(schema.string()),
           namespaces: schema.maybe(
             schema.oneOf([schema.string(), schema.arrayOf(schema.string())])
           ),
@@ -59,6 +60,20 @@ export const registerFindRoute = (router: IRouter, { coreUsageData }: RouteDepen
       const usageStatsClient = coreUsageData.getClient();
       usageStatsClient.incrementSavedObjectsFind({ request: req }).catch(() => {});
 
+      // manually validation to avoid using JSON.parse twice
+      let aggs;
+      if (query.aggs) {
+        try {
+          aggs = JSON.parse(query.aggs);
+        } catch (e) {
+          return res.badRequest({
+            body: {
+              message: 'invalid aggs value',
+            },
+          });
+        }
+      }
+
       const result = await context.core.savedObjects.client.find({
         perPage: query.per_page,
         page: query.page,
@@ -72,6 +87,7 @@ export const registerFindRoute = (router: IRouter, { coreUsageData }: RouteDepen
         hasReferenceOperator: query.has_reference_operator,
         fields: typeof query.fields === 'string' ? [query.fields] : query.fields,
         filter: query.filter,
+        aggs,
         namespaces,
       });
 

src/core/server/saved_objects/service/lib/aggregations/aggs_types/bucket_aggs.ts

@@ -0,0 +1,86 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { schema as s, ObjectType } from '@kbn/config-schema';
+
+/**
+ * Schemas for the Bucket aggregations.
+ *
+ * Currently supported:
+ * - filter
+ * - histogram
+ * - terms
+ *
+ * Not implemented:
+ * - adjacency_matrix
+ * - auto_date_histogram
+ * - children
+ * - composite
+ * - date_histogram
+ * - date_range
+ * - diversified_sampler
+ * - filters
+ * - geo_distance
+ * - geohash_grid
+ * - geotile_grid
+ * - global
+ * - ip_range
+ * - missing
+ * - multi_terms
+ * - nested
+ * - parent
+ * - range
+ * - rare_terms
+ * - reverse_nested
+ * - sampler
+ * - significant_terms
+ * - significant_text
+ * - variable_width_histogram
+ */
+export const bucketAggsSchemas: Record<string, ObjectType> = {
+  filter: s.object({
+    term: s.recordOf(s.string(), s.oneOf([s.string(), s.boolean(), s.number()])),
+  }),
+  histogram: s.object({
+    field: s.maybe(s.string()),
+    interval: s.maybe(s.number()),
+    min_doc_count: s.maybe(s.number()),
+    extended_bounds: s.maybe(
+      s.object({
+        min: s.number(),
+        max: s.number(),
+      })
+    ),
+    hard_bounds: s.maybe(
+      s.object({
+        min: s.number(),
+        max: s.number(),
+      })
+    ),
+    missing: s.maybe(s.number()),
+    keyed: s.maybe(s.boolean()),
+    order: s.maybe(
+      s.object({
+        _count: s.string(),
+        _key: s.string(),
+      })
+    ),
+  }),
+  terms: s.object({
+    field: s.maybe(s.string()),
+    collect_mode: s.maybe(s.string()),
+    exclude: s.maybe(s.oneOf([s.string(), s.arrayOf(s.string())])),
+    include: s.maybe(s.oneOf([s.string(), s.arrayOf(s.string())])),
+    execution_hint: s.maybe(s.string()),
+    missing: s.maybe(s.number()),
+    min_doc_count: s.maybe(s.number()),
+    size: s.maybe(s.number()),
+    show_term_doc_count_error: s.maybe(s.boolean()),
+    order: s.maybe(s.oneOf([s.literal('asc'), s.literal('desc')])),
+  }),
+};

src/core/server/saved_objects/service/lib/aggregations/aggs_types/index.ts

@@ -0,0 +1,15 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { bucketAggsSchemas } from './bucket_aggs';
+import { metricsAggsSchemas } from './metrics_aggs';
+
+export const aggregationSchemas = {
+  ...metricsAggsSchemas,
+  ...bucketAggsSchemas,
+};