Saved object filter KQL do not work with multiple nested filters

[nchaulet]

Description

I am trying to use nested filter in a saved object filter query using KQL, and it's throwing a validation error if I have multiple items in my nested query

This work

fleet-agents.attributes.inputs:{ package: "endpoint"}

This do not work

fleet-agents.attributes.inputs:{  status: "error" AND package: "endpoint"   }

The error

This key 'status' need to be wrapped by a saved object type like fleet-agents: Bad Request

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[pgayvallet]

@nchaulet I'm currently looking at the ast validation we are using for SO to try to fix that.

In the meantime, looking at the tests, I think that expanding the query should work?

fleet-agents.attributes.inputs.status: "error" AND fleet-agents.attributes.inputs.package: "endpoint"

Could you confirm that this 'workaround' is correctly working for your case.

[pgayvallet]

So, I can confirm we got an issue in validateFilterKueryNode (src/core/server/saved_objects/service/lib/filter_utils.ts). The function's logic doesn't seem to properly handle 'nested' function-type nodes/arguments, resulting on the AND function inside the nested filter to not being properly parsed/validated.

Note that this seems to only be a validation issue. By removing the validation step, I think the query is executed just fine against the index.

A simple test to reproduce the behavior (in src/core/server/saved_objects/service/lib/filter_utils.test.ts)

    test('Validate multiple items nested filter query through KueryNode', () => {
      const validationObject = validateFilterKueryNode({
        astFilter: esKuery.fromKueryExpression(
          'alert.attributes.actions:{ actionTypeId: ".server-log" AND actionRef: "foo" }'
        ),
        types: ['alert'],
        indexMapping: mockMappings,
        hasNestedKey: true,
      });

      // nodes will have errors in the array
      expect(validationObject).toEqual(...);
    });

@lukasolson (and @elastic/kibana-app-arch in general) I (well, the team in general) kinda lack KQL knowledge to be able to fix this. From what I see, the current validation is rather simple, and totally lack nested functions logic (it only handle the nested type functions), and I fear going 'further' may be quite complicated. Would you mind taking a look to see if there is an obvious way to improve/fix our validation here?

[nchaulet]

@pgayvallet I think the workaround fleet-agents.attributes.inputs.status: "error" AND fleet-agents.attributes.inputs.package: "endpoint" is not going to provide correct results for us

if we have this document `{inputs: [{package: endpoint, status: healthy}, {package: nginx, status: error}]`` it's going to be returned by your workaround but a nested query will not return it.

[pgayvallet]

You are right. Did not thought about inputs being an array...

Waiting on @elastic/kibana-app-arch insight then.

[lukasolson]

Hmm, yeah this seems like a bug that falls on our team. What's the priority of fixing this?

[pgayvallet]

I'll let @nchaulet answer that, but I think this is required for a feature they are planning for 7.11

[nchaulet]

Yes we would like to have this for 7.11 it will allow us a better filtering of Fleet agents

[nchaulet]

Hi some news we will not use that feature on fleet for 7.11, so there is no urgence on our side to fix that

[XavierM]

This PR should fix it, #64002

[joshdover]

@XavierM would you mind extracting that fix into a dedicated PR?

[lukasolson]

Looked some more at this today, I don't think this is anything relating to KQL itself, and I believe the fix that @XavierM has in the other PR should do the trick.

[pgayvallet]

@XavierM would you mind extracting the fix you performed in #64002 to an isolated PR?

[rudolf]

Closed by #96292