[Feature Request] Conditional Search Processor

[austintlee]

Is your feature request related to a problem? Please describe

I would like to be able to define DAGs on search processors and have a processor run if a certain condition (an output or a result of another processor) is met.

Describe the solution you'd like

In terms of the language of describing a DAG, we should leverage what's already there in OpenSearch. I think there was some recent work in the AI Workflows project for conditional workflows? That and the stateful search process will probably give us most of what we need. We should also borrow from the Ingest pipeline as much as we can.

Related component

Search

Describe alternatives you've considered

No response

Additional context

No response

[msfroh]

This is consistent with the original proposal around search pipelines. Down at the bottom of that issue, there's:

Just as ingest pipelines support conditional execution of processors and nested pipelines, we could add similar capabilities to search processors to effectively turn the pipeline into a directed acyclic graph. If that becomes the norm, we would likely want a visualization tool to view and edit a search pipeline (since nested JSON would be hard for a human to understand).

As it happens, that kind of visualization may come via the Flow framework: #9213

[dbwiddis]

@austintlee we're starting to ideate on how to do that now, please comment on opensearch-project/flow-framework#367

[msfroh]

This is a duplicate of #6715 and #6714.

I'm going to suggest that we close out those issues, since we should probably deliver conditional processing in one shot (rather than breaking down request vs response).

[macohen]

@msfroh, in the search relevance weekly meeting, suggested that Ingest processors may have a good example that could be referenced for this; can also be used as a DAG potentially. The Flow project may also have a good connection to this issue: opensearch-project/flow-framework#367 and #9213