[Search Pipelines] Support ad hoc pipelines

[msfroh]

Description

This change allows a search pipeline to be defined within a search request body. This will take precedence over any other search pipeline, and allows ad hoc testing of pipeline configurations before persisting the pipeline definition in cluster state.

The new search source field is gated behind the search pipelines feature flag. Also, serialization of the field is gated behind a 3.0 version check. I don't think we should backport to the 2.x branch before we release 2.7. I think we'll want to update that version check to 2.8.

Issues Resolved

#6717

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14348/
• CommitID: 5d05e5c
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14350/
• CommitID: bbd7c90
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/14416/
• CommitID: 9469e77
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[msfroh]

@andrross, @reta -- you two have context on Search Pipelines. Would either of you be able to take a look at this?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/14911/
• CommitID: c0bdd3d

[codecov-commenter]

Codecov Report

Merging #7253 (24be027) into main (2646d40) will decrease coverage by 0.04%.
The diff coverage is 80.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #7253      +/-   ##
============================================
- Coverage     70.60%   70.56%   -0.04%     
+ Complexity    59582    59489      -93     
============================================
  Files          4877     4878       +1     
  Lines        285806   285844      +38     
  Branches      41165    41175      +10     
============================================
- Hits         201787   201713      -74     
- Misses        67446    67460      +14     
- Partials      16573    16671      +98     

Impacted Files	Coverage Δ
...pensearch/search/RandomSearchRequestGenerator.java	94.87% <0.00%> (-0.99%)	⬇️
...opensearch/search/builder/SearchSourceBuilder.java	68.02% <60.00%> (-1.16%)	⬇️
...nsearch/search/pipeline/SearchPipelineService.java	83.15% <80.95%> (+0.54%)	⬆️
...pensearch/action/search/TransportSearchAction.java	71.21% <100.00%> (+1.58%)	⬆️
.../java/org/opensearch/search/pipeline/Pipeline.java	84.61% <100.00%> (+3.13%)	⬆️
...g/opensearch/search/pipeline/PipelinedRequest.java	100.00% <100.00%> (ø)

... and 504 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/14925/
• CommitID: 24be027

[reta]

@andrross mind to take a look? thank you! ;-)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/14970/
• CommitID: fbd5321

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7253-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c43d7138636023315605f8a4caf91cf407c8753a
# Push it to GitHub
git push --set-upstream origin backport/backport-7253-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7253-to-2.x.