Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter out invalid URI and HTTP method in the error message of no handler found for a REST request #3459

Merged

Conversation

tlfeng
Copy link
Collaborator

@tlfeng tlfeng commented May 27, 2022

Description

Filter out invalid URI and HTTP method of a error message, which shown when there is no handler found for a REST request sent by user, so that HTML special characters <>&"' will not shown in the error message.

The error message is return as mine-type application/json, which can't contain active (script) content, so it's not a vulnerability. Besides, no browsers are going to render as html when the mine-type is that.
While the common security scanners will raise a false-positive alarm for having HTML tags in the response without escaping the HTML special characters, so the solution only aims to satisfy the code security scanners.

If the URI is valid, the error message remains.
If the URI is not valid, the change is:
old error message: {"error":"no handler found for uri [/<script>cross_site_scripting.nasl</script>.asp] and method [GET]"}
new error message: {"error":"invalid uri has been requested"}

If the HTTP method is not defined in the enum: https://github.com/opensearch-project/OpenSearch/blob/2.0.0/server/src/main/java/org/opensearch/rest/RestRequest.java#L236, the change of error message is:
old error message: {"error":"Unexpected http method: <script>alert(\"!\")</script>","status":405}
new error message: {"error":"Unexpected http method","status":405}

Reference: #3459 (comment)

Issues Resolved

Resolve #3453

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@tlfeng tlfeng added enhancement Enhancement or improvement to existing feature or request Severity-Minor v3.0.0 Issues and PRs related to version 3.0.0 labels May 27, 2022
@tlfeng tlfeng force-pushed the sanitize-html-http-bad-request-handler branch from ec2012e to 17b7947 Compare May 27, 2022 06:59
@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure ec2012e127c0514e9022fbc09dccc98645b700a7
Log 5623

Reports 5623

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure 17b7947628a8cdbb54d298c1a7827c4c7dfc453d
Log 5624

Reports 5624

@tlfeng tlfeng force-pushed the sanitize-html-http-bad-request-handler branch from 17b7947 to 5e1934b Compare May 27, 2022 07:35
@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 5e1934be34eb90390e5ef82f33256b4b59ec3d8c
Log 5625

Reports 5625

@tlfeng tlfeng force-pushed the sanitize-html-http-bad-request-handler branch from 5e1934b to 39d0951 Compare May 27, 2022 21:10
@tlfeng tlfeng added v2.1.0 Issues and PRs related to version 2.1.0 backport 2.x Backport to 2.x branch labels May 27, 2022
@tlfeng tlfeng marked this pull request as ready for review May 27, 2022 21:11
@tlfeng tlfeng requested review from a team and reta as code owners May 27, 2022 21:11
…nd for a REST request

Signed-off-by: Tianli Feng <ftianli@amazon.com>
@tlfeng tlfeng force-pushed the sanitize-html-http-bad-request-handler branch from 39d0951 to ef4e190 Compare May 27, 2022 21:16
StringBuilder out = new StringBuilder();
for (int i = 0; i < text.length(); i++) {
char c = text.charAt(i);
if (c == '"' || c == '\'' || c == '<' || c == '>' || c == '&') {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OWasp also recommends '/'

I think that will conflict with the URI/route path though.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your review! 👍
Ah, I didn't realized that the slash "/" is also a character that recommended to deal with. 😂 But your thought is correct that it will make unreadable for the URI in the error message.
Frankly speaking, I even think my current code to do the HTML escape is over-engineering. URLEncode (which will escape the <, >, and & characters to %3C, %3E, and %26, respectively) is likely enough to satisfy the requirement, and there is native Java API to do that, but it's just not a professional solution for XSS problem (https://security.stackexchange.com/questions/8867/urlencode-vs-htmlencode-for-xss-prevention).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tlfeng I am a bit concerned with rolling out own sanitization schemes, may be we should use OWASP HTML sanitizer instead [1]?

[1] https://github.com/OWASP/java-html-sanitizer/blob/master/docs/getting_started.md

Copy link
Collaborator Author

@tlfeng tlfeng May 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @reta, thanks for the opinion!
My reason for not using the additional library for sanitizing HTML is:
The purpose of the code change is to satisfy the security scanners so that they won't complain, instead of resolving actual vulnerability, so I don't think it worth to add a new Java dependency to the code base. Using additional library to the code will enlarge the file size for the build artifact, and adding maintenance burden for upgrading the library.

So I would rather using a simpler solution, to use a Java native API URLEncoder, URLEncoder.encode(String s, String enc), which achieves the goal in the issue, but it's not a quite correct way to solve XSS problem.

That's my personal thought, so may not reasonable. What do you think of it 😄

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @tlfeng, I think URLEncoder.encode won't fit here: {"error":"no handler found for uri [/a/b/c] and method [DELETE]"} would become something like {"error":"no handler found for uri [%2Fa%2Fb%2Fc] and method [DELETE]"}, not very readable.

What we are dealing with is actually the fact that any "garbage" could be sent to Opensearch, for example {"error":"no handler found for uri [/a/b/c<script>alert(\"!\")</script>] and method [DELETE]"} - this is not even valid URI. May be what we should do instead, is to distinguish those two issue:

  • uri is not valid: return {"error":"invalid uri has been requested"}
  • uri is valid but not found, in this case it is sufficient to include uri.getPath() only: return {"error":"no handler found for uri [${uri.getPath()}] and method [DELETE]"}

The funny part is that HTTP method is also vulnerable to XSS attacks:

 curl -ik -X '<script>alert("!")</script>' 'https://localhost:9200/a/b/c'

{"error":"Unexpected http method: <script>alert(\"!\")</script>","status":405}

We should probably follow the same approach, the list of HTTP methods is more or less stable, we could reply to the "garbage" something like: {"error":"Unsupported http method","status":405}

Copy link
Collaborator Author

@tlfeng tlfeng Jun 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@reta Thanks for your comprehensive solution! And you are so careful to find the HTTP method will be shown in the error message as well. 😂
I think it's really reasonable. Instead of encoding HTML special characters, not showing the invalid user input sounds like a simple and effective solution. 😄 I will update the code based on it.

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 39d0951777db309459b64de12ac05b2d80eadfa0
Log 5639

Reports 5639

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success ef4e190
Log 5640

Reports 5640

… REST request

Signed-off-by: Tianli Feng <ftianli@amazon.com>
@tlfeng tlfeng changed the title Escape HTML special characters in the error message of no handler found for a REST request Filter out invalid URI in the error message of no handler found for a REST request Jun 1, 2022
// Validate input URI to filter out HTML special characters in the error message,
// in case false-positive cross site scripting vulnerability is detected by common security scanners.
uri = new URI(uri).getPath();
builder.field("error", "no handler found for uri [" + uri + "] and method [" + method + "]");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, do you want to sanitize method as well?

Copy link
Collaborator Author

@tlfeng tlfeng Jun 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your review is so quick!
Sure, I can sanitize the error message for the HTTP method, although it may not necessary, since there is no code security scanners have an alarm on that. 😄
I will update the code then.

Note:
After taking a look at the code, the error message of invalid HTTP method is directly obtained from the
https://github.com/opensearch-project/OpenSearch/blob/2.0.0/server/src/main/java/org/opensearch/rest/RestController.java#L450
https://github.com/opensearch-project/OpenSearch/blob/2.0.0/modules/transport-netty4/src/main/java/org/opensearch/http/netty4/Netty4HttpRequest.java#L148

Copy link
Collaborator Author

@tlfeng tlfeng Jun 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in the new commit 863e272 . 👍

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 34f1b8f
Log 5706

Reports 5706

@tlfeng tlfeng changed the title Filter out invalid URI in the error message of no handler found for a REST request Filter out invalid URI and HTTP method in the error message of no handler found for a REST request Jun 1, 2022
…nd for a REST request

Signed-off-by: Tianli Feng <ftianli@amazon.com>
@tlfeng tlfeng force-pushed the sanitize-html-http-bad-request-handler branch from cb681d5 to 863e272 Compare June 1, 2022 23:13
@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure cb681d542046b48c1c4f7f9306c5ae5763c4c98f
Log 5708

Reports 5708

msg.append(exception.getMessage());
// Not using the error message directly from 'exception.getMessage()' to avoid unescaped HTML special characters,
// in case false-positive cross site scripting vulnerability is detected by common security scanners.
msg.append("Unexpected http method");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super nit-pick (sorry): Unexpected HTTP method (just to make it concise with previous error Incorrect HTTP method phrasing.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haha, that definitely makes the messages neat.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in the commit c586b5b

Signed-off-by: Tianli Feng <ftianli@amazon.com>
@@ -623,6 +632,7 @@ public Exception getInboundException() {
assertTrue(channel.getSendResponseCalled());
assertThat(channel.getRestResponse().getHeaders().containsKey("Allow"), equalTo(true));
assertThat(channel.getRestResponse().getHeaders().get("Allow"), hasItem(equalTo(RestRequest.Method.GET.toString())));
assertThat(channel.getRestResponse().content().utf8ToString(), containsString("Unexpected http method"));
Copy link
Collaborator

@reta reta Jun 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this case case throws an exception when getHttpMethod() is requested, could you please add a test case with HTTP method like "<script>alert('xss');alert(\"&#x6A&#x61&#x76&#x61\");</script>" to make sure it won't sneak in? Thank you.

Copy link
Collaborator Author

@tlfeng tlfeng Jun 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this test case shows the result of any unsupported HTTP method, since the handler for HTTP method will always throw an exception https://github.com/opensearch-project/OpenSearch/blob/2.0.0/server/src/test/java/org/opensearch/rest/RestControllerTests.java#L560.
I will add a separate test for it, to use the real scenario.

Copy link
Collaborator Author

@tlfeng tlfeng Jun 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@reta Sorry I find it very hard to simulate the real process after receiving an specific wrong HTTP method in the unit test.
The HTTP method that used to build RestRequest comes from interface HttpRequest, and it has to be an item of the enum. I couldn't find an existing class that implements interface HttpRequest can accept arbitrary HTTP method, and can be used in the test class.
In real scenario, using any HTTP method other than those listed in the enum will get an exception, and the current code directly throws an exception, so I think the current unit test is likely enough.
To make the assertion more effective, I changed to use equalTo instead of containString to validate the error message, in the new commit 3a7a484

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, sorry about that @tlfeng , thank you!

Copy link
Collaborator Author

@tlfeng tlfeng Jun 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never mind at all 😁. Thanks for your opinion.

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 863e272
Log 5709

Reports 5709

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure c586b5b
Log 5711

Reports 5711

Signed-off-by: Tianli Feng <ftianli@amazon.com>
@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 3a7a484
Log 5715

Reports 5715

@tlfeng tlfeng merged commit 2bfe8b3 into opensearch-project:main Jun 2, 2022
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jun 2, 2022
…dler found for a REST request (#3459)

Filter out invalid URI and HTTP method of a error message, which shown when there is no handler found for a REST request sent by user, so that HTML special characters <>&"' will not shown in the error message.

The error message is return as mine-type `application/json`, which can't contain active (script) content, so it's not a vulnerability. Besides, no browsers are going to render as html when the mine-type is that.
While the common security scanners will raise a false-positive alarm for having HTML tags in the response without escaping the HTML special characters, so the solution only aims to satisfy the code security scanners.

Signed-off-by: Tianli Feng <ftianli@amazon.com>
(cherry picked from commit 2bfe8b3)
@tlfeng tlfeng deleted the sanitize-html-http-bad-request-handler branch June 2, 2022 16:30
reta pushed a commit that referenced this pull request Jun 2, 2022
…dler found for a REST request (#3459) (#3485)

Filter out invalid URI and HTTP method of a error message, which shown when there is no handler found for a REST request sent by user, so that HTML special characters <>&"' will not shown in the error message.

The error message is return as mine-type `application/json`, which can't contain active (script) content, so it's not a vulnerability. Besides, no browsers are going to render as html when the mine-type is that.
While the common security scanners will raise a false-positive alarm for having HTML tags in the response without escaping the HTML special characters, so the solution only aims to satisfy the code security scanners.

Signed-off-by: Tianli Feng <ftianli@amazon.com>
(cherry picked from commit 2bfe8b3)

Co-authored-by: Tianli Feng <ftianli@amazon.com>
Bukhtawar pushed a commit that referenced this pull request Jun 20, 2022
…dler found for a REST request (#3459)

Filter out invalid URI and HTTP method of a error message, which shown when there is no handler found for a REST request sent by user, so that HTML special characters <>&"' will not shown in the error message.

The error message is return as mine-type `application/json`, which can't contain active (script) content, so it's not a vulnerability. Besides, no browsers are going to render as html when the mine-type is that.
While the common security scanners will raise a false-positive alarm for having HTML tags in the response without escaping the HTML special characters, so the solution only aims to satisfy the code security scanners.

Signed-off-by: Tianli Feng <ftianli@amazon.com>
Bukhtawar pushed a commit to Bukhtawar/OpenSearch that referenced this pull request Jun 21, 2022
…dler found for a REST request (opensearch-project#3459)

Filter out invalid URI and HTTP method of a error message, which shown when there is no handler found for a REST request sent by user, so that HTML special characters <>&"' will not shown in the error message.

The error message is return as mine-type `application/json`, which can't contain active (script) content, so it's not a vulnerability. Besides, no browsers are going to render as html when the mine-type is that.
While the common security scanners will raise a false-positive alarm for having HTML tags in the response without escaping the HTML special characters, so the solution only aims to satisfy the code security scanners.

Signed-off-by: Tianli Feng <ftianli@amazon.com>
Bukhtawar added a commit that referenced this pull request Jun 27, 2022
* Bump reactor-netty-core from 1.0.16 to 1.0.19 in /plugins/repository-azure (#3360)

* Bump reactor-netty-core in /plugins/repository-azure

Bumps [reactor-netty-core](https://github.com/reactor/reactor-netty) from 1.0.16 to 1.0.19.
- [Release notes](https://github.com/reactor/reactor-netty/releases)
- [Commits](reactor/reactor-netty@v1.0.16...v1.0.19)

---
updated-dependencies:
- dependency-name: io.projectreactor.netty:reactor-netty-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* [Type removal] _type removal from mocked responses of scroll hit tests (#3377)

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* [Type removal] Remove _type deprecation from script and conditional processor (#3239)

* [Type removal] Remove _type deprecation from script and conditional processor

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless check apply

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* [Type removal] Remove _type from _bulk yaml test, scripts, unused constants (#3372)

* [Type removal] Remove redundant _type deprecation checks in bulk request

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* [Type removal] bulk yaml tests validating deprecation on _type and removal from scripts

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix Lucene-snapshots repo for jdk 17. (#3396)

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Replace internal usages of 'master' term in 'server/src/internalClusterTest' directory (#2521)

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* [REMOVE] Cleanup deprecated thread pool types (FIXED_AUTO_QUEUE_SIZE) (#3369)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* [Type removal] _type removal from tests of yaml tests (#3406)

* [Type removal] _type removal from tests of yaml tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix spotless failures

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix assertion failures

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix assertion failures in DoSectionTests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add release notes for version 2.0.0 (#3410)


Signed-off-by: Rabi Panda <adnapibar@gmail.com>

* [Upgrade] Lucene-9.2.0-snapshot-ba8c3a8 (#3416)

Upgrades to latest snapshot of lucene 9.2.0 in preparation for GA release.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Fix release notes for 2.0.0-rc1 version (#3418)

This change removes some old commits from the 2.0.0-rc1 release notes. These commits were already released as part of 1.x releases.

Add back some missing type removal commits to the 2.0.0 release notes

Signed-off-by: Rabi Panda <adnapibar@gmail.com>

* Bump version 2.1 to Lucene 9.2 after upgrade (#3424)

Bumps Version.V_2_1_0 lucene version to 9.2 after backporting upgrage.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Bump com.gradle.enterprise from 3.10 to 3.10.1 (#3425)

Bumps com.gradle.enterprise from 3.10 to 3.10.1.

---
updated-dependencies:
- dependency-name: com.gradle.enterprise
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump reactor-core from 3.4.17 to 3.4.18 in /plugins/repository-azure (#3427)

Bumps [reactor-core](https://github.com/reactor/reactor-core) from 3.4.17 to 3.4.18.
- [Release notes](https://github.com/reactor/reactor-core/releases)
- [Commits](reactor/reactor-core@v3.4.17...v3.4.18)

---
updated-dependencies:
- dependency-name: io.projectreactor:reactor-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Bump gax-httpjson from 0.101.0 to 0.103.1 in /plugins/repository-gcs (#3426)

Bumps [gax-httpjson](https://github.com/googleapis/gax-java) from 0.101.0 to 0.103.1.
- [Release notes](https://github.com/googleapis/gax-java/releases)
- [Changelog](https://github.com/googleapis/gax-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/gax-java/commits)

---
updated-dependencies:
- dependency-name: com.google.api:gax-httpjson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* [segment replication]Introducing common Replication interfaces for segment replication and recovery code paths (#3234)

* RecoveryState inherits from ReplicationState + RecoveryTarget inherits from ReplicationTarget

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* Refactoring: mixedClusterVersion error fix + move Stage to ReplicationState

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* pull ReplicationListener into a top level class + add javadocs + address review comments

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* fix javadoc

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* review changes

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* Refactoring the hierarchy relationship between repl and recovery

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* style fix

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* move package common under replication

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* rename to replication

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* rename and doc changes

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* [Type removal] Remove type from BulkRequestParser (#3423)

* [Type removal] Remove type handling in bulk request parser

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* [Type removal] Remove testTypesStillParsedForBulkMonitoring as it is no longer present in codebase

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Adding CheckpointRefreshListener to trigger when Segment replication is turned on and Primary shard refreshes (#3108)

* Intial PR adding classes and tests related to checkpoint publishing

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Putting a Draft PR with all changes in classes. Testing is still not included in this commit.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Wiring up index shard to new engine, spotless apply and removing unnecessary tests and logs

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Adding Unit test for checkpointRefreshListener

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Applying spotless check

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Fixing import statements *

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* removing unused constructor in index shard

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Addressing comments from last commit

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Adding package-info.java files for two new packages

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Adding test for null checkpoint publisher and addreesing PR comments

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Add docs for indexshardtests and remove shard.refresh

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Add a new Engine implementation for replicas with segment replication enabled. (#3240)

* Change fastForwardProcessedSeqNo method in LocalCheckpointTracker to persisted checkpoint.

This change inverts fastForwardProcessedSeqNo to fastForwardPersistedSeqNo for use in
Segment Replication.  This is so that a Segrep Engine can match the logic of InternalEngine
where the seqNo is incremented with each operation, but only persisted in the tracker on a flush.
With Segment Replication we bump the processed number with each operation received index/delete/noOp, and
invoke this method when we receive a new set of segments to bump the persisted seqNo.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Extract Translog specific engine methods into an abstract class.

This change extracts translog specific methods to an abstract engine class so that other engine
implementations can reuse translog logic.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add a separate Engine implementation for replicas with segment replication enabled.

This change adds a new engine intended to be used on replicas with segment replication enabled.
This engine does not wire up an IndexWriter, but still writes all operations to a translog.
The engine uses a new ReaderManager that refreshes from an externally provided SegmentInfos.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix spotless checks.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix :server:compileInternalClusterTestJava compilation.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix failing test naming convention check.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* PR feedback.

- Removed isReadOnlyReplica from overloaded constructor and added feature flag checks.
- Updated log msg in NRTReplicationReaderManager
- cleaned up store ref counting in NRTReplicationEngine.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix spotless check.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove TranslogAwareEngine and build translog in NRTReplicationEngine.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix formatting

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add missing translog methods to NRTEngine.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove persistent seqNo check from fastForwardProcessedSeqNo.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* PR feedback.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add test specific to translog trimming.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Javadoc check.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add failEngine calls to translog methods in NRTReplicationEngine.
Roll xlog generation on replica when a new commit point is received.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Rename master to cluster_manager in the XContent Parser of ClusterHealthResponse (#3432)

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Bump hadoop-minicluster in /test/fixtures/hdfs-fixture (#3359)

Bumps hadoop-minicluster from 3.3.2 to 3.3.3.

---
updated-dependencies:
- dependency-name: org.apache.hadoop:hadoop-minicluster
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump avro from 1.10.2 to 1.11.0 in /plugins/repository-hdfs (#3358)

* Bump avro from 1.10.2 to 1.11.0 in /plugins/repository-hdfs

Bumps avro from 1.10.2 to 1.11.0.

---
updated-dependencies:
- dependency-name: org.apache.avro:avro
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Fix testSetAdditionalRolesCanAddDeprecatedMasterRole() by removing the initial assertion (#3441)

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Replace internal usages of 'master' term in 'server/src/test' directory (#2520)

* Replace the non-inclusive terminology "master" with "cluster manager" in code comments, internal variable/method/class names, in `server/src/test` directory.
* Backwards compatibility is not impacted.
* Add a new unit test `testDeprecatedMasterNodeFilter()` to validate using `master:true` or `master:false` can filter the node in [Cluster Stats](https://opensearch.org/docs/latest/opensearch/rest-api/cluster-stats/) API, after the `master` role is deprecated in PR #2424

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Removing unused method from TransportSearchAction (#3437)

* Removing unused method from TransportSearchAction

Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>

* Set term vector flags to false for ._index_prefix field (#1901). (#3119)

* Set term vector flags to false for ._index_prefix field (#1901).

Signed-off-by: Vesa Pehkonen <vesa.pehkonen@intel.com>

* Replaced the FieldType copy ctor with ctor for the prefix field and replaced
setting the field type parameters with setIndexOptions(). (#1901)

Signed-off-by: Vesa Pehkonen <vesa.pehkonen@intel.com>

* Added tests for term vectors. (#1901)

Signed-off-by: Vesa Pehkonen <vesa.pehkonen@intel.com>

* Fixed code formatting error.

Signed-off-by: Vesa Pehkonen <vesa.pehkonen@intel.com>

Co-authored-by: sdp <sdp@9049fa06826d.jf.intel.com>

* [BUG] Fixing org.opensearch.monitor.os.OsProbeTests > testLogWarnCpuMessageOnlyOnes when cgroups are available but cgroup stats is not (#3448)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* [Segment Replication] Add SegmentReplicationTargetService to orchestrate replication events. (#3439)

* Add SegmentReplicationTargetService to orchestrate replication events.

This change introduces  boilerplate classes for Segment Replication and a target service
to orchestrate replication events.

It also includes two refactors of peer recovery components for reuse.
1. Rename RecoveryFileChunkRequest to FileChunkRequest and extract code to handle throttling into
ReplicationTarget.
2. Extracts a component to execute retryable requests over the transport layer.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Code cleanup.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Make SegmentReplicationTargetService component final so that it can not
be extended by plugins.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Bump azure-core-http-netty from 1.11.9 to 1.12.0 in /plugins/repository-azure (#3474)

Bumps [azure-core-http-netty](https://github.com/Azure/azure-sdk-for-java) from 1.11.9 to 1.12.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-java/releases)
- [Commits](Azure/azure-sdk-for-java@azure-core-http-netty_1.11.9...azure-core_1.12.0)

---
updated-dependencies:
- dependency-name: com.azure:azure-core-http-netty
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update to Apache Lucene 9.2 (#3477)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Bump protobuf-java from 3.20.1 to 3.21.1 in /plugins/repository-hdfs (#3472)

Signed-off-by: dependabot[bot] <support@github.com>

* [Upgrade] Lucene-9.3.0-snapshot-823df23 (#3478)

Upgrades to latest snapshot of lucene 9.3.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Filter out invalid URI and HTTP method in the error message of no handler found for a REST request (#3459)

Filter out invalid URI and HTTP method of a error message, which shown when there is no handler found for a REST request sent by user, so that HTML special characters <>&"' will not shown in the error message.

The error message is return as mine-type `application/json`, which can't contain active (script) content, so it's not a vulnerability. Besides, no browsers are going to render as html when the mine-type is that.
While the common security scanners will raise a false-positive alarm for having HTML tags in the response without escaping the HTML special characters, so the solution only aims to satisfy the code security scanners.

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Support use of IRSA for repository-s3 plugin credentials (#3475)

* Support use of IRSA for repository-s3 plugin credentials

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Address code review comments

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Address code review comments

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Bump google-auth-library-oauth2-http from 0.20.0 to 1.7.0 in /plugins/repository-gcs (#3473)

* Bump google-auth-library-oauth2-http in /plugins/repository-gcs

Bumps google-auth-library-oauth2-http from 0.20.0 to 1.7.0.

---
updated-dependencies:
- dependency-name: com.google.auth:google-auth-library-oauth2-http
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Use variable to define the version of dependency google-auth-library-java

Signed-off-by: Tianli Feng <ftianli@amazon.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tianli Feng <ftianli@amazon.com>

* [Segment Replication] Added source-side classes for orchestrating replication events (#3470)

This change expands on the existing SegmentReplicationSource interface and its corresponding Factory class by introducing an implementation where the replication source is a primary shard (PrimaryShardReplicationSource). These code paths execute on the target. The primary shard implementation creates the requests to be send to the source/primary shard.

Correspondingly, this change also defines two request classes for the GET_CHECKPOINT_INFO and GET_SEGMENT_FILES requests as well as an abstract superclass.

A CopyState class has been introduced that captures point-in-time, file-level details from an IndexShard. This implementation mirrors Lucene's NRT CopyState implementation.

Finally, a service class has been introduce for segment replication that runs on the source side (SegmentReplicationSourceService) which handles these two types of incoming requests. This includes private handler classes that house the logic to respond to these requests, with some functionality stubbed for now. The service class also uses a simple map to cache CopyState objects that would be needed by replication targets.

Unit tests have been added/updated for all new functionality.

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* [Dependency upgrade] google-oauth-client to 1.33.3 (#3500)

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* move bash flag to set statement (#3494)

Passing bash with flags to the first argument of /usr/bin/env requires
its own flag to interpret it correctly.  Rather than use `env -S` to
split the argument, have the script `set -e` to enable the same behavior
explicitly in preinst and postinst scripts.

Also set `-o pipefail` for consistency.

Closes: #3492

Signed-off-by: Cole White <cwhite@wikimedia.org>

* Support use of IRSA for repository-s3 plugin credentials: added YAML Rest test case (#3499)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Bump azure-storage-common from 12.15.0 to 12.16.0 in /plugins/repository-azure (#3517)

* Bump azure-storage-common in /plugins/repository-azure

Bumps [azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.15.0 to 12.16.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-java/releases)
- [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.15.0...azure-storage-blob_12.16.0)

---
updated-dependencies:
- dependency-name: com.azure:azure-storage-common
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Bump google-oauth-client from 1.33.3 to 1.34.0 in /plugins/discovery-gce (#3516)

* Bump google-oauth-client from 1.33.3 to 1.34.0 in /plugins/discovery-gce

Bumps [google-oauth-client](https://github.com/googleapis/google-oauth-java-client) from 1.33.3 to 1.34.0.
- [Release notes](https://github.com/googleapis/google-oauth-java-client/releases)
- [Changelog](https://github.com/googleapis/google-oauth-java-client/blob/main/CHANGELOG.md)
- [Commits](googleapis/google-oauth-java-client@v1.33.3...v1.34.0)

---
updated-dependencies:
- dependency-name: com.google.oauth-client:google-oauth-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Fix the support of RestClient Node Sniffer for version 2.x and update tests (#3487)

Fix the support of RestClient Node Sniffer for OpenSearch 2.x, and update unit tests for OpenSearch.
The current code contains the logic to be compatible with Elasticsearch 2.x version, which is conflict with OpenSearch 2.x, so removed that part of legacy code.

* Update the script create_test_nodes_info.bash to dump the response of Nodes Info API GET _nodes/http for OpenSearch 1.0 and 2.0 version, which used for unit test.
* Remove the support of Elasticsearch version 2.x for the Sniffer
* Update unit test to validate the Sniffer compatible with OpenSearch 1.x and 2.x
* Update the API response parser to meet the array notation (in ES 6.1 and above) for the node attributes setting. It will result the value of `node.attr` setting will not be parsed as array in the Sniffer, when using the Sniffer on cluster in Elasticsearch 6.0 and above.
* Replace "master" node role with "cluster_manager" in unit test

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Bump com.diffplug.spotless from 6.6.1 to 6.7.0 (#3513)

Bumps com.diffplug.spotless from 6.6.1 to 6.7.0.

---
updated-dependencies:
- dependency-name: com.diffplug.spotless
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump guava from 18.0 to 23.0 in /plugins/ingest-attachment (#3357)

* Bump guava from 18.0 to 23.0 in /plugins/ingest-attachment

Bumps [guava](https://github.com/google/guava) from 18.0 to 23.0.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](google/guava@v18.0...v23.0)

---
updated-dependencies:
- dependency-name: com.google.guava:guava
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Add more ingorance of using internal java API sun.misc.Unsafe

Signed-off-by: Tianli Feng <ftianli@amazon.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tianli Feng <ftianli@amazon.com>

* Added bwc version 2.0.1 (#3452)

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com>

* Add release notes for 1.3.3 (#3549)

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

* [Upgrade] Lucene-9.3.0-snapshot-b7231bb (#3537)

Upgrades to latest snapshot of lucene 9.3; including reducing maxFullFlushMergeWaitMillis 
in LuceneTest.testWrapLiveDocsNotExposeAbortedDocuments to 0 ms to ensure aborted 
docs are not merged away in the test with the new mergeOnRefresh default policy.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* [Remote Store] Upload segments to remote store post refresh (#3460)

* Add RemoteDirectory interface to copy segment files to/from remote store

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Co-authored-by: Sachin Kale <kalsac@amazon.com>

* Add index level setting for remote store

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Co-authored-by: Sachin Kale <kalsac@amazon.com>

* Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener

Co-authored-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>

* Upload segment to remote store post refresh

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Co-authored-by: Sachin Kale <kalsac@amazon.com>

* Fixing VerifyVersionConstantsIT test failure (#3574)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Bump jettison from 1.4.1 to 1.5.0 in /plugins/discovery-azure-classic (#3571)

* Bump jettison from 1.4.1 to 1.5.0 in /plugins/discovery-azure-classic

Bumps [jettison](https://github.com/jettison-json/jettison) from 1.4.1 to 1.5.0.
- [Release notes](https://github.com/jettison-json/jettison/releases)
- [Commits](jettison-json/jettison@jettison-1.4.1...jettison-1.5.0)

---
updated-dependencies:
- dependency-name: org.codehaus.jettison:jettison
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Bump google-api-services-storage from v1-rev20200814-1.30.10 to v1-rev20220608-1.32.1 in /plugins/repository-gcs (#3573)

* Bump google-api-services-storage in /plugins/repository-gcs

Bumps google-api-services-storage from v1-rev20200814-1.30.10 to v1-rev20220608-1.32.1.

---
updated-dependencies:
- dependency-name: com.google.apis:google-api-services-storage
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Upgrade Google HTTP Client to 1.42.0

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xue Zhou <xuezhou@amazon.com>

* Add flat_skew setting to node overload decider (#3563)

* Add flat_skew setting to node overload decider

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

* Bump xmlbeans from 5.0.3 to 5.1.0 in /plugins/ingest-attachment (#3572)

* Bump xmlbeans from 5.0.3 to 5.1.0 in /plugins/ingest-attachment

Bumps xmlbeans from 5.0.3 to 5.1.0.

---
updated-dependencies:
- dependency-name: org.apache.xmlbeans:xmlbeans
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Bump google-oauth-client from 1.34.0 to 1.34.1 in /plugins/discovery-gce (#3570)

* Bump google-oauth-client from 1.34.0 to 1.34.1 in /plugins/discovery-gce

Bumps [google-oauth-client](https://github.com/googleapis/google-oauth-java-client) from 1.34.0 to 1.34.1.
- [Release notes](https://github.com/googleapis/google-oauth-java-client/releases)
- [Changelog](https://github.com/googleapis/google-oauth-java-client/blob/main/CHANGELOG.md)
- [Commits](googleapis/google-oauth-java-client@v1.34.0...v1.34.1)

---
updated-dependencies:
- dependency-name: com.google.oauth-client:google-oauth-client
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Fix for bug showing incorrect awareness attributes count in AwarenessAllocationDecider (#3428)

* Fix for bug showing incorrect awareness attributes count in AwarenessAllocationDecider

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Added bwc version 1.3.4 (#3552)

Signed-off-by: GitHub <noreply@github.com>

Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com>

* Support dynamic node role (#3436)

* Support unknown node role

Currently OpenSearch only supports several built-in nodes like data node
role. If specify unknown node role, OpenSearch node will fail to start.
This limit how to extend OpenSearch to support some extension function.
For example, user may prefer to run ML tasks on some dedicated node
which doesn't serve as any built-in node roles. So the ML tasks won't
impact OpenSearch core function. This PR removed the limitation and user
can specify any node role and OpenSearch will start node correctly with
that unknown role. This opens the door for plugin developer to run
specific tasks on dedicated nodes.

Issue: #2877

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* fix cat nodes rest API spec

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* fix mixed cluster IT failure

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* add DynamicRole

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* change generator method name

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* fix failed docker test

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* transform role name to lower case to avoid confusion

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* transform the node role abbreviation to lower case

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* fix checkstyle

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* add test for case-insensitive role name change

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* Rename package 'o.o.action.support.master' to 'o.o.action.support.clustermanager' (#3556)

* Rename package org.opensearch.action.support.master to org.opensearch.action.support.clustermanager

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Rename classes with master term in the package org.opensearch.action.support.master

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Deprecate classes in org.opensearch.action.support.master

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Remove pakcage o.o.action.support.master

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Move package-info back

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Move package-info to new folder

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Correct the package-info

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Fixing flakiness of ShuffleForcedMergePolicyTests (#3591)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Deprecate classes in org.opensearch.action.support.master (#3593)

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Add release notes for version 2.0.1 (#3595)

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

* Fix NPE when minBound/maxBound is not set before being called. (#3605)

Signed-off-by: George Apaaboah <george.apaaboah@gmail.com>

* Added bwc version 2.0.2 (#3613)

Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com>

* Fix false positive query timeouts due to using cached time (#3454)

* Fix false positive query timeouts due to using cached time

Signed-off-by: Ahmad AbuKhalil <abukhali@amazon.com>

* delegate nanoTime call to SearchContext

Signed-off-by: Ahmad AbuKhalil <abukhali@amazon.com>

* add override to SearchContext getRelativeTimeInMillis to force non cached time

Signed-off-by: Ahmad AbuKhalil <abukhali@amazon.com>

* Fix random gradle check failure issue 3584. (#3627)

* [Segment Replication] Add components for segment replication to perform file copy. (#3525)

* Add components for segment replication to perform file copy.

This change adds the required components to SegmentReplicationSourceService to initiate copy and react to lifecycle events.
Along with new components it refactors common file copy code from RecoverySourceHandler into reusable pieces.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Deprecate public methods and variables with master term in package 'org.opensearch.action.support.master' (#3617)

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Add replication orchestration for a single shard (#3533)

* implement segment replication target

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* test added

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* changes to tests + finalizeReplication

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* fix style check

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* addressing comments + fix gradle check

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* added test + addressed review comments

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* [BUG] opensearch crashes on closed client connection before search reply (#3626)

* [BUG] opensearch crashes on closed client connection before search reply

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Addressing code review comments

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Add all deprecated method in the package with new name 'org.opensearch.action.support.clustermanager' (#3644)

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Introduce TranslogManager implementations decoupled from the Engine (#3638)

* Introduce decoupled translog manager interfaces

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Adding onNewCheckpoint to Start Replication on Replica Shard when Segment Replication is turned on (#3540)

* Adding onNewCheckpoint and it's test to start replication. SCheck for latestcheckpoint and replaying logic is removed from this commit and will be added in a different PR

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Changing binding/inject logic and addressing comments from PR

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Applying spotless check

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Moving shouldProcessCheckpoint() to IndexShard, and removing some trace logs

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* applying spotlessApply

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Adding more info to log statement in targetservice class

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* applying spotlessApply

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Addressing comments on PR

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Adding teardown() in SegmentReplicationTargetServiceTests.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* fixing testShouldProcessCheckpoint() in SegmentReplicationTargetServiceTests

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Removing CheckpointPublisherProvider in IndicesModule

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* spotless check apply

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Remove class org.opensearch.action.support.master.AcknowledgedResponse (#3662)

* Remove class org.opensearch.action.support.master.AcknowledgedResponse

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Remove class org.opensearch.action.support.master.AcknowledgedRequest RequestBuilder ShardsAcknowledgedResponse

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Restore AcknowledgedResponse and AcknowledgedRequest to package org.opensearch.action.support.master (#3669)

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* [BUG] Custom POM configuration for ZIP publication produces duplicit tags (url, scm) (#3656)

* [BUG] Custom POM configuration for ZIP publication produces duplicit tags (url, scm)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Added test case for pluginZip with POM

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Support both Gradle 6.8.x and Gradle 7.4.x

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Adding 2.2.0 Bwc version to main (#3673)

* Upgraded to t-digest 3.3. (#3634)

* Revert renaming method onMaster() and offMaster() in interface LocalNodeMasterListener (#3686)

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Upgrading AWS SDK dependency for native plugins (#3694)

* Merge branch 'feature/point_in_time' of https://github.com/opensearch-project/OpenSearch into fb

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Suraj Singh <surajrider@gmail.com>
Co-authored-by: Marc Handalian <handalm@amazon.com>
Co-authored-by: Tianli Feng <ftianli@amazon.com>
Co-authored-by: Andriy Redko <andriy.redko@aiven.io>
Co-authored-by: Rabi Panda <adnapibar@gmail.com>
Co-authored-by: Nick Knize <nknize@apache.org>
Co-authored-by: Poojita Raj <poojiraj@amazon.com>
Co-authored-by: Rishikesh Pasham <62345295+Rishikesh1159@users.noreply.github.com>
Co-authored-by: Ankit Jain <jain.ankitk@gmail.com>
Co-authored-by: vpehkone <101240162+vpehkone@users.noreply.github.com>
Co-authored-by: sdp <sdp@9049fa06826d.jf.intel.com>
Co-authored-by: Kartik Ganesh <gkart@amazon.com>
Co-authored-by: Cole White <42356806+shdubsh@users.noreply.github.com>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com>
Co-authored-by: Xue Zhou <85715413+xuezhou25@users.noreply.github.com>
Co-authored-by: Sachin Kale <sachinpkale@gmail.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Xue Zhou <xuezhou@amazon.com>
Co-authored-by: Rishab Nahata <rishabnahata07@gmail.com>
Co-authored-by: Anshu Agarwal <anshuagarwal11@gmail.com>
Co-authored-by: Yaliang Wu <ylwu@amazon.com>
Co-authored-by: Kunal Kotwani <kkotwani@amazon.com>
Co-authored-by: George Apaaboah <35894485+GeorgeAp@users.noreply.github.com>
Co-authored-by: Ahmad AbuKhalil <105249973+aabukhalil@users.noreply.github.com>
Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>
Co-authored-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
Co-authored-by: Daniel (dB.) Doubrovkine <dblock@dblock.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch enhancement Enhancement or improvement to existing feature or request Severity-Minor v2.1.0 Issues and PRs related to version 2.1.0 v3.0.0 Issues and PRs related to version 3.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sanitize HTML in HTTP error messages
4 participants