Update merge on refresh and merge on commit defaults in Opensearch (Lucene 9.3) #3561

reta · 2022-06-10T19:19:03Z

Signed-off-by: Andriy Redko andriy.redko@aiven.io

Description

The Apache Lucene 9.3 made merge-on-refresh enabled by default [1], [2]. The OpenSearch merge on refresh and merge on commit defaults have to be updated in order to accommodate this change.

Additional context, the merge-on-refresh is now fully implemented by MergePolicy. Some merge polices (like TieredMergePolicy and LogMergePolicy) already benefit from this right away, but for others the wrapping merge policy (like
MergeOnFlushMergePolicy) still has (or could) to be applied. This pull request acknowledged the merge-on-refresh as default and also allows to disable it or supply the merge policy to be used.

[1] apache/lucene#921
[2] apache/lucene#935

Issues Resolved

Closes #3553

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-ci-bot · 2022-06-10T19:53:25Z

❌ Gradle Check failure a32befcdc868023d73566a56ce7d4c6077ff6e0d
Log 5898

Reports 5898

opensearch-ci-bot · 2022-06-10T20:42:29Z

❌ Gradle Check failure 387ee14c2e2d0b68752ae72a35ee85247a410390
Log 5899

Reports 5899

opensearch-ci-bot · 2022-06-10T22:30:40Z

❌ Gradle Check failure 32ae3b1908d519e255b83f0c348a8442ff60f38a
Log 5901

Reports 5901

reta · 2022-06-11T01:50:06Z

start gradle check

opensearch-ci-bot · 2022-06-11T03:00:41Z

❌ Gradle Check failure 32ae3b1908d519e255b83f0c348a8442ff60f38a
Log 5904

Reports 5904

reta · 2022-06-11T13:36:27Z

start gradle check

opensearch-ci-bot · 2022-06-11T14:46:20Z

❌ Gradle Check failure 32ae3b1908d519e255b83f0c348a8442ff60f38a
Log 5907

Reports 5907

reta · 2022-06-13T14:55:30Z

start gradle check

opensearch-ci-bot · 2022-06-13T16:03:08Z

❌ Gradle Check failure 32ae3b1908d519e255b83f0c348a8442ff60f38a
Log 5938

Reports 5938

opensearch-ci-bot · 2022-06-13T17:47:11Z

❌ Gradle Check failure d592c66312e86fcaaae27c17b3420fe9c2ab52fc
Log 5939

Reports 5939

opensearch-ci-bot · 2022-06-13T18:15:06Z

❌ Gradle Check failure 941fe47335cfb16d0375b3acb933c702d4d8a7f2
Log 5941

Reports 5941

opensearch-ci-bot · 2022-06-13T18:15:06Z

❌ Gradle Check failure a3a355339ef093356afee7acf39ec1fbe8b1ebd1
Log 5942

Reports 5942

opensearch-ci-bot · 2022-06-13T18:36:50Z

❌ Gradle Check failure 6837d1bb076a4eff4fbba2682b5a4b31a540b1a2
Log 5943

Reports 5943

opensearch-ci-bot · 2022-06-13T20:56:48Z

❌ Gradle Check failure d8cfd01d8b2d4a7b93f5da1eb34520d19b316f93
Log 5945

Reports 5945

opensearch-ci-bot · 2022-06-13T21:30:39Z

✅ Gradle Check success 6c33cb70bd11084133984dcb206009a51284e09b
Log 5947

Reports 5947

opensearch-ci-bot · 2022-06-14T13:24:27Z

✅ Gradle Check success aac0ad7188301f1891caadfa8a44538e5638e673
Log 5978

Reports 5978

server/src/main/java/org/opensearch/index/IndexSettings.java

opensearch-ci-bot · 2022-06-14T17:05:20Z

✅ Gradle Check success 43d03ee329d25bdc90ef815497645e96f13f8234
Log 5986

Reports 5986

nknize

Looking pretty good. Just a general question on whether or not we want to switch away from a boolean setting to using a policy name.

server/src/main/java/org/opensearch/index/IndexSettings.java

nknize · 2022-06-29T16:18:37Z

server/src/test/java/org/opensearch/index/engine/InternalEngineTests.java

@@ -574,7 +573,7 @@ public void testMergeSegmentsOnCommit() throws Exception {
        final Settings.Builder settings = Settings.builder()
            .put(defaultSettings.getSettings())
            .put(IndexSettings.INDEX_MERGE_ON_FLUSH_MAX_FULL_FLUSH_MERGE_WAIT_TIME.getKey(), TimeValue.timeValueMillis(5000))
-            .put(IndexSettings.INDEX_MERGE_ON_FLUSH_ENABLED.getKey(), true);
+            .put(IndexSettings.INDEX_MERGE_ON_FLUSH_POLICY.getKey(), "merge-on-flush");


Do we need to switch away from a boolean here at this point? I'm not sure we expect any merge on flush policy variations other than the default?

@nknize the INDEX_MERGE_ON_FLUSH_POLICY is a new setting (semantics of INDEX_MERGE_ON_FLUSH_ENABLED is kept as boolean). The reason for that is that not every policy may implement merge-on-flush/commit but the composite one could be used for this reasons.

reta · 2022-07-04T16:11:19Z

@nknize may I ask you please to take a second look? Thank you!

github-actions · 2022-07-06T20:19:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ABORTED ❌
URL: https://build.ci.opensearch.org/job/gradle-check/187/
CommitID: 91d67c7547f2c2d4517b9be4091c6d2700c750fc

github-actions · 2022-07-06T20:26:42Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL: null
CommitID: 91d67c7547f2c2d4517b9be4091c6d2700c750fc

github-actions · 2022-07-07T13:38:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/226/
CommitID: 91d67c7547f2c2d4517b9be4091c6d2700c750fc

reta · 2022-07-12T12:50:24Z

@nknize may I ask you please to take a second look? Thank you!

…ucene 9.3) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

reta · 2022-07-25T16:22:17Z

@dblock @dblock @Bukhtawar sorry guys, may I ask for your help here? sadly cannot get back @nknize attention on this change .... thank you!

nknize · 2022-07-25T16:43:00Z

@dblock @dblock @Bukhtawar sorry guys, may I ask for your help here? sadly cannot get back @nknize attention on this change .... thank you!

Hey @reta sorry about that! I had gh status set to away while I was out and it suppressed my notifications so I missed the ping. Will check it here shortly.

github-actions · 2022-07-25T16:50:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/958/
CommitID: aa8e8d4

github-actions · 2022-07-25T17:24:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/960/
CommitID: aa8e8d4

nknize

Thanks for doing this@reta!

reta · 2022-07-25T23:42:39Z

Thanks for doing this@reta!

Thanks a lot for review @nknize, really appreciate it !

opensearch-trigger-bot · 2022-07-26T14:00:13Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-3561-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a34a4c04250b5c4c210ce79d14611953f5267963
# Push it to GitHub
git push --set-upstream origin backport/backport-3561-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-3561-to-2.x.

…ucene 9.3) (opensearch-project#3561) * Update merge on refresh and merge on commit defaults in Opensearch (Lucene 9.3) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Addressing code review comments, adding more tests Signed-off-by: Andriy Redko <andriy.redko@aiven.io> (cherry picked from commit a34a4c0) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

…ucene 9.3) (#3561) (#4013) * Update merge on refresh and merge on commit defaults in Opensearch (Lucene 9.3) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Addressing code review comments, adding more tests Signed-off-by: Andriy Redko <andriy.redko@aiven.io> (cherry picked from commit a34a4c0) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

reta force-pushed the issue-3553 branch from a32befc to 387ee14 Compare June 10, 2022 19:27

reta force-pushed the issue-3553 branch from 387ee14 to 32ae3b1 Compare June 10, 2022 20:46

reta force-pushed the issue-3553 branch from 32ae3b1 to d592c66 Compare June 13, 2022 15:40

reta force-pushed the issue-3553 branch 3 times, most recently from a3a3553 to 6837d1b Compare June 13, 2022 18:12

reta force-pushed the issue-3553 branch from 6837d1b to d8cfd01 Compare June 13, 2022 19:26

reta force-pushed the issue-3553 branch from d8cfd01 to 6c33cb7 Compare June 13, 2022 21:01

reta force-pushed the issue-3553 branch from 6c33cb7 to aac0ad7 Compare June 14, 2022 12:51

Bukhtawar reviewed Jun 14, 2022

View reviewed changes

server/src/main/java/org/opensearch/index/IndexSettings.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/index/IndexSettings.java Outdated Show resolved Hide resolved

reta force-pushed the issue-3553 branch from 43d03ee to e10a21a Compare June 14, 2022 17:08

reta marked this pull request as ready for review June 14, 2022 17:10

reta requested a review from a team as a code owner June 14, 2022 17:10

reta added the backport 2.x Backport to 2.x branch label Jun 27, 2022

nknize requested changes Jun 29, 2022

View reviewed changes

reta force-pushed the issue-3553 branch 2 times, most recently from 1abf3f6 to 913868d Compare June 29, 2022 18:14

Rishikesh1159 mentioned this pull request Jul 5, 2022

[BUG] Test Failure org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded #3579

Closed

reta force-pushed the issue-3553 branch from 913868d to 91d67c7 Compare July 6, 2022 19:50

dreamer-89 mentioned this pull request Jul 11, 2022

Mute testHighWatermarkNotExceeded #3844

Merged

reta added 2 commits July 25, 2022 12:12

Update merge on refresh and merge on commit defaults in Opensearch (L…

517b5ae

…ucene 9.3) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Addressing code review comments, adding more tests

aa8e8d4

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

reta force-pushed the issue-3553 branch from 91d67c7 to aa8e8d4 Compare July 25, 2022 16:20

nknize approved these changes Jul 25, 2022

View reviewed changes

reta merged commit a34a4c0 into opensearch-project:main Jul 26, 2022

reta mentioned this pull request Jul 26, 2022

Fixing flaky org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded test case #4012

Merged

5 tasks

reta mentioned this pull request Jul 26, 2022

[Backport] [2.x] Update merge on refresh and merge on commit defaults in Opensearch (Lucene 9.3) (#3561) #4013

Merged

5 tasks

reta mentioned this pull request Jul 27, 2022

[DOC]Documentation - Enable merge on refresh and merge on commit in Opensearch 2.1.0+ opensearch-project/documentation-website#477

Closed

shwetathareja mentioned this pull request Aug 18, 2023

[Merge] Optimistic merge policy that self corrects its selections if a much better score is available in future or current merge scores are much worse from previous scores #9383

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update merge on refresh and merge on commit defaults in Opensearch (Lucene 9.3) #3561

Update merge on refresh and merge on commit defaults in Opensearch (Lucene 9.3) #3561

reta commented Jun 10, 2022 •

edited

Loading

opensearch-ci-bot commented Jun 10, 2022

opensearch-ci-bot commented Jun 10, 2022

opensearch-ci-bot commented Jun 10, 2022

reta commented Jun 11, 2022

opensearch-ci-bot commented Jun 11, 2022

reta commented Jun 11, 2022

opensearch-ci-bot commented Jun 11, 2022

reta commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

nknize left a comment

nknize Jun 29, 2022

reta Jun 29, 2022

reta commented Jul 4, 2022

github-actions bot commented Jul 6, 2022

github-actions bot commented Jul 6, 2022

github-actions bot commented Jul 7, 2022

reta commented Jul 12, 2022

reta commented Jul 25, 2022

nknize commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

nknize left a comment

reta commented Jul 25, 2022

opensearch-trigger-bot bot commented Jul 26, 2022

Update merge on refresh and merge on commit defaults in Opensearch (Lucene 9.3) #3561

Update merge on refresh and merge on commit defaults in Opensearch (Lucene 9.3) #3561

Conversation

reta commented Jun 10, 2022 • edited Loading

Description

Issues Resolved

Check List

opensearch-ci-bot commented Jun 10, 2022

opensearch-ci-bot commented Jun 10, 2022

opensearch-ci-bot commented Jun 10, 2022

reta commented Jun 11, 2022

opensearch-ci-bot commented Jun 11, 2022

reta commented Jun 11, 2022

opensearch-ci-bot commented Jun 11, 2022

reta commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 13, 2022

opensearch-ci-bot commented Jun 14, 2022

opensearch-ci-bot commented Jun 14, 2022

nknize left a comment

Choose a reason for hiding this comment

nknize Jun 29, 2022

Choose a reason for hiding this comment

reta Jun 29, 2022

Choose a reason for hiding this comment

reta commented Jul 4, 2022

github-actions bot commented Jul 6, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 6, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 7, 2022

Gradle Check (Jenkins) Run Completed with:

reta commented Jul 12, 2022

reta commented Jul 25, 2022

nknize commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 25, 2022

Gradle Check (Jenkins) Run Completed with:

nknize left a comment

Choose a reason for hiding this comment

reta commented Jul 25, 2022

opensearch-trigger-bot bot commented Jul 26, 2022

reta commented Jun 10, 2022 •

edited

Loading