[Remote Store] Add support to provide separate segment metadata repository #12993

sachinpkale · 2024-04-01T05:38:22Z

Description

Currently, when we configure remote store for a cluster, both data and metadata (of segment, translog and cluster state) is stored in the same store.
But the access pattern and consistency requirements for data and metadata are different.
For example, data files are uploaded and downloaded by providing a filename whereas metadata is fetched based on certain characteristics such as recency.
Due to these differences, it is possible that metadata can be stored in a different data store (for example, a key-value store) as we store the data files in a blob store.
In this PR, we provide support to provide a separate repository for segment metadata. This will be followed by separation of translog metadata repository.

Related Issues

[Feature Request] [Remote Store] Make metadata store independent of data store #12919

Check List

New functionality includes testing.
- All tests pass
~~New functionality has been documented.~~
- ~~New functionality has javadoc added~~
~~Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)~~
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
~~Public documentation issue/PR created~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-04-01T05:59:11Z

Compatibility status:

Checks if related components are compatible with change 95d18c7

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git]

github-actions · 2024-04-01T06:29:13Z

❌ Gradle check result for 0dea8c1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-01T07:30:34Z

❌ Gradle check result for 940ce2e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-01T07:40:12Z

❌ Gradle check result for eb0918f: ABORTED

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-03T05:24:11Z

❌ Gradle check result for 87ce493: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-03T07:56:36Z

❌ Gradle check result for a844c8a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-04T12:34:51Z

❕ Gradle check result for d260b1c: UNSTABLE

TEST FAILURES:

      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2024-04-04T12:36:56Z

✅ Gradle check result for 2852ca4: SUCCESS

codecov · 2024-04-04T12:38:16Z

Codecov Report

Attention: Patch coverage is 73.91304% with 18 lines in your changes missing coverage. Please review.

Project coverage is 71.44%. Comparing base (b15cb0c) to head (2852ca4).
Report is 373 commits behind head on main.

❗ Current head 2852ca4 differs from pull request most recent head 95d18c7

Please upload reports for the commit 95d18c7 to get more accurate results.

Files	Patch %	Lines
...rch/node/remotestore/RemoteStoreNodeAttribute.java	43.75%	7 Missing and 2 partials ⚠️
...org/opensearch/cluster/metadata/IndexMetadata.java	76.92%	2 Missing and 1 partial ⚠️
...h/cluster/metadata/MetadataCreateIndexService.java	77.77%	1 Missing and 1 partial ⚠️
...c/main/java/org/opensearch/index/IndexService.java	0.00%	2 Missing ⚠️
...ndex/store/RemoteSegmentStoreDirectoryFactory.java	95.00%	0 Missing and 1 partial ⚠️
...in/java/org/opensearch/indices/IndicesService.java	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12993      +/-   ##
============================================
+ Coverage     71.42%   71.44%   +0.02%     
- Complexity    59978    60419     +441     
============================================
  Files          4985     5026      +41     
  Lines        282275   284475    +2200     
  Branches      40946    41202     +256     
============================================
+ Hits         201603   203237    +1634     
- Misses        63999    64389     +390     
- Partials      16673    16849     +176

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Bukhtawar

Overall LGTM, with increased repositories whats the impact to node attributes and join validations?

server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java

server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectoryFactory.java

Signed-off-by: Sachin Kale <kalsac@amazon.com>

github-actions · 2024-04-12T05:25:09Z

❌ Gradle check result for 1391737: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sachin Kale <kalsac@amazon.com>

github-actions · 2024-04-12T06:03:52Z

❌ Gradle check result for 95d18c7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

linuxpi · 2024-04-16T08:07:14Z

server/src/main/java/org/opensearch/snapshots/RestoreService.java

-            SETTING_REMOTE_SEGMENT_STORE_REPOSITORY,
-            SETTING_REMOTE_TRANSLOG_STORE_REPOSITORY
+            SETTING_REMOTE_SEGMENT_STORE_DATA_REPOSITORY,
+            SETTING_REMOTE_TRANSLOG_STORE_DATA_REPOSITORY


do we need to add the metadata repo as well to unmodified settings during snapshot restore?

+1. We will need corresponding ITs as well to restore .

linuxpi · 2024-04-16T08:12:25Z

server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java

@@ -2702,7 +2702,7 @@ public void snapshotRemoteStoreIndexShard(
                        indexTotalNumberOfFiles,
                        indexTotalFileSize,
                        store.indexSettings().getUUID(),
-                        store.indexSettings().getRemoteStoreRepository(),
+                        store.indexSettings().getRemoteSegmentStoreDataRepository(),


i think we need to add another parameter for metadata repository as well since lock files are in metadata repo. currently the remoteStoreRepository is used for both source data and lock files.

https://github.com/linuxpi/OpenSearch/blob/08e8325f03cd7f0c205c74bd9892677fb3cd4b59/server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java#L669-L675

Right! Thanks for catching this, let me make the required change.

Lock files are stored separately stored and not in metadata repo .

Lock file path is generated with segment repository when data and metadata is the same repo.
Once we separate data and metadata, we need to associate lock files with one of these repos. As per the comment: #12993 (comment), it makes more sense to have lock files stored in metadata repo.

gbbafna · 2024-04-22T12:14:46Z

server/src/main/java/org/opensearch/snapshots/RestoreService.java

We will need to add sourceRemoteStoreMetadataRepository as well to RestoreSnapshotRequest .

harishbhakuni · 2024-05-08T18:20:54Z

server/src/main/java/org/opensearch/index/shard/StoreRecovery.java

@@ -407,6 +407,7 @@ void recoverFromSnapshotAndRemoteStore(
                    threadPool
                );
                RemoteSegmentStoreDirectory sourceRemoteDirectory = (RemoteSegmentStoreDirectory) directoryFactory.newDirectory(
+                    remoteStoreRepository,


so, right now we pull this remote store repository value from shard level snapshot metadata, we need to make related changes there to update that with metadata directory now.

Also, i don't like the idea of using same repository name here for metadata and data, we should maybe create a remote directory instance from directoryFactory which have access to metadata repository only.

checked more on this, actually we will need segments_N file for restoring snapshot, which is in data directory, so we will need to add both segment data and metadata repository info in snapshot metadata. unless we keep segments_N file in metadata directory as well.

harishbhakuni · 2024-05-08T18:25:13Z

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

@@ -903,6 +903,7 @@ public static void remoteDirectoryCleanup(
    ) {
        try {
            RemoteSegmentStoreDirectory remoteSegmentStoreDirectory = (RemoteSegmentStoreDirectory) remoteDirectoryFactory.newDirectory(
+                remoteStoreRepoForIndex,


here as well, as mentioned in above comment, we can create the same remoteDirectory instance which have access only to metadata repository.

harishbhakuni · 2024-05-08T18:31:56Z

server/src/main/java/org/opensearch/index/shard/StoreRecovery.java

@@ -407,6 +407,7 @@ void recoverFromSnapshotAndRemoteStore(
                    threadPool
                );


Along with the doc changes of this feature, we need to update doc for source_remote_store_repository which will be now used to override remote segment metadata directory: https://opensearch.org/docs/latest/api-reference/snapshots/restore-snapshot/

opensearch-trigger-bot · 2024-06-08T15:21:01Z

This PR is stalled because it has been open for 30 days with no activity.

sachinpkale force-pushed the metadata-repo branch from eb0918f to 0dea8c1 Compare April 1, 2024 05:53

sachinpkale force-pushed the metadata-repo branch from 0dea8c1 to 940ce2e Compare April 1, 2024 06:52

sachinpkale force-pushed the metadata-repo branch from 940ce2e to 87ce493 Compare April 3, 2024 05:16

sachinpkale force-pushed the metadata-repo branch from a844c8a to 506d6b4 Compare April 3, 2024 12:30

sachinpkale marked this pull request as ready for review April 3, 2024 12:35

sachinpkale requested review from abbashus, adnapibar, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, ryanbogan and saratvemulapalli as code owners April 3, 2024 12:35

Bukhtawar reviewed Apr 6, 2024

View reviewed changes

ashking94 reviewed Apr 7, 2024

View reviewed changes

server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java Show resolved Hide resolved

ashking94 reviewed Apr 7, 2024

View reviewed changes

server/src/main/java/org/opensearch/cluster/metadata/IndexMetadata.java Show resolved Hide resolved

linuxpi reviewed Apr 8, 2024

View reviewed changes

server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java Show resolved Hide resolved

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectoryFactory.java Outdated Show resolved Hide resolved

This was referenced Apr 9, 2024

[Feature Request] Add support to separate segment infos snapshot from remote segment metadata #13129

Open

[Remote Store] Add support to separate segment infos snapshot from metadata #13114

Open

Sachin Kale added 7 commits April 12, 2024 10:48

Add support to provide separate segment metadata repository

f262f39

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Add support to integ tests to pass separate metadata repo

9a0822a

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Add changelog entry

ac165de

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Fix compile error

44573c6

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Fix null metadata repo

06b1222

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Add entry of the metadata repository setting into built in settings

7d16206

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Fix unit test

1391737

Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale force-pushed the metadata-repo branch from 2852ca4 to 1391737 Compare April 12, 2024 05:20

Address PR comments

95d18c7

Signed-off-by: Sachin Kale <kalsac@amazon.com>

linuxpi reviewed Apr 16, 2024

View reviewed changes

itiyama approved these changes Apr 19, 2024

View reviewed changes

gbbafna reviewed Apr 22, 2024

View reviewed changes

gbbafna requested review from harishbhakuni and removed request for adnapibar and abbashus April 22, 2024 12:16

harishbhakuni suggested changes May 8, 2024

View reviewed changes

opensearch-trigger-bot bot added the stalled Issues that have stalled label Jun 8, 2024

gbbafna mentioned this pull request Aug 8, 2024

Add Varun Bansal as maintainer #15163

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Add support to provide separate segment metadata repository #12993

[Remote Store] Add support to provide separate segment metadata repository #12993

sachinpkale commented Apr 1, 2024 •

edited

Loading

github-actions bot commented Apr 1, 2024 •

edited

Loading

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

github-actions bot commented Apr 4, 2024

github-actions bot commented Apr 4, 2024

codecov bot commented Apr 4, 2024 •

edited

Loading

Bukhtawar left a comment

github-actions bot commented Apr 12, 2024

github-actions bot commented Apr 12, 2024

linuxpi Apr 16, 2024

gbbafna Apr 22, 2024

linuxpi Apr 16, 2024

sachinpkale Apr 22, 2024

gbbafna Apr 22, 2024

sachinpkale Apr 22, 2024

gbbafna Apr 22, 2024

harishbhakuni May 8, 2024

harishbhakuni May 8, 2024

harishbhakuni May 8, 2024

harishbhakuni May 8, 2024

harishbhakuni May 8, 2024

opensearch-trigger-bot bot commented Jun 8, 2024

		@@ -407,6 +407,7 @@ void recoverFromSnapshotAndRemoteStore(
		threadPool
		);

[Remote Store] Add support to provide separate segment metadata repository #12993

Are you sure you want to change the base?

[Remote Store] Add support to provide separate segment metadata repository #12993

Conversation

sachinpkale commented Apr 1, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Apr 1, 2024 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

github-actions bot commented Apr 4, 2024

github-actions bot commented Apr 4, 2024

codecov bot commented Apr 4, 2024 • edited Loading

Codecov Report

Bukhtawar left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 12, 2024

github-actions bot commented Apr 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Jun 8, 2024

sachinpkale commented Apr 1, 2024 •

edited

Loading

github-actions bot commented Apr 1, 2024 •

edited

Loading

codecov bot commented Apr 4, 2024 •

edited

Loading