[Remote Store] Add Segment download stats to remotestore stats API #8718

shourya035 · 2023-07-17T09:56:54Z

Re-opening since #8440 was closed accidentally

Description

Adding segment download stats to the _remotestore/stats API
Refactored API output to accommodate both segment and translog stats

The new API output would show a blank JSON node {} if the corresponding values in that node are zero. This has been done intentionally to ensure that we do not show 0 stats to the end user which in turn would avoid confusion. This is important since a newly created replica would not upload anything to the remote store and a newly created primary would not download from the remote store.

The only exception to this behavior would be a primary shard which has been restored from a remote store. In that case, this API would show both segment upload and download stats for the shard. This behavior would be documented as well.

Sample API outputs from running in local dev setup

RemoteStore Stats with all nodes up : https://gist.github.com/shourya035/69d1e217c8b5a854f4b652bb290c1e5e
RemoteStore stats with replica being promoted to primary : https://gist.github.com/shourya035/af3a26202d061b4f147ac84f7b805802
RemoteStore Stats with replica disabled : https://gist.github.com/shourya035/2302f53e8774d28488160f6d365670e0

P.S: RemoteRefreshSegmentTracker was renamed to RemoteRefreshTransferTracker, because of which the number of Files changes has increased from 13 to 17.

Related Issues

Resolves #8395

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions · 2023-07-17T10:32:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

URL: https://build.ci.opensearch.org/job/gradle-check/20276/
CommitID: b60bb40
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-07-17T10:33:18Z

Codecov Report

Merging #8718 (91378d1) into main (b2a7348) will decrease coverage by 0.17%.
The diff coverage is 90.08%.

❗ Current head 91378d1 differs from pull request most recent head 508eb22. Consider uploading reports for the commit 508eb22 to get more accurate results

@@             Coverage Diff              @@
##               main    #8718      +/-   ##
============================================
- Coverage     71.04%   70.88%   -0.17%     
+ Complexity    57246    57071     -175     
============================================
  Files          4764     4757       -7     
  Lines        270153   270016     -137     
  Branches      39532    39490      -42     
============================================
- Hits         191937   191399     -538     
- Misses        62027    62433     +406     
+ Partials      16189    16184       -5

Files Changed	Coverage Δ
...testore/stats/TransportRemoteStoreStatsAction.java	`44.44% <0.00%> (-1.51%)`	⬇️
...search/index/shard/RemoteStoreRefreshListener.java	`84.06% <ø> (-0.55%)`	⬇️
...ex/remote/RemoteRefreshSegmentPressureService.java	`78.75% <63.63%> (ø)`
...in/java/org/opensearch/index/shard/IndexShard.java	`69.66% <80.85%> (+0.62%)`	⬆️
...er/remotestore/stats/RemoteStoreStatsResponse.java	`74.28% <84.61%> (+27.22%)`	⬆️
...in/cluster/remotestore/stats/RemoteStoreStats.java	`94.25% <94.80%> (-0.99%)`	⬇️
...rch/index/remote/RemoteSegmentTransferTracker.java	`91.87% <100.00%> (ø)`

... and 566 files with indirect coverage changes

server/src/main/java/org/opensearch/index/store/DirectoryFileTransferTracker.java

github-actions · 2023-08-01T06:06:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21543/
CommitID: 6ce96a8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-01T06:27:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21542/
CommitID: 6ce96a8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions · 2023-08-01T09:42:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/21550/
CommitID: 508eb22

sachinpkale · 2023-08-01T11:48:33Z

This is a user facing change. We should have a corresponding changlelog entry.

sachinpkale · 2023-08-01T11:50:19Z

Going ahead without changelog entry. Please make sure to add it in the next PR.

opensearch-trigger-bot · 2023-08-01T12:17:01Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8718-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 77074b4a81f88bab44e9e39e0b6eb9323880e740
# Push it to GitHub
git push --set-upstream origin backport/backport-8718-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8718-to-2.x.

ashking94 · 2023-08-01T13:11:34Z

@shourya035 let's raise a manual backport for this.

mch2 · 2023-08-01T23:58:04Z

server/src/main/java/org/opensearch/index/store/Store.java

        private final Logger deletesLogger;

+        public final DirectoryFileTransferTracker directoryFileTransferTracker;


This is now going to apply to every use of the local store.copyFrom. I'm not sure this belongs here? Can we wrap and collect these required stats directly in the remote store directory or a related component?

I wonder if we should think about making this tracking optional by way of an expert setting?

dreamer-89 · 2023-08-02T18:19:02Z

@shourya035 : Can you please create the backport PR to 2.x for this change. This is blocking other PRs which have changes on top of yours.

…pensearch-project#8718) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Shourya <114977491+shourya035@users.noreply.github.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Ashish Singh <ssashish@amazon.com> (cherry picked from commit 77074b4)

…8718) (#9089) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Shourya <114977491+shourya035@users.noreply.github.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Ashish Singh <ssashish@amazon.com> (cherry picked from commit 77074b4)

…pensearch-project#8718) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Shourya <114977491+shourya035@users.noreply.github.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

…pensearch-project#8718) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Shourya <114977491+shourya035@users.noreply.github.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…pensearch-project#8718) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Shourya <114977491+shourya035@users.noreply.github.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

shourya035 added 23 commits July 5, 2023 14:15

Adding Segment download stats to remotestore stats API

d1e688a

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Merge branch 'opensearch-project:main' into segment-download-stats

8086eb9

Removed unused loggers and updated upload stats field names

97b733a

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Fixing spotless errors

e667270

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Adding JavaDocs and fixing divide by zero errors on UTs

743c09a

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Merge branch 'opensearch-project:main' into segment-download-stats

94f0f58

Excluding stats publishing on non-remote store indices

ac3fbc0

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Merge branch 'opensearch-project:main' into segment-download-stats

bb20d64

Excluding stats publishing on non-remote store indices

b42912f

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Merge branch 'opensearch-project:main' into segment-download-stats

83f2bd6

Removing unused logger

80cf62b

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Addressing comments

d826579

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Merge branch 'opensearch-project:main' into segment-download-stats

2213186

Fixing API field name from routing to shards

f282d54

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Merge branch 'opensearch-project:main' into segment-download-stats

331cb81

Changing file upload stats to sync level from file level

8de6d00

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Retrigger integs

9daeb74

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Empty Commit

d507ca8

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Removing unused toString method

8c4d189

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Merge from previous branch

27b6ad0

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Removing unused window size vars

cdf9a49

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Merge branch 'opensearch-project:main' into segment-download-stats-2

c4393b0

Adding Integ tests for download stats

b60bb40

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

shourya035 marked this pull request as ready for review July 17, 2023 10:33

shourya035 requested review from reta, anasalkouz, andrross and Bukhtawar as code owners July 17, 2023 10:33

sachinpkale reviewed Aug 1, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/store/DirectoryFileTransferTracker.java Outdated Show resolved Hide resolved

Merge branch 'opensearch-project:main' into segment-download-stats-2

6ce96a8

Addressing comments

508eb22

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

sachinpkale approved these changes Aug 1, 2023

View reviewed changes

sachinpkale added skip-changelog and removed skip-changelog labels Aug 1, 2023

sachinpkale changed the title ~~[Remote Store] Adding Segment download stats to remotestore stats API~~ [Remote Store] Add Segment download stats to remotestore stats API Aug 1, 2023

sachinpkale merged commit 77074b4 into opensearch-project:main Aug 1, 2023
11 of 12 checks passed

sachinpkale added the backport 2.x Backport to 2.x branch label Aug 1, 2023

shourya035 deleted the segment-download-stats-2 branch August 1, 2023 12:39

mch2 reviewed Aug 1, 2023

View reviewed changes

dreamer-89 mentioned this pull request Aug 2, 2023

[Segment Replication] Refactor RemoteStoreReplicationSource (#8767) #9076

Merged

This was referenced Aug 3, 2023

[Backport-2.x] Add Segment download stats to remotestore stats API #9087

Closed

[Backport-2.x] Add Segment download stats to remotestore stats API #9089

Merged

ashking94 mentioned this pull request Aug 28, 2023

[DOC] Update remote store stats api documentation with latest segments and translog fields opensearch-project/documentation-website#4904

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Add Segment download stats to remotestore stats API #8718

[Remote Store] Add Segment download stats to remotestore stats API #8718

shourya035 commented Jul 17, 2023 •

edited

Loading

github-actions bot commented Jul 17, 2023

codecov bot commented Jul 17, 2023 •

edited

Loading

github-actions bot commented Aug 1, 2023

github-actions bot commented Aug 1, 2023

github-actions bot commented Aug 1, 2023

sachinpkale commented Aug 1, 2023

sachinpkale commented Aug 1, 2023

opensearch-trigger-bot bot commented Aug 1, 2023

ashking94 commented Aug 1, 2023

mch2 Aug 1, 2023 •

edited

Loading

nknize Aug 2, 2023

dreamer-89 commented Aug 2, 2023

		private final Logger deletesLogger;

		public final DirectoryFileTransferTracker directoryFileTransferTracker;

[Remote Store] Add Segment download stats to remotestore stats API #8718

[Remote Store] Add Segment download stats to remotestore stats API #8718

Conversation

shourya035 commented Jul 17, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Jul 17, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Jul 17, 2023 • edited Loading

Codecov Report

github-actions bot commented Aug 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 1, 2023

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Aug 1, 2023

sachinpkale commented Aug 1, 2023

opensearch-trigger-bot bot commented Aug 1, 2023

ashking94 commented Aug 1, 2023

mch2 Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

nknize Aug 2, 2023

Choose a reason for hiding this comment

dreamer-89 commented Aug 2, 2023

shourya035 commented Jul 17, 2023 •

edited

Loading

codecov bot commented Jul 17, 2023 •

edited

Loading

mch2 Aug 1, 2023 •

edited

Loading