Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add "Using search latency monitoring at the coordinator node level" #3261

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Add "Using search latency monitoring at the coordinator node level" #3261

wants to merge 11 commits into from

Conversation

Naarcha-AWS
Copy link
Contributor

Description

Adds the Using search latency monitoring at the coordinator node level, written by @dzane17.

Since this is David's first blog, we'll still need to add his community member page to this PR. Will also need to add bio pages for Ankit.

Issues Resolved

Closes #3143

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

@Naarcha-AWS
Using search latency monitoring at the coordinator node level
1adfa56 
Signed-off-by: Archer <naarcha@amazon.com>
@Naarcha-AWS
Add David community page
fb9b0fb 
Signed-off-by: Archer <naarcha@amazon.com>
@Naarcha-AWS
Copy link
Contributor Author

Added David but still need to add bios for Ankit and Kunal. Working on gathering that information now.

@jainankitk
Copy link

Thanks @dzane17 & @Naarcha-AWS for putting this together. Looks great, just some minor comments!

@Naarcha-AWS
Copy link
Contributor Author

Thanks for the review @jainankitk. I'll get these updated now.

@Naarcha-AWS
Add additional technical feedback.
52f8538 
Signed-off-by: Archer <naarcha@amazon.com>
@Naarcha-AWS
Add Ankit as author. Additional feedback
8d3476c 
Signed-off-by: Archer <naarcha@amazon.com>
@Naarcha-AWS
Add Ankit as an author
178991c 
Signed-off-by: Archer <naarcha@amazon.com>
@Naarcha-AWS
Add Kunal. Update dates.
5a4589a 
Signed-off-by: Archer <naarcha@amazon.com>
jainankitk
jainankitk reviewed Sep 11, 2024
View reviewed changes
_posts/2024-09-12-using-search-latency-monitoring.markdown Outdated

In OpenSearch, the coordinator node plays a crucial role in search execution by acting as an intermediary between the client and the cluster. When a search request is received, the coordinator node fans the request to data nodes containing the relevant shards. Depending on the type of search, the request undergoes a series of search phases, or types of search operations, initiated by the coordinator node. These phases include the following:

* `can_match`: Pre-filtesr search shards based on query rewrites.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: filters

dzane17
dzane17 reviewed Sep 11, 2024
View reviewed changes
_community_members/davizane.md Outdated
- title: Members
url: /community/members/index.html
- title: 'David Zaneo&apos;s Profile'
url: '/community/members/davizane.html'
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Naarcha-AWS can we change this url to david-zane.html like others

_community_members/davizane.md Outdated
job_title_and_company: 'AWS Software Engineer'
personas:
- author
permalink: '/community/members/davizane.html'
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

_posts/2024-09-12-using-search-latency-monitoring.markdown Outdated

However, the ability to monitor search latency in OpenSearch versions earlier than 2.11 is limited. Tools such as the [Nodes Stats API](https://opensearch.org/docs/latest/api-reference/nodes-apis/nodes-stats/) and [shard slow logs](https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/logs/#shard-slow-logs) offer latency measurements based on shard-level operations but do not provide any visibility at the coordinator node level. Because search requests often hit multiple shards simultaneously, the measured amount of time spent in a single shard search phase doesn't provide comprehensive latency information.

Luckily, in OpenSearch 2.11 and later, OpenSearch offers search latency monitoring at the coordinator node level. This blog post provides information about several tools---the Node Stats API, the `phase_took` parameter, and slow logs---that can help you monitor coordinator-node-level statistics.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use full name "search request slow logs" so not confused with "shard slow logs"

_posts/2024-09-12-using-search-latency-monitoring.markdown Outdated

## The coordinator node and search phases

In OpenSearch, the coordinator node plays a crucial role in search execution by acting as an intermediary between the client and the cluster. When a search request is received, the coordinator node fans the request to data nodes containing the relevant shards. Depending on the type of search, the request undergoes a series of search phases, or types of search operations, initiated by the coordinator node. These phases include the following:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intermediary between the client and the cluster

Coordinator is still part of the cluster. "cluster" should be changed to something like "data nodes"

_posts/2024-09-09-using-search-latency-monitoring.markdown Outdated Show resolved Hide resolved
pajuric
pajuric reviewed Sep 11, 2024
View reviewed changes
_posts/2024-09-12-using-search-latency-monitoring.markdown Outdated
- search
excerpt:
meta_keywords:
meta_description:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the following meta:

meta_keywords: search latency, OpenSearch cluster, latency monitoring, coordinator node, search performance optimization

meta_description: Explore how enhanced search latency monitoring capabilities at the coordinator node level can help you optimize search performance in your OpenSearch cluster.

Naarcha-AWS and others added 2 commits September 12, 2024 12:25
@Naarcha-AWS
Apply suggestions from code review
ddd52f2 
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
@Naarcha-AWS
Add Sagar
9452904 
Signed-off-by: Archer <naarcha@amazon.com>
dzane17
dzane17 reviewed Sep 12, 2024
View reviewed changes
_community_members/upasagar.md Outdated
Comment on lines 16 to 17
github: uselmh
job_title_and_company: 'Social media specialist for the OpenSearch Project'
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sgup432 wow social media star. Are these correct?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dzane17 Do you have any doubt? 😁

dzane17
dzane17 reviewed Sep 12, 2024
View reviewed changes
_posts/2024-09-12-using-search-latency-monitoring.markdown
- davizane
- naarcha
- akjain
- kkhatua
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add upasagar here

@Naarcha-AWS
Add Sagar's name
ab6e027 
Signed-off-by: Archer <naarcha@amazon.com>
@Naarcha-AWS
Fix job title
3b657e8 
Signed-off-by: Archer <naarcha@amazon.com>
@Naarcha-AWS
Apply suggestions from code review
403e938 
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jainankitk jainankitk jainankitk left review comments

@dzane17 dzane17 dzane17 approved these changes

@sgup432 sgup432 sgup432 approved these changes

@pajuric pajuric pajuric left review comments

@elfisher elfisher Awaiting requested review from elfisher elfisher is a code owner

@AMoo-Miki AMoo-Miki Awaiting requested review from AMoo-Miki AMoo-Miki is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@krisfreedain krisfreedain Awaiting requested review from krisfreedain krisfreedain is a code owner

@peterzhuamazon peterzhuamazon Awaiting requested review from peterzhuamazon peterzhuamazon is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dtaivpp dtaivpp Awaiting requested review from dtaivpp dtaivpp is a code owner

@kolchfa-aws kolchfa-aws Awaiting requested review from kolchfa-aws kolchfa-aws is a code owner

@nateynateynate nateynateynate Awaiting requested review from nateynateynate nateynateynate is a code owner

@natebower natebower Awaiting requested review from natebower natebower is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Blog] New Search Latency Features in OpenSearch
5 participants
@Naarcha-AWS @jainankitk @dzane17 @sgup432 @pajuric