Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*: Using native histograms for grpc middleware metrics #7393

Merged
merged 3 commits into from
May 28, 2024
Merged

*: Using native histograms for grpc middleware metrics #7393

merged 3 commits into from
May 28, 2024

Conversation

pedro-stanaka
Copy link
Contributor

@pedro-stanaka pedro-stanaka commented May 28, 2024 •
edited
Loading

Since we updated the middleware library, we can now use native histograms to keep track of latencies in grpc calls.
This is a semi-breaking change if people enabled native histogram collection on their Prometheus monitoring Thanos instances.

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

  • Update grpc middleware library to latest version
  • Using native histograms for grpc middleware metrics (both server and client side).

Verification

image

@pedro-stanaka pedro-stanaka force-pushed the feat/native-histo-middleware branch 2 times, most recently from 87f64bd to 45b7b84 Compare May 28, 2024 07:15
@pedro-stanaka pedro-stanaka marked this pull request as ready for review May 28, 2024 07:15
@pedro-stanaka
*: Using native histograms for grpc middleware metrics
2e0c120 
Since we updated the middleware library, we can now use native histograms to keep track of latencies in grpc calls.
This is a semi-breaking change if people enabled native histogram collection on their Prometheus monitoring Thanos instances.

Signed-off-by: Pedro Tanaka <pedro.stanaka@gmail.com>

adding change log

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
fpetkovski
fpetkovski previously approved these changes May 28, 2024
View reviewed changes
@fpetkovski
Copy link
Contributor

Looks like the docs failure is legit.

@pedro-stanaka
removing empty space;
bd17754 
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
@MichaHoffmann
Copy link
Contributor

Can we still scrape classic histograms?

@pedro-stanaka
Copy link
Contributor Author

pedro-stanaka commented May 28, 2024 •
edited
Loading

Can we still scrape classic histograms?

Yes, if someone either did not enable native histogram feature flag or has scrape_classic_histograms: true for the scrape job, the classic histogram will still be scraped.

My local running instance scrape endpoint:

grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="0.001"} 1
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="0.01"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="0.1"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="0.3"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="0.6"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="1"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="3"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="6"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="9"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="20"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="30"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="60"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="90"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="120"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="+Inf"} 2

MichaHoffmann
MichaHoffmann previously approved these changes May 28, 2024
View reviewed changes
Copy link
Contributor

@MichaHoffmann MichaHoffmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@MichaHoffmann
Copy link
Contributor

If this breaking for some people then we should add the breaking emoji to the changelog probably

fpetkovski
fpetkovski previously approved these changes May 28, 2024
View reviewed changes
Copy link
Contributor

@fpetkovski fpetkovski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native histograms are opt-in, so there should be no breaking changes.

@fpetkovski fpetkovski enabled auto-merge May 28, 2024 08:42
@pedro-stanaka
Put full disclaimer in changelog
77afbda 
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
auto-merge was automatically disabled May 28, 2024 08:43

Head branch was pushed to by a user without write access

@MichaHoffmann
Copy link
Contributor

Native histograms are opt-in, so there should be no breaking changes.

Was just noting because the PR description mentioned that its semi-breaking

MichaHoffmann
MichaHoffmann approved these changes May 28, 2024
View reviewed changes
Copy link
Contributor

@MichaHoffmann MichaHoffmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@MichaHoffmann MichaHoffmann enabled auto-merge (squash) May 28, 2024 08:50
@MichaHoffmann MichaHoffmann merged commit dfa7dd5 into thanos-io:main May 28, 2024
36 checks passed
@BrewTestBot BrewTestBot mentioned this pull request Jul 31, 2024
Merged
hczhu-db pushed a commit to databricks/thanos that referenced this pull request Aug 22, 2024
@pedro-stanaka @hczhu-db
*: Using native histograms for grpc middleware metrics (thanos-io#7393)
b2c9883 
* *: Using native histograms for grpc middleware metrics

Since we updated the middleware library, we can now use native histograms to keep track of latencies in grpc calls.
This is a semi-breaking change if people enabled native histogram collection on their Prometheus monitoring Thanos instances.

Signed-off-by: Pedro Tanaka <pedro.stanaka@gmail.com>

adding change log

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* removing empty space;

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Put full disclaimer in changelog

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

---------

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
hczhu-db pushed a commit to databricks/thanos that referenced this pull request Aug 22, 2024
@pedro-stanaka @hczhu-db
*: Using native histograms for grpc middleware metrics (thanos-io#7393)
5a3bc00 
* *: Using native histograms for grpc middleware metrics

Since we updated the middleware library, we can now use native histograms to keep track of latencies in grpc calls.
This is a semi-breaking change if people enabled native histogram collection on their Prometheus monitoring Thanos instances.

Signed-off-by: Pedro Tanaka <pedro.stanaka@gmail.com>

adding change log

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* removing empty space;

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Put full disclaimer in changelog

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

---------

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MichaHoffmann MichaHoffmann MichaHoffmann approved these changes

@fpetkovski fpetkovski fpetkovski left review comments

Assignees
No one assigned
Labels
size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pedro-stanaka @fpetkovski @MichaHoffmann