Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proxy: Only generate debug messages in debug mode #6228

Merged
merged 3 commits into from
Mar 23, 2023

Conversation

jacobbaungard
Copy link
Contributor

@jacobbaungard jacobbaungard commented Mar 20, 2023

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

Verification

saswatamcode
saswatamcode previously approved these changes Mar 21, 2023
Copy link
Member

@saswatamcode saswatamcode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Some small suggestions, but looks good otherwise. Would be great to see a comparable profile!

@@ -229,7 +229,7 @@ func registerQuery(app *extkingpin.App) {
var storeRateLimits store.SeriesSelectLimits
storeRateLimits.RegisterFlags(cmd)

cmd.Setup(func(g *run.Group, logger log.Logger, reg *prometheus.Registry, tracer opentracing.Tracer, _ <-chan struct{}, _ bool) error {
cmd.Setup(func(g *run.Group, logger log.Logger, reg *prometheus.Registry, tracer opentracing.Tracer, _ <-chan struct{}, debug bool) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep it consistent with other components that use it.

Suggested change
cmd.Setup(func(g *run.Group, logger log.Logger, reg *prometheus.Registry, tracer opentracing.Tracer, _ <-chan struct{}, debug bool) error {
cmd.Setup(func(g *run.Group, logger log.Logger, reg *prometheus.Registry, tracer opentracing.Tracer, _ <-chan struct{}, debugLogging bool) error {

pkg/store/proxy.go Show resolved Hide resolved
@@ -273,7 +276,9 @@ func (s *ProxyStore) Series(originalRequest *storepb.SeriesRequest, srv storepb.
for _, st := range s.stores() {
// We might be able to skip the store if its meta information indicates it cannot have series matching our query.
if ok, reason := storeMatches(srv.Context(), st, originalRequest.MinTime, originalRequest.MaxTime, matchers...); !ok {
storeDebugMsgs = append(storeDebugMsgs, fmt.Sprintf("store %s filtered out: %v", st, reason))
if s.debug {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it would be a good idea to do the same for ProxyStore LabelNames and LabelValues methods as well, as they also print debug statements the same way?

@@ -68,6 +68,7 @@ type Client interface {
// ProxyStore implements the store API that proxies request to all given underlying stores.
type ProxyStore struct {
logger log.Logger
debug bool
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can iterate on this but would be nice to have a builder pattern, with option methods like WithProxyDebugLogging similar to BucketStore. That way we can avoid having to change signature everytime. 🙂

@jacobbaungard
Copy link
Contributor Author

Tried the address all the comments.

The e2e test failures seems to be due to Prometheus' busybox image has been updated - looks like there will be an automatic PR updating that soon.

In debug mode we log information about which endpoints we skip and
query, including the labels of each endpoint. These debug messages, can
use significant memory, especially in larger setups with larger number
of labelsets.

Previously we would generate the debug messages, even when not in debug
mode. With commit, we only generate the debug messages when debug
logging is enabled.

This fixes: thanos-io#6198

Signed-off-by: Jacob Baungard Hansen <jachanse@redhat.com>
- Consistent variable naming
- Also do conditional debug logging generation in: `LabelNames()` and
  `LabelValues`
- Use builder pattern for additional variables

Signed-off-by: Jacob Baungard Hansen <jachanse@redhat.com>
Signed-off-by: Jacob Baungard Hansen <jachanse@redhat.com>
Copy link
Member

@saswatamcode saswatamcode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Contributor

@fpetkovski fpetkovski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking care of this, LGTM

@fpetkovski fpetkovski merged commit 62ceb82 into thanos-io:main Mar 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Proxy: memory bloat caused by fmt.Sprintf
3 participants