Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flush metrics on exit #749

Merged
merged 6 commits into from
May 30, 2020
Merged

Conversation

aabmass
Copy link
Member

@aabmass aabmass commented May 28, 2020

For #664, in PushController before exit, flush the meter by calling tick().

Metrics are batched in the meter and flushed each tick() of the PushController. The fix is to additionally call tick() before exiting.

In PushController before exit, flush the meter by calling tick()
@aabmass aabmass changed the title [WIP] Flush metrics on exit Flush metrics on exit May 28, 2020
@aabmass aabmass marked this pull request as ready for review May 28, 2020 23:46
@aabmass aabmass requested a review from a team May 28, 2020 23:46
@aabmass
Copy link
Member Author

aabmass commented May 29, 2020

@lzchen I have one concern about this flushing. On exit, the PushController's time interval would be cut short, creating an aggregation over a shorter period. This could accidentally look like an outlier e.g. if you were measuring QPS with a Counter and PushController interval of 1 second, but the flushed measurement was cut at 0.1 seconds.

Many aggregators have a last_update_timestamp that the exporter could forward on to the metric backend to account for this. Does that handle this case? Or maybe the meter should know if it needs to normalize the measurement somehow?

Python 3.5 doesn't have `assert_called_once()`
Copy link
Contributor

@lzchen lzchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just a non-blocking comment.

@lzchen
Copy link
Contributor

lzchen commented May 29, 2020

@aabmass
Yes that is a good concern. It is up to the ingestion service to handle outliers. We provide the timestamp as you've mentioned, and that should be sufficient enough information for whatever backend is analyzing this data. Our responsibility is to be transparent in the behaviour of SDK (it must be clear to users that there is a force flush before exit so they can readily handle these cases).

@aabmass aabmass changed the title Flush metrics on exit Flush metrics on exit, resolves #664 May 29, 2020
@aabmass aabmass changed the title Flush metrics on exit, resolves #664 Flush metrics on exit May 29, 2020
@aabmass aabmass requested a review from lzchen May 29, 2020 17:02
Copy link
Contributor

@codeboten codeboten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the contribution!

@toumorokoshi toumorokoshi merged commit 602ddb0 into open-telemetry:master May 30, 2020
@aabmass aabmass deleted the flush-metrics-664 branch May 30, 2020 14:35
srikanthccv pushed a commit to srikanthccv/opentelemetry-python that referenced this pull request Nov 1, 2020
* chore: rename registry to provider

* chore: revert changelog modification

* chore: remove examples and getting started from rename
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants