-
Notifications
You must be signed in to change notification settings - Fork 512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prevent ingesting samples older than past_grace_period
#8262
Conversation
This PR implements a new limit, similar to `creation_grace_period`, that limits the age of the ingested samples based on the wall clock. It also takes into account the OOO window, to avoid having to change both settings when OOO window is modified. The default value of 1 week should cover any observability needs. If some customer wants to backfill their samples in order, they can increase the value of that to `100y` in the config to cover 100 years, for example. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work!
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
I'll wait for @jdbaldry's review on the runbooks before merging. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Docs LGTM and are consistent with the rest of the runbook format.
nit: consider using contractions to match our tone guidelines: https://grafana.com/docs/writers-toolkit/write/style-guide/voice-tone-guidelines/#tone
Co-authored-by: Jack Baldry <jack.baldry@grafana.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
What this PR does
This PR implements a new limit, similar to
creation_grace_period
, that limits the age of the ingested samples based on the wall clock.It also takes into account the OOO window, to avoid having to change both settings when OOO window is modified.
The default value of 1 week should cover any observability needs. If some customer wants to backfill their samples in order, they can increase the value of that to
100y
in the config to cover 100 years, for example.Which issue(s) this PR fixes or relates to
Fixes #
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]
.about-versioning.md
updated with experimental features.