Add OoO instructions for playbook MimirCompactorHasNotSuccessfullyRunCompaction #1504
Conversation
|
just realized that these are public docs and we can't have internal links, I'll rework this... |
|
I think you should replace the Explore page links with just You'll also have to remove the mimir-squad link. |
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
| - **What it means**: The compactor successfully validated the source blocks. But the validation of the result block after the compaction did not succeed. The result block was not uploaded and the compaction job will be retried. | ||
| - Out-of-order chunks | ||
| - **How to detect**: Search compactor logs for `invalid result block` and `out-of-order chunks`. | ||
| - This is caused by a bug in the ingester. Ingesters upload blocks where the MinT and MaxT of some chunks don't match the first and last samples in the chunk. When the faulty chunks' MinT and MaxT overlap with other chunks, the compactor merges the chunks. Because one chunk's MinT and MaxT are incorrect the merge may be performed incorrectly, leading to OoO samples. |
There was a problem hiding this comment.
Is there any issue for this bug?
There was a problem hiding this comment.
There is but it is in mimir-squad - not public. I made a change in DT in a PR linked above that inserts these links to the playbook. it's mimir-squad 453. Do you think that should be moved to mimir?
There was a problem hiding this comment.
(also tagging you just in case @pracucci )
There was a problem hiding this comment.
There is but it is in mimir-squad - not public
Would be great to have a public issue too (redacting information from any sensitive data).
There was a problem hiding this comment.
👍 opened #1537
pracucci
deleted the
dimitar/update-MimirCompactorHasNotSuccessfullyRunCompaction
branch
…Compaction (#1504) * Add OoO instructions for MimirCompactorHasNotSuccessfullyRunCompaction * Strip out private links Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> * Address PR comments Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> * Fix JSON formatting Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
What this PR does
Add mitigation steps for
MimirCompactorHasNotSuccessfullyRunCompactionwhen the compactor produces out-of-order chunks.Which issue(s) this PR fixes or relates to
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]