-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failure in RetentionPolicyTest.test_changing_topic_retention_with_restart
#2406
Comments
Pending fix for redpanda-data#2406 Signed-off-by: John Spray <jcs@vectorized.io>
Thinking about this more, maybe this is not just a test behavior issue -- shouldn't redpanda handle the request even if the controller group is mid-election? Maybe the redpanda node should block the request until the controller group is ready to service the request. |
This test is now disabled in |
Today, I think rebased on dev,
Although I may have forgotten to pull upstream, so it's possible this build was missing the test disabling. |
Related: redpanda-data#2406 Signed-off-by: John Spray <jcs@vectorized.io>
Instead of returning `unknown_server_error`, return `not_controller` This is a retriable error code, with metadata refresh. Fixes redpanda-data#2406 Signed-off-by: Ben Pope <ben@vectorized.io>
Instead of returning `unknown_server_error`, return `not_controller` This is a retriable error code, with metadata refresh. Fixes #2406 Signed-off-by: Ben Pope <ben@vectorized.io>
this has reappeared in https://buildkite.com/vectorized/redpanda/builds/2706#0bfa0f0f-86a4-473f-a3eb-bea8d0699aa6 . should we reopen this or open a new issue? |
Thanks for reopening - here's another failure example from test-staging (https://buildkite.com/vectorized/redpanda/builds/2703#9e549b84-79ef-482a-8db9-4c38b68aa34a). |
#2428 has worked, but it looks like
|
Pending redpanda-data#2406 Signed-off-by: John Spray <jcs@vectorized.io>
PR redpanda-data#2428 changed the returned error code so that the error was retriable with a metadata refresh, but kafka-configs.sh doesn't retry. Call describe_topic to wait for the controller, as it performs a retry. Fix redpanda-data#2406 Signed-off-by: Ben Pope <ben@vectorized.io>
Pending redpanda-data#2406 Signed-off-by: John Spray <jcs@vectorized.io>
Seeing
In: https://buildkite.com/redpanda/redpanda/builds/11813#0181ad2c-892a-4d2e-a49e-3e5a71f4cc95/1602-8550 Should we reopen this or create a new issue? |
Reopening |
(Note: this appears to be a totally different failure to the one this ticket was originally for) |
This test was assuming that segment sizes apply exactly, but they are actually subject to a +/- 5% jitter. To reliably remove the expected number of segments, we must set our retention bytes to 5% less than the amount we really expect to retain. Fixes redpanda-data#2406
Dissecting most recent failure (https://buildkite.com/redpanda/redpanda/builds/12024#0181c5f8-c7d2-4e43-a227-5d7d8020d14c) I see that all nodes have prefix truncated their logs, and nodes 1,2,3 have 17,15,16 segments remaining respectively. The test asserts that at most 16 segments may remain. Node 1 has a slightly lower start offset on its oldest segment than the others. This may be just the segment size jitter conflicting with the test's expectations of a deterministic number of segments remaining after setting a lower retention. It makes sense that this would affect the _with_restart test and not the others in the same class, because the other tests in the class use smaller segment counts where the probability of cumulative jitter having this eeffect is much lower. |
This test was assuming that segment sizes apply exactly, but they are actually subject to a +/- 5% jitter. To reliably remove the expected number of segments, we must set our retention bytes to 5% less than the amount we really expect to retain. Fixes redpanda-data#2406
This test was assuming that segment sizes apply exactly, but they are actually subject to a +/- 5% jitter. To reliably remove the expected number of segments, we must set our retention bytes to 5% less than the amount we really expect to retain. Fixes redpanda-data#2406
Example failure: https://buildkite.com/vectorized/redpanda/builds/2496#a4a3cbd2-d134-4234-a7d3-6027cf50801a
The test tries to modify a topic config immediately after a cluster restart and the server rejects it.
A few milliseconds later the node that rejected the client request learns of the new controller leader
The text was updated successfully, but these errors were encountered: