tests: Deflake test_migrating_consume_offsets #5074
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@mmaslankaprv you added this test, you might have some context here, FYI. |
bharathv
force-pushed
the
test_migrating_consume_offsets
branch
from
ab7297b
to
68013af
Compare
dotnwat
reviewed
Jun 8, 2022
| # or restarts. We only want to do one of those at a time to avoid conflicts. | ||
| busy_nodes = set() | ||
| # synchronize access to busy_nodes set. | ||
| busy_nodes_lock = threading.RLock() |
There was a problem hiding this comment.
Fixes synchronization around failure injection and
node restarts.
could you add a little context into the commit message about what was wrong and how it was fixed?
Looped through the test 50 times locally and didn't
run into any issues
this is great context to add to the PR cover letter, but doesn't add much value in the commit message.
There was a problem hiding this comment.
I updated the commit message and did a force push.
Fixes synchronization around failure injection and node restarts. The test is structured as follows. 1. start a cluster without consumer_offsets support 2. Run an async failure injection loop that can potentially do a SIGKILL on a random redpanda process. 3. Do a consumer groups test on a sample topic. 4. Restart the cluster to enable the new __consumer_offsets topic feature. 5. Verify that __consumer_offsets functionality works as expected. It turns out that (2) and (4) are conflicting with each other due to poor synchronization. Due to some racy code these operations are interleaved resulting in failure injector attempting to kill a non-existent PID. This patch makes sure that only one of these two operations run at any given time. A given node is marked 'busy' in a thread-safe way if either of the operations is running and the other thread has to wait.
bharathv
force-pushed
the
test_migrating_consume_offsets
branch
from
68013af
to
1fb9dec
Compare
bharathv
commented
Jun 8, 2022
| # or restarts. We only want to do one of those at a time to avoid conflicts. | ||
| busy_nodes = set() | ||
| # synchronize access to busy_nodes set. | ||
| busy_nodes_lock = threading.RLock() |
There was a problem hiding this comment.
I updated the commit message and did a force push.
thanks! it's excellent ✏️ |
dotnwat
added
the
ci-repeat-10
vbotbuildovich
removed
the
ci-repeat-10
dotnwat
approved these changes
Jun 8, 2022
|
looks great |
bharathv
added
the
ci-repeat-10
vbotbuildovich
removed
the
ci-repeat-10
|
1/10 runs failed with I think this is good to go. |
|
/backport v22.1.x |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cover letter
Fixes synchronization around failure injection and node restarts.
Looped through the test 50 times locally and didn't
run into any issues
Fixes ##5034
Release notes
force-push Amends the commit message to add more detail.