Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Timeout waiting for partition balancer "ready" status in PartitionBalancerTest.test_full_nodes #5980

Closed
VladLazar opened this issue Aug 11, 2022 · 3 comments · Fixed by #6342
Closed

Timeout waiting for partition balancer "ready" status in PartitionBalancerTest.test_full_nodes #5980

VladLazar opened this issue Aug 11, 2022 · 3 comments · Fixed by #6342
Assignees
@ztlpn
Labels
area/redpanda area/tests ci-disabled-test ci-failure kind/bug Something isn't working

Comments

@VladLazar
Copy link
Contributor

VladLazar commented Aug 11, 2022 •
edited
Loading

Looks like the partition balancer did not stabilise for some reason. No crash or backtrace,
so probably not a blocking issue. Still needs looking into though.

File "/root/tests/rptest/tests/partition_balancer_test.py", line 467, in test_full_nodes
    self.wait_until_status(create_waiter("ready"))
File "/root/tests/rptest/util.py", line 74, in wait_until
    raise TimeoutError(
ducktape.errors.TimeoutError: failed to wait until status condition

https://buildkite.com/redpanda/redpanda/builds/13988#01828d2e-b0ad-431b-8918-d3f07ec35a8e
@VladLazar VladLazar added kind/bug Something isn't working ci-failure labels Aug 11, 2022
This was referenced Aug 11, 2022
Merged
Merged
@ztlpn ztlpn mentioned this issue Aug 15, 2022
Merged
5 tasks
@ztlpn
Copy link
Contributor

ztlpn commented Aug 16, 2022

https://buildkite.com/redpanda/redpanda/builds/14200#0182a524-3f13-4bda-a239-6de84ec021e0

@r-vasquez
Copy link
Contributor

Again in #6081

https://buildkite.com/redpanda/redpanda/builds/14592#0182cc46-ca8f-421b-9470-a57d78006ce1

@r-vasquez r-vasquez mentioned this issue Aug 23, 2022
Merged
1 task
@piyushredpanda
Copy link
Contributor

Requesting @ztlpn to take a look.

ztlpn added a commit to ztlpn/redpanda that referenced this issue Sep 8, 2022
@ztlpn
tests/e2e_finjector: add teardown method
c333fc7 
Previously we did not heal network failures when exiting tests that use
this base class (namely, AvailabilityTests). As possible failure types
included netem failures, that meant that all subsequent tests in a test
run used a node with crippled network, leading to flakiness.

Fixes redpanda-data#5980
@ztlpn ztlpn mentioned this issue Sep 8, 2022
Merged
2 tasks
vbotbuildovich pushed a commit to vbotbuildovich/redpanda that referenced this issue Sep 9, 2022
@ztlpn @vbotbuildovich
tests/e2e_finjector: add teardown method
38227b5 
Previously we did not heal network failures when exiting tests that use
this base class (namely, AvailabilityTests). As possible failure types
included netem failures, that meant that all subsequent tests in a test
run used a node with crippled network, leading to flakiness.

Fixes redpanda-data#5980

(cherry picked from commit c333fc7)
@vbotbuildovich vbotbuildovich mentioned this issue Sep 9, 2022
Closed
vbotbuildovich pushed a commit to vbotbuildovich/redpanda that referenced this issue Sep 9, 2022
@ztlpn @vbotbuildovich
tests/e2e_finjector: add teardown method
bce50fc 
Previously we did not heal network failures when exiting tests that use
this base class (namely, AvailabilityTests). As possible failure types
included netem failures, that meant that all subsequent tests in a test
run used a node with crippled network, leading to flakiness.

Fixes redpanda-data#5980

(cherry picked from commit c333fc7)
@vbotbuildovich vbotbuildovich mentioned this issue Sep 9, 2022
Closed
ballard26 pushed a commit to ballard26/redpanda that referenced this issue Sep 27, 2022
@ztlpn @ballard26
tests/e2e_finjector: add teardown method
7f20b8b 
Previously we did not heal network failures when exiting tests that use
this base class (namely, AvailabilityTests). As possible failure types
included netem failures, that meant that all subsequent tests in a test
run used a node with crippled network, leading to flakiness.

Fixes redpanda-data#5980
@BenPope BenPope mentioned this issue Oct 11, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ztlpn ztlpn

Labels
area/redpanda area/tests ci-disabled-test ci-failure kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Heal network failures in the end of EndToEndFinjectorTest ztlpn/redpanda
6 participants
@rystsov @ztlpn @mmedenjak @VladLazar @r-vasquez @piyushredpanda