-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Shutdown hang (Failure AvailabilityTests.test_recovery_after_catastrophic_failure) #5178
Comments
(2/738 failures in last 30 days, most recent is the one linked above) |
This is unrelated to #5078 -- that was a log error, this is a producer hang. |
Analyzed https://buildkite.com/redpanda/redpanda/builds/12962#018228b4-7137-45b5-a4db-e3d995339a6a This is a shutdown hang. docker-rp-22 is supposed to be stopped and then started, but it fails to stop. failure_injector catches + logs the exception of that happening, then proceeds not to schedule the action to recover the node, so it stays in its shut-down-but-still-running state. Only one node in the cluster is still alive, so clients do not progress. |
The last log message from a particular subsystem stopping is
It isn't necessarily that subsystem that's causing a problem, as they unfortunately don't all have log messages when they start stopping. The next one in the shutdown order that does have a log message is config_manager, so the hang is in one of these steps:
|
This motivates improvement of the controller shutdown logging: #5672 |
Previously, a hang on SIGINT would be logged, but then the test would proceed with a node in this zombie state. Since shutdown hangs are always a bug, it is correct to fail the test as early as we can with the right reason, so just re-raise the exception instead of trying to proceed. Related: redpanda-data#5178
This also motivates improving the test to fail at the point of the shutdown hang rather than later: #5677 |
Previously, a hang on SIGINT would be logged, but then the test would proceed with a node in this zombie state. Since shutdown hangs are always a bug, it is correct to fail the test as early as we can with the right reason, so just re-raise the exception instead of trying to proceed. Related: redpanda-data#5178
Reproduced (hopefully with new logging?) https://buildkite.com/redpanda/redpanda/builds/13352#01824c53-bbf6-4028-bcf2-9a1fac1ddc98 |
Logs appear to point to health_monitor_backend not shutting down properly while it had a request in flight to the suspended peer:
|
If refresh_cluster_health_cache was waiting on _refresh_mutex while ::stop ran, and another fiber had a refresh in progress, then ::stop cancels the other fiber's refresh + the first fiber proceeds to try and refresh again, holding the gate open while ::stop is waiting for it to close. Fixes redpanda-data#5178
If refresh_cluster_health_cache was waiting on _refresh_mutex while ::stop ran, and another fiber had a refresh in progress, then ::stop cancels the other fiber's refresh + the first fiber proceeds to try and refresh again, holding the gate open while ::stop is waiting for it to close. Fixes redpanda-data#5178 (cherry picked from commit d32c9a0)
Previously, a hang on SIGINT would be logged, but then the test would proceed with a node in this zombie state. Since shutdown hangs are always a bug, it is correct to fail the test as early as we can with the right reason, so just re-raise the exception instead of trying to proceed. Related: redpanda-data#5178
From PandaResults:
Assigning to @jcsp since you recently closed #5078
The text was updated successfully, but these errors were encountered: