Add debugging for hung tests #10
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I have a fake hanging test I can check in (with ok to fail) to test this out in our various CI environments.. LMK if you like that idea. |
ajfabbri
commented
Jun 10, 2022
ajfabbri
force-pushed
the
handle-hung-tests
branch
2 times, most recently
from
aa6bc31
to
70da78b
Compare
added 8 commits
August 16, 2022 12:43
- Don't double-print a stacktrace - Don't chain our TimeoutError with zmq.Again exception, etc. The cause of these exceptions are already clear from the message, and there are only a couple of cases that cause it.
The main test runner should be responsive to child process IPCs. Reduce the timeout for send and recv to parent from 3 to 1 seconds.
Instead of setting linger socket option to zero on a clean passive-side shutdown, set it to a reasonable number when we create the socket.
Add a SIGUSR1 handler to child runner client processes, which we can use to force a stuck test to print out stack traces.
This is used to force stuck child processes to print stacktraces.
To allow us to debug hard-to-reproduce "hung test" failures, this commit adds special handling of test timeouts which, for any still-running child processes: - Sends a SIGUSR1 which causes them to print stack traces of all threads. - Sends a SIGINT and wait for children processes to exit. The goals are to provide diagnostics on where child test clients are stuck, and to allow the ducktape processes to exit, which should allow CI to gather logs.
ajfabbri
force-pushed
the
handle-hung-tests
branch
from
70da78b
to
80360b1
Compare
|
Force-push: rebase on latest. I finally got around to porting this to upstream ducktape and submitting a PR here. |
|
Cleaning up old PRs. Closing this one until I get upstream merged. |
This pull request was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When debugging hard-to-reproduce "hung" test failures (like redpanda #4634), we ran into a couple of issues:
The solution makes a couple of modifications:
Here is an example of the new ouptut you should see for a hanging test: