Skip to content

Commit

Permalink
tests: don't ignore shutdown hangs in failure injector
Browse files Browse the repository at this point in the history
Previously, a hang on SIGINT would be logged, but then
the test would proceed with a node in this zombie state.

Since shutdown hangs are always a bug, it is correct to fail
the test as early as we can with the right reason, so just
re-raise the exception instead of trying to proceed.

Related: redpanda-data#5178
  • Loading branch information
jcsp committed Jul 28, 2022
1 parent 677155c commit 190a3a0
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions tests/rptest/services/failure_injector.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,13 @@ def inject_failure(self, spec):
self._start_func(spec.type)(spec.node)
except Exception as e:
self.redpanda.logger.info(f"injecting failure error: {e}")
if spec.type == FailureSpec.FAILURE_TERMINATE and isinstance(
e, TimeoutError):
# A timeout during termination indicates a shutdown hang in redpanda: this
# is a bug and we should fail the test on it. Otherwise we'd leave the node
# in a weird state & get some non-obvious failure later in the test, such
# as https://github.com/redpanda-data/redpanda/issues/5178
raise
finally:
if spec.length is not None:
if spec.length == 0:
Expand Down

0 comments on commit 190a3a0

Please sign in to comment.