Skip to content

Commit

Permalink
tests: don't ignore shutdown hangs in failure injector
Browse files Browse the repository at this point in the history
Previously, a hang on SIGINT would be logged, but then
the test would proceed with a node in this zombie state.

Since shutdown hangs are always a bug, it is correct to fail
the test as early as we can with the right reason, so just
re-raise the exception instead of trying to proceed.

Related: redpanda-data#5178
  • Loading branch information
jcsp committed Jul 27, 2022
1 parent 9ad7e9d commit ed64407
Showing 1 changed file with 8 additions and 1 deletion.
9 changes: 8 additions & 1 deletion tests/rptest/services/failure_injector.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,19 @@ def __enter__(self):
def __exit__(self, type, value, traceback):
self._heal_all()

def inject_failure(self, spec):
def inject_failure(self, spec, require_success=False):
self.redpanda.logger.info(f"injecting failure: {spec}")
try:
self._start_func(spec.type)(spec.node)
except Exception as e:
self.redpanda.logger.info(f"injecting failure error: {e}")
if spec.type == FailureSpec.FAILURE_TERMINATE and isinstance(
e, TimeoutError):
# A timeout during termination indicates a shutdown hang in redpanda: this
# is a bug and we should fail the test on it. Otherwise we'd leave the node
# in a weird state & get some non-obvious failure later in the test, such
# as https://github.com/redpanda-data/redpanda/issues/5178
raise
finally:
if spec.length is not None:
if spec.length == 0:
Expand Down

0 comments on commit ed64407

Please sign in to comment.