Make wait_until faster by ignoring backoff_sec on success

[rystsov]

it isn't uncommon to have dozens of wait_until in a test. With backoff_sec set to 2 seconds 30 successful wait_until sums up to a minute of wasted CI time.

[jcsp]

LGTM. I guess we should test that this doesn't destabilize any existing tests which might have implicit dependency on that extra sleep: probably need a throwaway vtools PR with ducktape pointed at this sha1 to check that.

[rystsov]

I've done it locally

Without the fix

ducktape version: 0.8.8
session_id:       2022-08-11--001
run time:         426 minutes 33.727 seconds
tests run:        729
passed:           503
failed:           185
ignored:          32
opassed:          8
ofailed:          1

With the fix

ducktape version: 0.8.8
session_id:       2022-08-12--001
run time:         299 minutes 42.224 seconds
tests run:        729
passed:           496
failed:           192
ignored:          32
opassed:          8
ofailed:          1

Look at the reduced duration!

Test failures missing in the first run:

rptest.tests.maintenance_test.MaintenanceTest.test_maintenance.use_rpk=True
rptest.tests.consumer_group_test.ConsumerGroupTest.test_dead_group_recovery.static_members=True
rptest.tests.maintenance_test.MaintenanceTest.test_exclusive_maintenance.use_rpk=False
rptest.tests.offset_for_leader_epoch_test.OffsetForLeaderEpochTest.test_offset_for_leader_epoch
rptest.tests.maintenance_test.MaintenanceTest.test_maintenance_sticky.use_rpk=True
rptest.tests.maintenance_test.MaintenanceTest.test_maintenance_sticky.use_rpk=False
rptest.tests.wasm_redpanda_failure_recovery_test.WasmRPMeshFailureRecoveryTest.verify_materialized_topics_test
rptest.tests.consumer_group_test.ConsumerGroupTest.test_dead_group_recovery.static_members=False
rptest.tests.cluster_features_test.FeaturesMultiNodeTest.test_explicit_activation
rptest.tests.maintenance_test.MaintenanceTest.test_exclusive_maintenance.use_rpk=True

I looked at them and it may be an effect of implicit dependency of extra sleep. But it seems that the change doesn't destabilize but reveals previously hidden flakiness; working on identifying them.

[dotnwat]

so good