tests: segregate resource-intensive tests in scale_tests/ #4683
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jcsp
added
kind/tech-debt
area/tests
ci-repeat-5
vbotbuildovich
removed
the
ci-repeat-5
|
Running with 5x repeats to measure how impactful this change is to the overall success rate of tests. |
|
🟢 5/5 of those runs are green. |
jcsp
force-pushed
the
segregate-intensive-tests
branch
3 times, most recently
from
bcb7ee9
to
7d4b976
Compare
These tests are too traffic-intensive for the docker test environment: they make other tests unreliable as well as themselves.
This is by nature a benchmark, putting more load on our docker test environment than it's comfortable with.
This test is best run with a substantial runtime on dedicated nodes. On docker, it's both too brief and too intensive.
These were orginally written for compatibility testing, but the workload in use is a benchmark: we shouldn't run it among all our functional CI tests.
jcsp
force-pushed
the
segregate-intensive-tests
branch
from
7d4b976
to
665b8d4
Compare
Lazin
approved these changes
May 13, 2022
This pull request was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cover letter
Our CI stability issues are spiking. Part of it is how overloaded the executor machines are. At the same time, our runtimes are becoming problematic.
Tests that do significant amounts of I/O are not a good fit for the docker execution environment where the main set of tests run: in docker containers all sharing the same disk and oversubscribed CPUs.
Move them into scale_tests/ which is run exclusively in clustered ducktape, on dedicated EC2 nodes.
Release notes