[v22.2.x] tests: update ManyPartitionsTest #6044
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was outputting number, should have been outputting message. (cherry picked from commit 8787da4)
The go dependencies are generally the fastest to build and should not get held up behind other things: - Move OMB (Java build) further up - split `kaf` install from unrelated non-go stuff. - move client-swarm build before go test utils (cherry picked from commit 73edc8a)
So that it shows the node name properly (cherry picked from commit 06e92c8)
This is for the benefit of scale tests, which would like to reduce their per-partition outputs to reflect how a user would configure the system, and to reduce any overhead from emitting millions of lines. (cherry picked from commit 8fb6d0d)
This wraps the new `kgo-repeater` traffic generator for scalable load generation. (cherry picked from commit 763381f)
It is helpful to print the error right at the point of failure, rather than after the (potentially long running) backtrace decode & log search jobs. It'll get printed again later as well, but this way I can search from the start of the file for the exception name, and jump straight to the timestamp of the failure. (cherry picked from commit d3a3087)
This is a nasty failure mode where we deploy fresh packages and accidentally wip out our /var/lib/redpanda symlink, resulting in running tests on very slow drives. (cherry picked from commit 087d653)
This is an efficiency/quality of life improvement for working with tests that start larger numbers of nodes. Leave the default as serial startup, because it makes logs easier to read. (cherry picked from commit ae72725)
This is useful if a test is running longer than you expected and you'd like to know how far through it is without doing your own calculation of message counts. (cherry picked from commit 1432357)
When using this function to query leadership for partitions, it is not necessary to exclude partitions just because they failed to get some metadata from the leader (e.g. NOT_LEADER errors for offets during transient leaderhsip change). Add a `tolerant` flag that permits returning partially populated RpkPartition results that just show the leader of a partition. (cherry picked from commit d50400a)
(cherry picked from commit b92846b)
The default mode is rather expensive for high partition counts, and complicates handling systems in transient states when one or more of the partitions is likely to be underoing leadership movement and therefore have NOT_LEADER errors etc in the default per-partition output. When all we want to know is the group's state, this lets us get that. (cherry picked from commit 0c687f9)
(cherry picked from commit bcb2102)
(cherry picked from commit 75f64f1)
(cherry picked from commit 4a82326)
(cherry picked from commit b24f4bd)
(cherry picked from commit 1a47771)
(cherry picked from commit 5e895f2)
This enables: - Running on different instance types without hacking the test - Running on local docker while developing the test itself. (cherry picked from commit e248199)
(cherry picked from commit e152c8e)
I think this is a bug with the workload generator (or, unlikely perhaps a problem with franz-go). It is usually only a few consumers that disappear from the group, so it doesn't hurt the validity of the overall scale test, and we can hunt it down separately. (cherry picked from commit c17e2cb)
This is followup from PR redpanda-data#5816 (cherry picked from commit 1689a72)
jcsp
approved these changes
Aug 17, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport from pull request: #5816.
Fixes #6043,
[jcsp] this also includes the 6 extra cleanup commits from https://github.com/redpanda-data/redpanda/pull/5970/commits