Reduce duration of partition_movement_test from 25min to 8min #5238

rystsov · 2022-06-27T02:25:25Z

Cover letter

reduces duration of partition_movement_test from 25min to 8min
introduce a workload generator / consistency checker to verified producer / consumers

The new verifier is based on the verifiers we use in chaos tests. The key features of this verifier:

It's an app using a client instead of being a CLI kafka tool running in a bash loop (KafProducer) which results in more organic usage pattern.
Low distance between kafka client and the app: app->driver vs app->kgo->driver. The lower distance means it's easier to debug when something goes wrong.
It uses the most popular client (easier to distinguish client issue vs redpanda issue).
It detects out of order and duplicated messages
It already detected this problem Fix duplicates consistency error by caching already translated offsets #5356
It treats timeouts as indecisive errors (not subject to Consumer failed to consume up to offsets in SIPartitionMovementTest.test_shadow_indexing #4702)
It's an online verifier which uses end-to-end validation with a bounded amount of RAM
It exposes http interface to control the workload and query statistics
- without bidirectional communication it's impossible to establish causality and to check progress at a given point of time
- http interface helps with clean shutdowns to avoid test: rptest.tests.topic_delete_test.TopicDeleteStressTest.stress_test unclean client shutdown #4326
It has an error log which is pulled after the test is over (to help with debugging the failing tests)
We don't adopt external code (VerifiableProducer) so no licensing questions

Release notes

none

jcsp

Most of the changes focussed on the test itself are fine, but I worry that we are repeating the same mistake of adding new workloads rather than extending/adapting existing ones.

I'm totally on board with removing KafProducer, but when it comes to replacements, we already have FranzGoVerifiableProducer which does pretty similar stuff to the new workloads added here (and I think the Java VerifiableProducer is not so different either?). If we were consolidating one one language I could understand, but the new workloads are in a mixture of languages too, so I'm left kind of uncertain about the motivation/direction.

tests/docker/Dockerfile

tests/rptest/remote_scripts/control/rw_alive.sh

tests/rptest/remote_scripts/w_workload.py

tests/rptest/services/w_workload.py

tests/rptest/remote_scripts/w_workload.py

NyaliaLui · 2022-07-14T18:21:49Z

@rystsov What's the status on this PR? Does it need review or are we waiting for some changes?

rystsov · 2022-07-14T19:40:45Z

@NyaliaLui I've addressed the comments and it waits for review

rystsov · 2022-07-14T20:12:51Z

Most of the changes focused on the test itself are fine, but I worry that we are repeating the same mistake of adding new workloads rather than extending/adapting existing ones.

@jcsp The new verifier follows a different approach to testing so extending the existing verifiers would require more work. Once this PR is merged I'll push for giving up on VerifiableProducer, VerifiableConsumer and KafProducer. I've described the key features of this verifier in the cover letter.

NyaliaLui

I like where this is going and the fact that the tool has helped detect a problem adds to the rw_worloads value.

tests/rptest/services/rw_workload.py

NyaliaLui

Y'all the PR was ready to merge a while ago, I fixed the conflicts and it's ready to go. IMHO we may do an exception and let it go in despite the blockers. They aren't caused by the PR and the PR introduces a verifier to solve the "Consumer failed to consume up to offsets" ci-failure. What do you think?

LGTM

Moving test debs which change rarely up the docker file to prevent long rebuilds caused by frequently changed debs in the head of the dockerfile before them

we don't use kafka-verifier, removing the dead code

it's wasteful to download the same dependencies over and over introducing a verifiers project to unite the java projects to share debs

rw_verifier is a replacement for verified producer and consumer services. rw_verifier is a workload generator and an online linear and incremental consistency checker capable of running for hours without an overhead. The validation happens as the write / read go. No need to pass written/read data to the test over ssh for vali- dation. The app exposes control and metrics over http. So there is no need to eyeball the number of records: - start the workload - use the api to check that the app doing a progress at key points of the test - stop the app when the need without waiting for the remaining messages to pass

duration of partition_movement_test.py falls down from 14 minutes 15.077 seconds to 12 minutes 39.137 seconds

duration of partition_movement_test.py falls down from 12 minutes 39.137 seconds to 8 minutes 53.753 seconds

dotnwat

Generally looks good to me.

I think several files need a copyright header
The set of features / benefits listed in the cover letter are impressive, but it's not clear why "Reduce duration of partition_movement_test from 25min to 8min" this is true. Is there some insight into the time savings?

tests/rptest/tests/partition_movement_test.py

rystsov · 2022-12-21T01:19:07Z

but it's not clear why "Reduce duration of partition_movement_test from 25min to 8min" this is true. Is there some insight into the time savings?

it was true back in summer but now the benefits should be smaller, the major boost came from optimizing wait_until which was put in a separate PR since them the second best boost came from using logic like:

start continuous workload
await n updates
perform partition movement
await n updates
stop the workload

instead of:

start workload & schedule N operations (where N is big enough)
perform partition movement
wait until the workload is stopped

I'll change the description to avoid the confusion

mmaslankaprv · 2022-12-21T10:28:17Z

tests/java/verifiers/src/main/java/io/vectorized/reads_writes/Workload.java

+
+  static volatile long started_s = -1;
+  static int workload_id_gen = 0;
+  static synchronized int get_workload_id() { return workload_id_gen++; }


can we stick to the same naming convention for all functions/methods i.e. camelCase ?

mmaslankaprv · 2022-12-21T10:29:07Z

tests/java/verifiers/src/main/java/io/vectorized/reads_writes/Workload.java

+
+  HashMap<Integer, Queue<Long>> oids;
+  HashMap<Long, OpInfo> ops;
+  HashMap<Integer, Semaphore> rsems;


would it be possible to make those variable names more descriptive ?

tests/rptest/tests/partition_movement_test.py

tests/java/verifiers/src/main/java/io/vectorized/reads_writes/Workload.java

mmaslankaprv · 2022-12-21T10:40:28Z

tests/java/verifiers/src/main/java/io/vectorized/reads_writes/Workload.java

+      ConsumerRecords<String, String> records
+          = consumer.poll(Duration.ofMillis(10000));
+      var it = records.iterator();
+      while (it.hasNext()) {


for this to be equivalent with existing VerifiableConsumer it would have to commit offsets, tolerate errors (redeliveries) and use consumer groups

I agree that RWVerifier doesn't cover consumer groups. But I don't think full equivalence is the right goal, instead of having the verifier we should have a set of specialized verifiers aimed at a particular set of features. With a single verifier we tend not to question how good it is and what it actually covers resulting in a false sense of security.

For example the pair of VerifiableConsumer & VerifiableProducer:

don't check if the consumed order matches the written order

ignore time travel

ignore when a consumer reads something what was never written

ignore indecisive errors on produce

has broken idempotency support (during rebalance a message is allowed to be consumed twice but the validation logic raise an error if there is any duplicate)

doesn't validate if a partition is assigned to multiple clients at the same time

RWVerifier isn't affected by all but the last point. And it already found a problem which VerifiableConsumer & VerifiableProducer missed.

Instead of adding weak support of consumer groups to RWVerifier we should add another verifier exclusively targeting the consumer groups unaffected by the existing flaws.

What I want to have an example of how to write remove services without relying on piping the output via ssh so that when there is a new PR introducing an end_to_end based test I could point to a RWVerifier and ask the author to do something similar.

What do you think of copying partition_movement_test into partition_movement_rw_test and marking original deprecated until we introduce a verifier with the consumer group support? It sounds ugly but postponing this PR also has cost, we'll continue using the regular approach and we'll depend on it more and more.

got it, thank you for explaining all the details i think there is no need to create duplicated tests the one we have now is enough, if the RWVerifier is going to be a goto tool to write EndToEnd tests what do you think about making it a little bit better documented in terms of usage/output and its code cleaned up so that the author of the new features will have an easier job.

dotnwat · 2022-12-24T22:59:00Z

it was true back in summer but now the benefits should be smaller, the major boost came from optimizing wait_until

ahh thanks. i remember that wait_until pr now. thanks!

rystsov requested review from a team, dotnwat and NyaliaLui as code owners June 27, 2022 02:25

rystsov requested review from ivotron, jcsp, mmaslankaprv, andrewhsu and andrwng and removed request for a team June 27, 2022 02:25

rystsov force-pushed the test-workloads branch 5 times, most recently from d019eec to 90a704e Compare June 27, 2022 21:30

jcsp reviewed Jun 28, 2022

View reviewed changes

rystsov force-pushed the test-workloads branch 4 times, most recently from d7084cb to d62a62e Compare July 14, 2022 19:39

rystsov requested a review from jcsp July 14, 2022 19:39

rystsov force-pushed the test-workloads branch from d62a62e to 9ac4f04 Compare July 14, 2022 20:30

NyaliaLui reviewed Jul 19, 2022

View reviewed changes

tests/rptest/services/rw_workload.py Outdated Show resolved Hide resolved

rystsov force-pushed the test-workloads branch from 9ac4f04 to 2e665c3 Compare July 20, 2022 01:03

mmedenjak added the kind/enhance New feature or request label Jul 20, 2022

rystsov requested review from jcsp and mmaslankaprv August 23, 2022 05:40

rystsov force-pushed the test-workloads branch 2 times, most recently from 97d6305 to 43f2b83 Compare September 1, 2022 00:22

NyaliaLui previously approved these changes Sep 1, 2022

View reviewed changes

rystsov dismissed NyaliaLui’s stale review via 548608a December 18, 2022 18:38

rystsov force-pushed the test-workloads branch from 43f2b83 to 548608a Compare December 18, 2022 18:38

rystsov added 6 commits December 18, 2022 10:45

ducky: copy remote scripts into docker

a14772e

ducky: optimize dockerfile

b8efc89

Moving test debs which change rarely up the docker file to prevent long rebuilds caused by frequently changed debs in the head of the dockerfile before them

ducky: remove kafka-verifier

de01609

we don't use kafka-verifier, removing the dead code

ducky: create mvn verifiers project

a31acee

it's wasteful to download the same dependencies over and over introducing a verifiers project to unite the java projects to share debs

ducky: tune partition_movement to be faster

36247ee

duration of partition_movement_test.py falls down from 14 minutes 15.077 seconds to 12 minutes 39.137 seconds

rystsov force-pushed the test-workloads branch from 548608a to 24f3e30 Compare December 18, 2022 18:47

rystsov mentioned this pull request Dec 18, 2022

SSHException in CompactionEndToEndTest test_basic_compaction #6792

Closed

partition_movement_test: use new verifier

996620a

duration of partition_movement_test.py falls down from 12 minutes 39.137 seconds to 8 minutes 53.753 seconds

rystsov force-pushed the test-workloads branch from 24f3e30 to 996620a Compare December 18, 2022 21:29

rystsov mentioned this pull request Dec 20, 2022

CI Failure (failed to stop after SIGKILL) in ConsumerGroupTest.test_dead_group_recovery #7681

Closed

dotnwat reviewed Dec 21, 2022

View reviewed changes

tests/rptest/tests/partition_movement_test.py Show resolved Hide resolved

mmaslankaprv reviewed Dec 21, 2022

View reviewed changes

tests/rptest/tests/partition_movement_test.py Show resolved Hide resolved

mmaslankaprv reviewed Dec 21, 2022

View reviewed changes

tests/java/verifiers/src/main/java/io/vectorized/reads_writes/Workload.java Show resolved Hide resolved

mmaslankaprv reviewed Dec 21, 2022

View reviewed changes

tests/java/verifiers/src/main/java/io/vectorized/reads_writes/Workload.java Show resolved Hide resolved

mmaslankaprv reviewed Dec 21, 2022

View reviewed changes

jcsp removed the consistency label Jan 20, 2023

rystsov marked this pull request as draft January 24, 2023 18:57

rystsov mentioned this pull request Jan 26, 2023

Introduces new compacted verifier unaffected by the "ssh" errors #8413

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce duration of partition_movement_test from 25min to 8min #5238

Reduce duration of partition_movement_test from 25min to 8min #5238

rystsov commented Jun 27, 2022 •

edited

Loading

jcsp left a comment

NyaliaLui commented Jul 14, 2022

rystsov commented Jul 14, 2022

rystsov commented Jul 14, 2022 •

edited

Loading

NyaliaLui left a comment

NyaliaLui left a comment

dotnwat left a comment

rystsov commented Dec 21, 2022

mmaslankaprv Dec 21, 2022

mmaslankaprv Dec 21, 2022

mmaslankaprv Dec 21, 2022

rystsov Dec 22, 2022 •

edited

Loading

rystsov Dec 22, 2022 •

edited

Loading

mmaslankaprv Dec 22, 2022

dotnwat commented Dec 24, 2022

Reduce duration of partition_movement_test from 25min to 8min #5238

Are you sure you want to change the base?

Reduce duration of partition_movement_test from 25min to 8min #5238

Conversation

rystsov commented Jun 27, 2022 • edited Loading

Cover letter

Release notes

jcsp left a comment

Choose a reason for hiding this comment

NyaliaLui commented Jul 14, 2022

rystsov commented Jul 14, 2022

rystsov commented Jul 14, 2022 • edited Loading

NyaliaLui left a comment

Choose a reason for hiding this comment

NyaliaLui left a comment

Choose a reason for hiding this comment

dotnwat left a comment

Choose a reason for hiding this comment

rystsov commented Dec 21, 2022

mmaslankaprv Dec 21, 2022

Choose a reason for hiding this comment

mmaslankaprv Dec 21, 2022

Choose a reason for hiding this comment

mmaslankaprv Dec 21, 2022

Choose a reason for hiding this comment

rystsov Dec 22, 2022 • edited Loading

Choose a reason for hiding this comment

rystsov Dec 22, 2022 • edited Loading

Choose a reason for hiding this comment

mmaslankaprv Dec 22, 2022

Choose a reason for hiding this comment

dotnwat commented Dec 24, 2022

rystsov commented Jun 27, 2022 •

edited

Loading

rystsov commented Jul 14, 2022 •

edited

Loading

rystsov Dec 22, 2022 •

edited

Loading

rystsov Dec 22, 2022 •

edited

Loading