Introduces new compacted verifier unaffected by the "ssh" errors #8413

rystsov · 2023-01-25T23:19:19Z

Introduces new compacted verifier unaffected by the "ssh" errors.

Fixes #7979

Backports Required

UX Changes

none

Release Notes

none

tests/docker/Dockerfile

tests/java/verifiers/src/main/java/io/vectorized/compaction/idempotency/Workload.java

tests/rptest/services/compacted_verifier.py

tests/docker/Dockerfile

tests/java/verifiers/src/main/java/io/vectorized/compaction/idempotency/App.java

tests/java/verifiers/src/main/java/io/vectorized/compaction/idempotency/Workload.java

tests/rptest/tests/compaction_e2e_test.py

rystsov · 2023-01-28T21:30:38Z

New issues:

CI Failure (transactions stuck in not_leader_for_partition loop) in CompactionE2EIdempotencyTest.test_basic_compaction #8486 (this test is more powerful than the previous and we run into a previously hidden issue, not related to the PR)
CI Failure (serde overflow or underflow) in CompactionE2EIdempotencyTest.test_basic_compaction #8491 (same, change in the python code can't explain RP failure)
CI Failure (wait_until segment_number_matches - compaction stuck) in CompactionE2EIdempotencyTest.test_basic_compaction #8492 (changes in the test can't explain why compaction has stuck; it could be an increased load but the producers were working for only for 30 secs and it shouldn't lead to stuck for at least 5 mins compaction)
CI Failure (Unable to receive response from driver) in KafkaStreamsSumLambda.test_kafka_streams #8588 (unrelated: change in one test can't explain failure in another)
CI Failure (connection failure in get_decommission_status) in RandomNodeOperationsTest.test_node_operations #8589 (unrelated: change in one test can't explain failure in another)
CI Failure (Unable to receive response from driver) in PartitionMovementTest.test_availability_when_one_node_down #8590 (unrelated: change in one test can't explain failure in another)

Known issues:

Moving test debs which change rarely up the docker file to prevent long rebuilds caused by frequently changed debs in the head of the dockerfile before them

we don't use kafka-verifier, removing the dead code

log-compacted-verifier is less powerful then the new compacted verifier. removing it to free name and to avoid using it because we have a more powerful alternative

it's wasteful to download the same dependencies over and over introducing a verifiers project to unite the java projects to share debs

it's a replacement for log-compacted-verifier. new veri- fier does stricter verification and doesn't stream data over ssh to avoid common paramiko errors. The app exposes control and metrics over http. So there is no need to eyeball the number of records: - start the workload - use the api to check that the app doing a progress at key points of the test - stop the app when the need without waiting for the remaining messages to pass

compaction e2e test is an iteration of compaction end to end test which switches to more powerful verifier for idempotency workload which also isn't affected by the ssh errors

redpanda-data#8491 redpanda-data#8486

This reverts commit e5a4f2f. After replacement of compaction_end_to_end_test.py the commit became obsolete.

This reverts commit e77f67e. After replacement of compaction_end_to_end_test.py the commit became obsolete.

rystsov requested review from mmaslankaprv and bharathv January 25, 2023 23:19

rystsov force-pushed the issue-7979 branch 4 times, most recently from d75253c to 558e9f2 Compare January 26, 2023 02:10

rystsov marked this pull request as ready for review January 26, 2023 02:12

rystsov requested a review from a team as a code owner January 26, 2023 02:12

rystsov requested review from gousteris and removed request for a team January 26, 2023 02:12