Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce duration of partition_movement_test from 25min to 8min #5238

Draft
wants to merge 7 commits into
base: dev
Choose a base branch
from
Draft

Reduce duration of partition_movement_test from 25min to 8min #5238

wants to merge 7 commits into from

Commits on Dec 18, 2022

  1. ducky: copy remote scripts into docker

    @rystsov
    rystsov committed Dec 18, 2022
    Configuration menu
    Copy the full SHA
    a14772e View commit details
    Browse the repository at this point in the history

  2. ducky: optimize dockerfile 

    Moving test debs which change rarely up the docker file to
prevent long rebuilds caused by frequently changed debs in
the head of the dockerfile before them
    @rystsov
    rystsov committed Dec 18, 2022
    Configuration menu
    Copy the full SHA
    b8efc89 View commit details
    Browse the repository at this point in the history

  3. ducky: remove kafka-verifier 

    we don't use kafka-verifier, removing the dead code
    @rystsov
    rystsov committed Dec 18, 2022
    Configuration menu
    Copy the full SHA
    de01609 View commit details
    Browse the repository at this point in the history

  4. ducky: create mvn verifiers project 

    it's wasteful to download the same dependencies over and over
introducing a verifiers project to unite the java projects to
share debs
    @rystsov
    rystsov committed Dec 18, 2022
    Configuration menu
    Copy the full SHA
    a31acee View commit details
    Browse the repository at this point in the history

  5. ducky: add read-write verifier 

    rw_verifier is a replacement for verified producer and
consumer services. rw_verifier is a workload generator
and an online linear and incremental consistency checker
capable of running for hours without an overhead.

The validation happens as the write / read go. No need
to pass written/read data to the test over ssh for vali-
dation.

The app exposes control and metrics over http. So there
is no need to eyeball the number of records:
 - start the workload
 - use the api to check that the app doing a progress at
   key points of the test
 - stop the app when the need without waiting for the
   remaining messages to pass
    @rystsov
    rystsov committed Dec 18, 2022
    Configuration menu
    Copy the full SHA
    9be500a View commit details
    Browse the repository at this point in the history

  6. ducky: tune partition_movement to be faster 

    duration of partition_movement_test.py falls down from
14 minutes 15.077 seconds to 12 minutes 39.137 seconds
    @rystsov
    rystsov committed Dec 18, 2022
    Configuration menu
    Copy the full SHA
    36247ee View commit details
    Browse the repository at this point in the history

  7. partition_movement_test: use new verifier 

    duration of partition_movement_test.py falls down from
12 minutes 39.137 seconds to 8 minutes 53.753 seconds
    @rystsov
    rystsov committed Dec 18, 2022
    Configuration menu
    Copy the full SHA
    996620a View commit details
    Browse the repository at this point in the history