Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add shuffle-sharding for the compactor #4433

Merged
merged 17 commits into from
Apr 7, 2022
Merged

Add shuffle-sharding for the compactor #4433

merged 17 commits into from
Apr 7, 2022

Commits on Apr 6, 2022

  1. Add metrics for remaining planned compactions 

    Signed-off-by: Albert <ac1214@users.noreply.github.com>
    @ac1214 @roystchiang
    ac1214 authored and roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    776bbce View commit details
    Browse the repository at this point in the history

  2. fix unit tests 

    Signed-off-by: Albert <ac1214@users.noreply.github.com>
    @ac1214 @roystchiang
    ac1214 authored and roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    e8881fb View commit details
    Browse the repository at this point in the history

  3. Add shuffle sharding for compactor 

    Signed-off-by: Albert <ac1214@users.noreply.github.com>
    @ac1214 @roystchiang
    ac1214 authored and roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    0da3e9b View commit details
    Browse the repository at this point in the history

  4. update changelog 

    Signed-off-by: Albert <ac1214@users.noreply.github.com>
    @ac1214 @roystchiang
    ac1214 authored and roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    acb1866 View commit details
    Browse the repository at this point in the history

  5. fix linting 

    Signed-off-by: Albert <ac1214@users.noreply.github.com>
    @ac1214 @roystchiang
    ac1214 authored and roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    2063e96 View commit details
    Browse the repository at this point in the history

  6. Fix build errors 

    Signed-off-by: Alvin Lin <alvinlin@amazon.com>
    @alvinlin123 @roystchiang
    alvinlin123 authored and roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    1c8d4d8 View commit details
    Browse the repository at this point in the history

  7. Fix up change log 

    Signed-off-by: Alvin Lin <alvinlin@amazon.com>
    @alvinlin123 @roystchiang
    alvinlin123 authored and roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    23e79a0 View commit details
    Browse the repository at this point in the history

  8. Fix linting error 

    Signed-off-by: Alvin Lin <alvinlin@amazon.com>
    @alvinlin123 @roystchiang
    alvinlin123 authored and roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    eafb14e View commit details
    Browse the repository at this point in the history

  9. Remove use of nolint 

    Signed-off-by: Alvin Lin <alvinlin@amazon.com>
    @alvinlin123 @roystchiang
    alvinlin123 authored and roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    fb15202 View commit details
    Browse the repository at this point in the history

  10. Compactor.ownUser now determines whether the user is owned by a compa… 

    …ctor via ring, instead of returning true if shuffle-sharding is enabled

Signed-off-by: Roy Chiang <roychi@amazon.com>
    @roystchiang
    roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    ccbf001 View commit details
    Browse the repository at this point in the history

  11. fix bug where multiple compactors are trying to cleanup the same tena… 

    …nt at once, which results in dangling bucket index

Signed-off-by: Roy Chiang <roychi@amazon.com>
    @roystchiang
    roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    3cb44ee View commit details
    Browse the repository at this point in the history

  12. set all remaining compation in one go, instead of slowly incrementing… 

    … it as plans get generated

Signed-off-by: Roy Chiang <roychi@amazon.com>
    @roystchiang
    roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    dd9681b View commit details
    Browse the repository at this point in the history

  13. rename ownUser function for better readability 

    Signed-off-by: Roy Chiang <roychi@amazon.com>
    @roystchiang
    roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    3ea1e36 View commit details
    Browse the repository at this point in the history

  14. address comments 

    Signed-off-by: Roy Chiang <roychi@amazon.com>
    @roystchiang
    roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    1a7f65a View commit details
    Browse the repository at this point in the history

  15. fixed rebase issues 

    Signed-off-by: Roy Chiang <roychi@amazon.com>
    @roystchiang
    roystchiang committed Apr 6, 2022
    Configuration menu
    Copy the full SHA
    3cd209b View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2022

  1. fix tests 

    Signed-off-by: Roy Chiang <roychi@amazon.com>
    @roystchiang
    roystchiang committed Apr 7, 2022
    Configuration menu
    Copy the full SHA
    18a8e70 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into add-sharding

    @alanprot
    alanprot committed Apr 7, 2022
    Configuration menu
    Copy the full SHA
    137a313 View commit details
    Browse the repository at this point in the history