Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Partition autobalancer full disk test #5839

Merged
merged 8 commits into from
Aug 5, 2022
Merged

Partition autobalancer full disk test #5839

merged 8 commits into from
Aug 5, 2022

Commits on Aug 5, 2022

  1. c/local_monitor: add ability to set mock disk size via env var

    @ztlpn
    ztlpn committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    4bd1500 View commit details
    Browse the repository at this point in the history

  2. tests/end_to_end: add ability to set environment for redpanda

    @ztlpn
    ztlpn committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    5eff865 View commit details
    Browse the repository at this point in the history

  3. tests: fix off-by-1 error in await_minimum_produced_records

    @ztlpn
    ztlpn committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    849b684 View commit details
    Browse the repository at this point in the history

  4. c/partition_balancer: fix calculating node_released_disk_size map

    @ztlpn
    ztlpn committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    8f9e4b8 View commit details
    Browse the repository at this point in the history

  5. c/partition_balancer: introduce hard_max_disk_usage_ratio 

    When moving partitions away from unavailable nodes it is desireable to
violate disk ratio that we use to determine when we need to move nodes
away from a node - this will allow us to save data if we have some free
space left. But we shouldn't go to 100%, so we use the
storage_space_alert_free_threshold_percent config value to determine the
hard limit.
    @ztlpn
    ztlpn committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    5f65dd0 View commit details
    Browse the repository at this point in the history

  6. c/partition_balancer: take already planned reallocations into account 

    Previously there was a following bug with moving partitions away from
nodes with full disks: because the planner tries to move all replicas
in a set from "bad" nodes in one reallocation, if we don't take into
account previous reallocations, we might consider that some node is
still full even if we have already planned several moves away from it.
This lead to excessive movements being planned: when a full node became
almost empty after executing a batch of reallocations. In this commit we
use "final" node disk usage (after all reallocations are finished) where
appropriate.
    @ztlpn
    ztlpn committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    dd9aa7c View commit details
    Browse the repository at this point in the history

  7. tests/partition_balancer: full nodes test

    @ztlpn
    ztlpn committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    d58ecbc View commit details
    Browse the repository at this point in the history

  8. c/partition_balancer: rename planned_movement_disk_size

    @ztlpn
    ztlpn committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    a484b37 View commit details
    Browse the repository at this point in the history