Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: per-shard limit on memory for spill_key_index #5722

Merged
merged 7 commits into from
Aug 5, 2022
Merged

storage: per-shard limit on memory for spill_key_index #5722

merged 7 commits into from
Aug 5, 2022

Commits on Aug 5, 2022

  1. storage: add compaction index memory semaphore to storage_resources 

    This is an additional bound, on top of the existing _max_mem
in spill_key_index: it will now also avoid using more memory
in total per shard.

This commit uses a static total of 128MB, which enables up to 256
compacted partitions to use the same 512kiB per-partition allowance
that they were using before, then as the partition count gets higher
it starts throttling back, although each partition always gets to
use at least 32kib memory each, to avoid a pathological case where
they spill on every key add.
    @jcsp
    jcsp committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    823d672 View commit details
    Browse the repository at this point in the history

  2. storage: respect shard-wide memory limit in spill_key_index 

    Fixes redpanda-data#4645
    @jcsp
    jcsp committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    a60f1f2 View commit details
    Browse the repository at this point in the history

  3. config: add storage_compaction_index_memory 

    This property controls the new bound on per-shard
memory used for compaction indices at scale.
    @jcsp
    jcsp committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    3244040 View commit details
    Browse the repository at this point in the history

  4. storage: make compaction memory limit configurable 

    This will probably be rarely changed in practice, but
it mitigates the risk that we have people using compaction
at high scale and experiencing performance issues from
index spills happening more often than they hoped.
    @jcsp
    jcsp committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    53fcd5d View commit details
    Browse the repository at this point in the history

  5. bytes: publicly expose the inline buffer size of bytes 

    This enables other code to accurately account for memory use.
    @jcsp
    jcsp committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    c1b1e9c View commit details
    Browse the repository at this point in the history

  6. storage: more accurate accounting in spill_key_index 

    Previously, we ignored:
- inline buffer in `bytes` which affects the actual memory utilization
- the map value.

Actual memory footprint still depends on how these sizes
get rounded up to allocator boundaries, but this is an improvement.
    @jcsp
    jcsp committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    87437ff View commit details
    Browse the repository at this point in the history

  7. storage: make memory units release safer 

    While in general the accounting is robust, this is a little
fragile in error paths (or when the code is changed in future),
as it is an exception-generating condition to release more
units from a sem_units than you took: a double release
could perhaps occur if we released before spill(), then
there was an exception in spill() that caused us to iterate.
    @jcsp
    jcsp committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    5481557 View commit details
    Browse the repository at this point in the history