Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: checkpoint offset_translator more frequently when using many tiny partitions #4600

Closed
jcsp opened this issue May 6, 2022 · 0 comments
Assignees
Labels
area/storage kind/enhance New feature or request

Comments

@jcsp
Copy link
Contributor

jcsp commented May 6, 2022

Currently offset_translator::maybe_checkpoint uses a fixed 64MiB threshold to decide whether to write a checkpoint.

If I have e.g. 40k partitions on a node with 2TB of storage, the most any partition ever stores is 50MB, so the threshold is never reached. This results in a large amount of read IO on restart: currently ManyPartitionsTest has to wait as long for node startup as it spent writing the data in to begin with (many minutes).

  • We could make this configurable, similar to the falloc step size.
  • We could use our knowledge of the disk size and partition count to dynamically select a size that makes sure we are never reading more than a certain fraction of the disk size on startup.
  • We could keep a global count of the number of un-checkpointed bytes across all partitions, and trigger checkpoints based on that -- this would be the most direct way of bounding the amount of data that redpanda has to replay on startup, at the cost of more coordination.

It may be that the solution to this can also be used to drive a dynamic falloc step size (this has a similar issue where the default 32MiB threshold doesn't make much sense for systems with huge partition counts).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/storage kind/enhance New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant