Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve memory efficiency of many OptimisticTransactionDBs #11439

Closed
wants to merge 4 commits into from

Commits on May 11, 2023

  1. Improve memory efficiency of many OptimisticTransactionDBs

    Summary: Currently it's easy to use a ton of memory with many small
    OptimisticTransactionDB instances, because each one by default allocates
    (by default) a million mutexes (40 bytes each on my compiler) for
    validating transactions. It even puts a lot of pressure on the allocator
    by allocating each one individually!
    
    In this change:
    * Create a new object and option that enables sharing these buckets of
    mutexes between instances. This is generally good for load balancing
    potential contention as various DBs become hotter or colder with txn
    writes. About the only cases where this sharing wouldn't make sense
    (e.g. each DB usually written by one thread) are cases that would be
    better off with OccValidationPolicy::kValidateSerial which doesn't use
    the buckets anyway.
    * Allocate the mutexes in a contiguous array, for efficiency
    * Add an option to ensure the mutexes are cache-aligned. In several
    other places we use cache-aligned mutexes but OptimisticTransactionDB
    historically does not. It should be a space-time trade-off the user can
    choose.
    * Provide some visibility into the memory used by the mutex buckets with
    an ApproximateMemoryUsage() function (also used in unit testing)
    * Share code with other users of "striped" mutexes, appropriate
    refactoring for customization & efficiency (e.g. using FastRange instead
    of modulus)
    
    Test Plan: unit tests added. Ran sized-up versions of stress test in unit
    test, including a before-and-after performance test showing no
    consistent difference. (NOTE: OptimisticTransactionDB not currently covered
    by db_stress!)
    pdillinger committed May 11, 2023
    Configuration menu
    Copy the full SHA
    f7a3f72 View commit details
    Browse the repository at this point in the history
  2. Fixes

    pdillinger committed May 11, 2023
    Configuration menu
    Copy the full SHA
    d2dc631 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2023

  1. Configuration menu
    Copy the full SHA
    8e0022e View commit details
    Browse the repository at this point in the history
  2. Some polish from feedback

    pdillinger committed May 23, 2023
    Configuration menu
    Copy the full SHA
    db40f19 View commit details
    Browse the repository at this point in the history