Commits on Nov 2, 2023

1. walk: Use unbounded channels …

   We originally switched to bounded channels for backpressure to fix sharkdp#918.
   However, bounded channels have a significant initialization overhead as
   they pre-allocate a fixed-size buffer for the messages.
   
   This implementation uses a different backpressure strategy: each thread
   gets a limited-size pool of WorkerResults.  When the size limit is hit,
   the sender thread has to wait for the receiver thread to handle a result
   from that pool and recycle it.
   
   Inspired by [snmalloc], results are recycled by sending the boxed result
   over a channel back to the thread that allocated it.  By allocating and
   freeing each WorkerResult from the same thread, allocator contention is
   reduced dramatically.  And since we now pass results by pointer instead
   of by value, message passing overhead is reduced as well.
   
   Fixes sharkdp#1408.
   
   [snmalloc]: https://github.com/microsoft/snmalloc

   

   tavianator committed Nov 2, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for d588971
   • Browse repository at this point

   Copy the full SHA

   d588971 View commit details

   Browse the repository at this point in the history

2. walk: Hold the Sender by reference in WorkerMsg

   

   tavianator committed Nov 2, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for 46967b8
   • Browse repository at this point

   Copy the full SHA

   46967b8 View commit details

   Browse the repository at this point in the history

3. walk: Switch to a semaphore for ResultPool

   

   tavianator committed Nov 2, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for 346afd1
   • Browse repository at this point

   Copy the full SHA

   346afd1 View commit details

   Browse the repository at this point in the history