fix ccache memory leak

[robert-milan]

might help with #1057 or #1058

for testing and results see: https://gist.github.com/robert-milan/20d87f10456b23f297c9eb9e7d730190

[replay]

Those results look really great. Looking forward to see how this behaves in a prod env

[Dieterbe]

for the record, this reverts cce8a97

[Dieterbe]

i want a bit more time to dig into this

[woodsaj]

@Dieterbe the memory leak is a pretty big issue for production systems. I dont see any reason why this change should be blocked.

[DanCech]

Looking at the benchmarks on cce8a97 I don't see that there were any meaningful gains. Unless I'm reading the benchmarks wrong I'd say go ahead and merge this, especially since it's just going back to previously-working code.

[Dieterbe]

ok that's fair. especially since we now track time we block on the queue.
i don't see how this would fix #1058 though as in that issue cache limit is not reached and there are no evictions.
While we're not sure this will fix #1057 either, it should at least have a positive impact on memory in some cases. (it will allow to reclaim memory earlier than before and may lead to an overall reduced heap size)

PS @robert-milan please make it a habit to describe the problem and the solution in either the commit message or in the PR (if you put it in the former, it'll automatically go into the PR description as well if there's only 1 commin in the PR). personally i'm a fan of putting it in both.

[Dieterbe]

so basically:

• previous behavior: range used 1 slice, memory could only be freed once all chunks in the range were evicted
• new behavior: allocate chunks individually, allowing to free memory earlier / more granularly, as chunks for the range are being evicted. the payoff happens when chunks that were queried in the same range don't expire at the same time.

tradeoffs:

• use a bit more cpu, may slow down requests, but probably negligible
• more live pointers, may slow down GC (and hence requests)

we choose not do synthetic benchmarks to look into these trade-offs. with some luck our prod enviroments are not too noisy so we'll be able to tell from those.