Scale+compaction tests + make spill_key_index memory size dynamic

[jcsp]

Followup from #5150 (comment) -- in that PR, spill_key_index got a reference to a storage_resources but not yet any logic to choose its max_mem parameter based on the number of partitions in the system (it still uses a fixed 512kiB).

I didn't do it yet because I didn't have a nice scale test with compaction enabled to validate the new settings, but it's a next step.