etcdserver: Implement running defrag if freeable space will exceed provided threshold (on boot)

[serathius]

Defragment is an expensive operation to execute when serving traffic as requires locking database for multiple seconds. Instead of defragmenting during operation of Etcd we can mitigate this cost by moving this process to server bootstrap. This also has benefit of reducing maintenance cost for users that didn't setup full automation to trigger defrag periodically, but still do some other operations like upgrades.

This PR adds new --experimental-bootstrap-defrag-threshold-megabytes flag to etcdserver that allows users to set a disk size in megabytes. During bootstrap if disk size that would be freed by defrag is greater then threshold set, etcdserver will automatically execute defrag before starting to serve traffic.

@ptabor

[ptabor]

Thank you. This will be very helpful for an infrequent defragmentation that does not impacts tile-latency of requests served by nodes in clusters. If member is updated or is restared (e.g in response to NO_SPACE alarm), it can be configured to perform some autohealing action, like 'defrag'.

This mitigation was discussed on community meeting on July 30, 2020 in response to
kubernetes/kubernetes#93280.

---

Marek: Please add a test (e.g. e2e) to at least guarantee that setting this flag does not crashes the server.

[gyuho]

This will be very helpful for an infrequent defragmentation that does not impacts tile-latency of requests served by nodes in clusters.

Yes, defragging live node is quite disruptive, even causing error too many requests.

Defrag on bootstrap seems safe, on the other hand.

[serathius]

Added e2e test

[hexfusion]

Overall this makes sense, while this can fairly trivially be added as an init container it does have value as a server runtime.

[hexfusion]

Just a thought do we have any concerns with race condition and file locks? I am thinking about OS that does not support flock? Should we consider gating this on a supported arch(s)?

[ptabor]

Just a thought do we have any concerns with race condition and file locks? I am thinking about OS that does not support flock? Should we consider gating this on a supported arch(s)?

What scenario do you envision ?

My reasoning: I assume we are running this before 'concurrent' code within etcd is initialized. We could even move it before ci.SetBackend(be) to make it even more guaranteed. So whetever concurrent process would access the file directly it would either disrupt 'backup' or the main's etcd operations.

[hexfusion]

Just a thought do we have any concerns with race condition and file locks? I am thinking about OS that does not support flock? Should we consider gating this on a supported arch(s)?

What scenario do you envision ?

My reasoning: I assume we are running this before 'concurrent' code within etcd is initialized. We could even move it before ci.SetBackend(be) to make it even more guaranteed. So whetever concurrent process would access the file directly it would either disrupt 'backup' or the main's etcd operations.

Just high level I have not dug into this yet but if this code happens before listeners would anything block defrag from attempting to run against an already running etcd process?

• etcd process running (graceful term)
• new process starts
• file lock race?

[xiang90]

We probably also should log here for the non-skipping case.

[ptabor]

I used to think so, but be.Defrag internally has pretty decent logging already.

[xiang90]

@serathius

The reclamation of disk space can be important. But I guess the most important role is to rearrange the on-disk pages and reduce the size of freelist. Thus it can also help with write throughput/latency (reduce random writes). So I am not sure if the size factor is the most important one to config. We could check how many small holes are there and let users config it. But I do not really want to make the flag configuration too complicated either.

[ptabor]

@xiang90 I assume that size is the good proxy to number of pages being cleaned up (size/4096), so number of entries being deleted from the free-pages list. From my perspective the goals are:

• reducing size of snapshots being sent during node recovery
• reducing size of database being mmaped/mlocked (so RAM consumption)
  mostly in situations that DB had temporary spike of 'usage' but later it shrank / got compacted.

[xiang90]

@ptabor

1. DB tends to just grow in size over time. It is also true for Kubernetes use cases where the cluster normally just keep growing.
2. When etcd does release some pages, it is probably because of compaction then it will grow back soon.

Because of these two, defrag only at the boot time might not be super effective on space reclaim or RAM space saving.

If we really want to reduce the snapshot sending size, we'd better compact it before sending (do not send empty pages as it is now?).

If we really want to save RAM size, we need to be smarter on mmap control (do not map large holes) and page allocations.

Most of the issues we see in production for large cluster are huge freelist (increase search time) and the random writes (multiple non-leaf branches needs to be updates as well comparing to seq writes).

[xiang90]

I assume that size is the good proxy to number of pages being cleaned up (size/4096), so number of entries being deleted from the free-pages list.

Not entirely true when there are a lot of small holes vs big holes. The item in the freelist should be span instead of items? But I guess most of the time it can be a good indicator?

[ptabor]

xiang90

• We are thinking about in-place defrag in the bolt layer that will take care of long-running servers fragmentation. That one would mark the 'last' pages as 'dirty' and let them be moved to the beginning of the file in background without the need to 'block' all ongoing RW & RO transactions as the current 'deep' defrag do.
• The full defrag on boot would be additional level of protection.