vpa-{updater,recommender}: Add support for leader election

[ialidzhikov]

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds support for leader election in vpa-updater and vpa-recommender.

Which issue(s) this PR fixes:

Fixes #6846

Special notes for your reviewer:

This PR is strongly inspired by how leader election is built for the cluster-autoscaler.

Does this PR introduce a user-facing change?

vpa-updater and vpa-recommender do now support leader election.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[ialidzhikov]

/cc @kwiesmueller

[kwiesmueller]

/assign @voelzmo @raywainman
Could you have a look please.

[adrianmoisey]

I build and tested this locally, and it all works as expected. Very nice, this is something I've also been wanting.

[voelzmo]

/lgtm

Thanks! Great catch of taking care of the /health-check endpoint for the passive replicas!
I think it is also important that the recommender gets killed and re-created when it loses leadership, so we don't end up in a situation where oudated in-memory aggregations are taken into account. The way it is implemented here, we should always end up using a fresh initialization from history.

[raywainman]

/lgtm

[voelzmo]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ialidzhikov, voelzmo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• vertical-pod-autoscaler/OWNERS [voelzmo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment