Provide way of disabling/configuring Leader Election Provider on Fleet Enrolled Elastic Agents

[BenB196]

Describe the enhancement:

The Elastic Agent's Leader Election Provider can only be configured when using a standalone agent currently. It would be nice to be able to be able to configure/disable this provider on Fleet enrolled Elastic Agents

Describe a specific use case for the enhancement or feature:

I recently ran into an issue where I was deploying two Daemonsets of the Elastic Agent, see elastic/kibana#143905 (comment), one odd side-effect was that we noticed that things relying on leader election functionality would randomly break. What we found was that both Daemonsets were using leader election and contending over the same lease name, and only one Daemonset could hold the lease at a time.

(For now the "hacky" work around was just to remove the k8s service account permissions required for Leader election to even work on the one Daemonset that doesn't need it).

What is the definition of done?

In my case, only one of the Daemonsets really needs leader election, so just being able to disable the leader election provider on the other would suffice. But probably a more complete solution would be to have a way to configure all of the provider settings via Fleet, not just disabling it.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

It should be possible to disable the leader election provider by editing the elastic-agent.yml configuration the agent starts up with to contain providers.kubernetes_leaderelection.enabled: false. This enabled flag is only read when the agent first starts up. @gizas this is probably something we should document with the example for this you have.

Long term the most convenient solution is definitely to expose the providers configuration in Fleet.

[amitkanfer]

@nimarezainia , another one to the the list of knobs ^^

[cmacknz]

I am going to close this as a duplicate of #4145, which is more general and wants to allow controlling all of the providers and ideally anything in the initial agent configuration.

[gizas]

Just adding here that relevant documentation page for disabling the leader election provider, has been available: https://www.elastic.co/guide/en/fleet/current/advanced-kubernetes-managed-by-fleet.html

[strawgate]

It might be important to note that there is some weird behavior when following those instructions including agent exiting with an error the first time it's started with that configuration