raft: notify peers about startup to avoid vote/pre-vote

[dotnwat]

Cover letter

The PR #2071 provides a heuristic fix for leadership instability after
followers restart due to an interaction with leader rpc backoff rules and
follower timeouts (more information here #2048).

The heuristic works most of the time, but still depends on several moons to
align: failed heartbeats, prevote phase starting etc... This patch does away
with the heuristic and adds a new RPC for communicating an explicit startup
event to peers within a raft group via a new 'hello' rpc when a raft group
first starts. Peers now use this startup message to reset backoff (as it did
with the heuristic) but the reset is more precise because it is tied a specific
event rather than attempting to infer the scenario of interest.

Fixes: #4083

Release notes

Improvements

• Improved behavior for restarted raft groups that reduces election churn.

[jcsp]

I'm a bit concerned about the scale implications of implementing this at a per-group level: as we progress to having 100,000s of partitions per node, the queuing latency of emitting all these messages will be substantial. I think we mainly just need to know that the server is up, rather than that an individual group is starting up.

The implementation of consensus::hello is to call down through client_protocol and ultimately into connection_cache to reset the backoff on the restarted node, so if there are lots of partitions they'll all ultimately be resetting the same thing.

[dotnwat]

I'm a bit concerned about the scale implications of implementing this at a per-group level: as we progress to having 100,000s of partitions per node, the queuing latency of emitting all these messages will be substantial. I think we mainly just need to know that the server is up, rather than that an individual group is starting up.

Ahh, yeh good point. Debouncing the message should not be a problem, either remaining at a per-group level or bumping up to the controller/node level for the message.

The implementation of consensus::hello is to call down through client_protocol and ultimately into connection_cache to reset the backoff on the restarted node, so if there are lots of partitions they'll all ultimately be resetting the same thing.

I had naively been operating under the assumption that this back-off was per group. Now I'm wondering if this is more complicated.

At scale with 100K partitions I would expect the start-up time of any given raft group to be much longer than the default vote timeout relative to the start of the node or some other raft group. The effect of this would seem to be that an initial backoff reset on the leader wouldn't help for a raft group that starts up much later on the restarted follower.

I guess an option would be to wait until all of the raft groups on a restarted are recovered/ready before saying hello and starting the vote timeout. But that would seem to potentially be problematic for boot up time and later on when we want to "pause" a raft group.

---

EDIT.0: I guess once the backoff is reset then it won't backoff as long as something on the connection looks healthy, even if some raft group on the connection hasn't booted up yet. I was thinking that the backoff was initiated any time a raft group was unresponsive. I need to go look a little closer at this mechanism.

EDIT.1: Ok now I see the backoff is applied down at the reconnection transport level. I was working at the wrong level of abstraction. I think the question about how to handle raft groups at scale when they can't be treated equally in a gang scheduled way is still relevant, but for more appropriate at a later time.

[dotnwat]

@jcsp ok think this solution should be more palatable.

[mmaslankaprv]

This looks really good, clean and simple

[dotnwat]

Forced push:

1. Say hello and setup connections in the background
2. Include process start time in hello message

The goods https://github.com/redpanda-data/redpanda/compare/7ba472aa5fce41a0efc6c2b3d13a1046e438fbf6..6a4a1cee2581417d06ea3212c3bec221d02a1ab0

[jcsp]

👍