cluster: many redundant config status log messages may be written if follower is not seeing updates promptly #4923
Assignees
Labels
Comments
jcsp
added a commit
to jcsp/redpanda
that referenced
this issue
May 25, 2022
If a follower isn't seeing controller log updates promptly, it may issue many set_status RPCs while it's waiting. The controller leader should not turn all of these into log writes: if the status of the node already matches what it is reporting, then do not write anything. Fixes redpanda-data#4923
jcsp
added a commit
to jcsp/redpanda
that referenced
this issue
May 25, 2022
On a healthy system, we do want to send set_status RPCs as soon as we're ready. However, if the controller log updates are not being seen promptly, this would lead to the follower spamming the controller leader with very many set_status RPCs in a tight loop. Nodes will still send their status immediately when a config change occurs: this change only effects the behaviour if _another_ config change occurs while it is reporting status from the first change: in this case the follower will wait 5 seconds before sending its next status RPC. Related redpanda-data#4923
vbotbuildovich
pushed a commit
to vbotbuildovich/redpanda
that referenced
this issue
May 27, 2022
If a follower isn't seeing controller log updates promptly, it may issue many set_status RPCs while it's waiting. The controller leader should not turn all of these into log writes: if the status of the node already matches what it is reporting, then do not write anything. Fixes redpanda-data#4923 (cherry picked from commit 0a2b991)
vbotbuildovich
pushed a commit
to vbotbuildovich/redpanda
that referenced
this issue
May 27, 2022
On a healthy system, we do want to send set_status RPCs as soon as we're ready. However, if the controller log updates are not being seen promptly, this would lead to the follower spamming the controller leader with very many set_status RPCs in a tight loop. Nodes will still send their status immediately when a config change occurs: this change only effects the behaviour if _another_ config change occurs while it is reporting status from the first change: in this case the follower will wait 5 seconds before sending its next status RPC. Related redpanda-data#4923 (cherry picked from commit 3b1e56e)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
reconcile_status emits a set_status rpc to the leader any time it sees that the contents of its status table do not match its own self-described status. If it is not receiving updates from the leader promptly, it may emit many messages like this in a loop.
The leader should check if any set_status calls from a follower are no-ops, and drop them, rather than writing multiple identical messages to the controller log.
The text was updated successfully, but these errors were encountered: