Fix a rare panic during consumer group rebalance #791
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently, there is a sync.WaitGroup that is used for process accounting with go routines started by Generation.Start. It is used when a generation is ending to ensure that all of the Start-ed functions exit before the next generation can start. There is an edge case that can cause panic as written up in #786 due to unsafe use of the WaitGroup. Basically, it is possible that close() is already in process and is waiting on WaitGroup.Wait() when Start() comes in and calls WaitGroup.Add(1). The contract of Start() is that the provided function is to be run. Code in the wild may depend on this behavior, so it's not an option to return from Start() without runing the provided function. Accordingly, this PR updates the code to coordinate between close() and Start() such that the panic case is no longer possible while preserving the existing contract. It uses channels and a mutex in order to create two cases: the normal case where the generation is alive and the edge case where the generation has already ended.
Co-authored-by: Achille <achille@segment.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, there is a
sync.WaitGroupthat is used for process accountingwith go routines started by
Generation.Start. It is used when ageneration is ending to ensure that all of the Start-ed functions exit
before the next generation can start. There is an edge case that can
cause panic as written up in #786 due to unsafe use of the WaitGroup.
Basically, it is possible that
close()is already in process and iswaiting on
WaitGroup.Wait()whenStart()comes in and callsWaitGroup.Add(1).The contract of
Start()is that the provided function is to be run.Code in the wild may depend on this behavior, so it's not an option to
return from
Start()without running the provided function.Accordingly, this PR updates the code to coordinate between close() and
Start()such that the panic case is no longer possible while preservingthe existing contract. It uses channels and a mutex in order to create
two cases: the normal case where the generation is alive and the edge
case where the generation has already ended.