-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test/raft: segment appender concurrent flush and write sanitizer error #3841
Comments
@andrewhsu if you scroll up further in the log you'll see:
Looks like it is coming from segment appender
We made the segment appender to be safe for concurrent flush and append, but this error is originating from the file handler sanitizer wrapper. It's not clear if this is a false positive, or it really is an invalid use of the file handler. |
test_raft_rpunit
CI failure triage: this is very rare, but I bet it is a genuine bug. Need to read the code carefully. |
agree. this needs someone to sent down expecting a full day of analyzing this. |
Ok looks like there is a fair amount of red herrings in this issue :) Red herring 1:
Red herring 2: concurrent flush and append have nothing to do with the test failure! Confusingly, these "flush() called concurrently" messages contain stack traces and that makes them look similar to assertion failure messages, but really they are just warnings and don't affect any test outcomes. Moreover, the messages (there are several) appear after the test failure. So either there are two separate issues or the causality is in reverse. |
So I managed to reproduce concurrent flush+write with some sleeps, it is indeed allowed by
I can't see an easy fix without sacrificing concurrency. But maybe concurrent flush/dma_writes are not that bad? Not sure. cc @dotnwat |
@ztlpn nice analysis. when i added background flushing into the segment appender i wanted to allow appends and flushes to run concurrently--that was a key part of the optimization by which raft would be able to pipeline i/o. however, i definitely wasn't considering that the file handles that the segment appender was using would be wrapped in the sanitizer which is catching the is the sanitizer too conservative for use in the segment appender? if the following are true, then I think we could probably relax the rule:
|
i don't think this is a proper bug in the segment appender (or a violation of the appender's api rules). if we start getting CI failures because of this we should look again and fixing it but it's clearly very rare. |
this one, #5433, though, we might want to investigate |
from nightly test job on
dev
branch: https://buildkite.com/redpanda/redpanda/builds/7400#23a4a0e4-52db-44b8-ada4-bd2cd6982708/6-5332earlier in the same logs:
The text was updated successfully, but these errors were encountered: