-
Notifications
You must be signed in to change notification settings - Fork 579
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
storage: crash at offset_translator_state.cc:88) 'data_offset >= min_data_offset'
#3323
Comments
~file_data_source_impl(): Assertion
_reads_in_progress == 0' failed.`offset_translator_state.cc:88) 'data_offset >= min_data_offset'
Backtrace of the assert at node5:
So I guess it is not directly related to shadow indexing - I think what happened is that prefix truncation happened right after we read some data from the local start of the log, but before we got the aborted transactions for that range of offsets. And the fix is to hold onto segments that we read from for a little longer in the fetch session. |
Just reproduced the |
|
Just confirmed both assertions are reproducible with 21.11.3-beta2. Agree with Alexey that these are distinct. On subsequent runs they happened independent of one another, so I think it was just coincidence that in my original test run they happened about the same time. I'll open a separate ticket. |
offset_translator_state.cc:88) 'data_offset >= min_data_offset'
offset_translator_state.cc:88) 'data_offset >= min_data_offset'
Just reproduced this on a non-SI topic -- 4/6 nodes asserted out in about 20 minutes of runtime. The case of an extremely short retention window is likely to be more common in SI systems, but this appears to be a preexisting bug. |
The problem arises because log eviction proceeds in the following stages:
The assert fires because between 2 and 3 we can still have readers for the prefix-truncated offset range but offset translation information is already gone. Discussed mitigation strategies:
|
The linked pr #3372 implements the last strategy (actually delete the segments immediately after writing the raft snapshot instead of relying on the next iteration of the log eviction loop). |
Version
21.11.1
Node 5 crashed first, then nodes 3+4 crashed few minutes later.
Summary of crashes (including one line before for the timestamp)
node3.log
node4.log
node5.log
Workload leading up to crash:
Topic details:
The text was updated successfully, but these errors were encountered: