Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfault (various tests) #6391

Closed
jcsp opened this issue Sep 14, 2022 · 2 comments
Closed

Segfault (various tests) #6391

jcsp opened this issue Sep 14, 2022 · 2 comments
Labels
area/redpanda kind/bug Something isn't working

Comments

@jcsp
Copy link
Contributor

jcsp commented Sep 14, 2022

A segfault caught by AddressSanitizer but without a backtrace.

Two instances of this in last 24h, in revision 2158117

As a "Cluster membership did not stabilize" (node crashed while starting up):
https://buildkite.com/redpanda/redpanda/builds/15280#01833a7b-d21d-4a6e-9d5e-8f9505fda77c

TRACE 2022-09-14 07:22:00,410 [shard 0] kafka - request_context.h:160 - [172.16.16.27:42132] sending 3:metadata response {throttle_time_ms=0 brokers={{node_id=1 host=docker-rp-5 port=9092 rack={nullopt}}, {node_id=3 host=docker-rp-15 port=9092 rack={nullopt}}, {node_id=2 host=docker-rp-22 port=9092 rack={nullopt}}} cluster_id={redpanda.initializing} controller_id=1 topics={} cluster_authorized_operations=-2147483648}
TRACE 2022-09-14 07:22:00,412 [shard 0] rpc - conn_quota.cc:118 - put(172.16.16.27)
TRACE 2022-09-14 07:22:00,412 [shard 0] rpc - conn_quota.cc:130 - leaving put(172.16.16.27)
AddressSanitizer:DEADLYSIGNAL
=================================================================
==9378==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7fe32e24618f bp 0x7ffd5a46dc00 sp 0x7ffd5a46d960 T0)
==9378==The signal is caused by a READ memory access.
==9378==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer: nested bug in the same thread, aborting.

As a NodeCrash (test succeeded but one of the nodes died in the background):
https://buildkite.com/redpanda/redpanda/builds/15279#01833a71-a4f6-46de-8cb9-3207eec4586

TRACE 2022-09-14 06:31:45,490 [shard 0] rpc - conn_quota.cc:118 - put(172.16.16.27)
TRACE 2022-09-14 06:31:45,490 [shard 0] rpc - conn_quota.cc:130 - leaving put(172.16.16.27)
TRACE 2022-09-14 06:31:45,491 [shard 0] raft - [group_id:0, {redpanda/controller/0}] replicate_entries_stm.cc:96 - Sending append entries request {raft_group:{0}, commit_index:{24}, term:{1}, prev_log_index:{24}, prev_log_term:{1}, last_visible_index:{24}} to {id: {4}, revision: {0}}
TRACE 2022-09-14 06:31:45,495 [shard 0] raft - [group_id:0, {redpanda/controller/0}] replicate_entries_stm.cc:96 - Sending append entries request {raft_group:{0}, commit_index:{24}, term:{1}, prev_log_index:{24}, prev_log_term:{1}, last_visible_index:{24}} to {id: {2}, revision: {0}}
AddressSanitizer:DEADLYSIGNAL
=================================================================
==1741==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7fefc41b418f bp 0x7fffb95cc880 sp 0x7fffb95cc5e0 T0)
==1741==The signal is caused by a READ memory access.
==1741==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer: nested bug in the same thread, aborting.

No other failures like those in the past week, which would ordinarily point to something merging recently having caused the problem, but not a lot of substantial PRs merged in the last day.

@jcsp jcsp added kind/bug Something isn't working area/redpanda labels Sep 14, 2022
@jcsp
Copy link
Contributor Author

jcsp commented Sep 15, 2022

Maybe same as #5575

@jcsp
Copy link
Contributor Author

jcsp commented Sep 15, 2022

I don't see anything to distinguish this from #5575, so closing as a duplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/redpanda kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant