Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: test_migrating_consume_offsets.failurer: AddressSanitizer: heap-use-after-free #5261

Closed
dotnwat opened this issue Jun 28, 2022 · 8 comments · Fixed by #6738
Closed

test: test_migrating_consume_offsets.failurer: AddressSanitizer: heap-use-after-free #5261

dotnwat opened this issue Jun 28, 2022 · 8 comments · Fixed by #6738
Assignees
@dlex
Labels
area/redpanda area/tests ci-disabled-test ci-failure kind/bug Something isn't working

Comments

@dotnwat
Copy link
Member

dotnwat commented Jun 28, 2022

https://buildkite.com/redpanda/redpanda/builds/11804#0181ab92-27a7-4657-864e-2ac060cde87a/1575-8093

test_id:    rptest.tests.consumer_offsets_migration_test.ConsumerOffsetsMigrationTest.test_migrating_consume_offsets.failures=True.cpus=1
--
  | status:     FAIL
  | run time:   4 minutes 1.639 seconds
  |  
  |  
  | <BadLogLines nodes=docker-rp-3(1) example="==1744==ERROR: AddressSanitizer: heap-use-after-free on address 0x618000005b80 at pc 0x5624f699dece bp 0x7fff519e7b60 sp 0x7fff519e7b58">
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
  | return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  | File "/root/tests/rptest/services/cluster.py", line 48, in wrapped
  | self.redpanda.raise_on_bad_logs(allow_list=log_allow_list)
  | File "/root/tests/rptest/services/redpanda.py", line 1078, in raise_on_bad_logs
  | raise BadLogLines(bad_lines)
  | rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-3(1) example="==1744==ERROR: AddressSanitizer: heap-use-after-free on address 0x618000005b80 at pc 0x5624f699dece bp 0x7fff519e7b60 sp 0x7fff519e7b58">
  |  
@dotnwat dotnwat added kind/bug Something isn't working ci-failure labels Jun 28, 2022
@dlex
Copy link
Contributor

dlex commented Aug 5, 2022

ASAN has detected heap-use-after-free during RP shutdown, when many services including raft::group_manager have been stopped and destroyed, but reactor still runs. Adopted stack trace:

std::exception_ptr::operator bool()
ss::basic_semaphore<>::signal(unsigned long) at 'if (_ex)'
ss::semaphore_units<>::~semaphore_units() at '_sem->signal(_n);'
std::vector<ss::semaphore_units<>>::~vector()
rpc::transport::entry::~entry()
rpc::transport::fail_outstanding_futures() at '_requests_queue.clear();'
net::base_transport::stop()
rpc::reconnect_transport::stop()::$_1::operator()() const

Data structure containment:

void* exception_ptr::_M_exception_object;  <-- the invalid pointer
std::exception_ptr basic_semaphore::_ex;
ss:semaphore<>* ss::semaphore_units::_sem;  <-- may be broken in first place
ss::foreign_ptr<ss::lw_shared_ptr<std::vector<ss::semaphore_units<>>>> rpc::transport::entry::resource_units;
absl::btree_map<sequence_t, std::unique_ptr<transport::entry>> rpc::transport::_requests_queue;

ss::foreign_ptr is destroyed synchronously because the dtor is called in the same shard where the ptr was constructed.

Allocation and deallocation stacks of the accessed memory block point to raft::group_manager which doesn't make sense - it it not the object that would store the exception_object the semaphore points to. Most probably it is the semaphore object itself who had been destroyed before it was used in this case. Since rpc::transport::entry stores semaphore units supplied by an RPC caller, it is not practical to try to determine what was the missing semaphore object in this case.

A possible strategy to resolve this case is to let it go and hope that the next time ASAN will trigger the error while accessing the semaphore object memory (the basic_semaphore::_ex member), and then the alloc/free stack traces will point to the missing semaphore.

Another option is to increase the probability for ASAN to triggier the error at the right place is to add the code to ss::semaphore_units::~semaphore_units() that will explicitly read sizeof(ss::semaphore<>) bytes pointed by `ss::semaphore_units::_sem' in debug build.

@dlex
Copy link
Contributor

dlex commented Aug 5, 2022

Triage bottomline: extremely likely an RP bug, medium or low severity because it apparenlty happens during shutdowm only and because it is rare.

@dlex dlex mentioned this issue Aug 11, 2022
Closed
@dotnwat dotnwat mentioned this issue Aug 15, 2022
Merged
5 tasks
@ztlpn
Copy link
Contributor

ztlpn commented Aug 16, 2022

The same in PartitionMoveInterruption.test_cancelling_partition_move_x_core

@ztlpn
Copy link
Contributor

ztlpn commented Aug 18, 2022

one more https://buildkite.com/redpanda/redpanda/builds/14283#0182af58-13bd-411e-bd33-50cf3bd0764a

@mmaslankaprv
Copy link
Member 

[Backtrace #0]
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x345c53fd: std::exception_ptr::operator bool() const at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3488f877: seastar::basic_semaphore<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::signal(unsigned long) at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3488f7ae: seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::return_all() at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3488f658: seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::~semaphore_units() at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x35534170: void std::__1::__destroy_at<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, 0>(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3553410c: void std::__1::destroy_at<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, 0>(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x355340e0: void std::__1::allocator_traits<std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::destroy<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, void, void>(std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >&, seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3677a10f: std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::__base_destruct_at_end(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at admin_server.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x36779ff9: std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::__clear() at admin_server.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x36779f1d: std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::~vector() at admin_server.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x36779e8c: seastar::lw_shared_ptr_no_esft<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >::~lw_shared_ptr_no_esft() at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x36779e23: seastar::internal::lw_shared_ptr_accessors_no_esft<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >::dispose(seastar::lw_shared_ptr_counter_base*) at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x36779c7f: seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >::operator=(seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >&&) at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x36774ac5: seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >::operator=(decltype(nullptr)) at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x36774305: seastar::foreign_ptr<seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > > >::destroy_on(seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >, unsigned int) at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x36773e71: seastar::foreign_ptr<seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > > >::destroy(seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >, unsigned int) at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x367737bb: seastar::foreign_ptr<seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > > >::~foreign_ptr() at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e36ac: rpc::transport::entry::~entry() at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e356c: std::__1::default_delete<rpc::transport::entry>::operator()(rpc::transport::entry*) const at transport.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e3344: std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >::reset(rpc::transport::entry*) at transport.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e314c: std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >::~unique_ptr() at transport.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e30ec: std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >::~pair() at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e3090: void std::__1::__destroy_at<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >, 0>(std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at transport.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e302c: void std::__1::destroy_at<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >, 0>(std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at transport.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e3000: void std::__1::allocator_traits<std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > > >::destroy<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >, void, void>(std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >&, std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e2fca: void absl::lts_20210324::container_internal::map_slot_policy<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >::destroy<std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > > >(std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >*, absl::lts_20210324::container_internal::map_slot_type<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e2af4: absl::lts_20210324::container_internal::common_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false, absl::lts_20210324::container_internal::map_slot_policy<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >::destroy(std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >*, absl::lts_20210324::container_internal::map_slot_type<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e1924: absl::lts_20210324::container_internal::btree_node<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> >::value_destroy_n(unsigned char, unsigned char, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >*) at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e0e63: absl::lts_20210324::container_internal::btree_node<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> >::clear_and_delete(absl::lts_20210324::container_internal::btree_node<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> >*, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >*) at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426e0c12: absl::lts_20210324::container_internal::btree<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> >::clear() at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426db188: absl::lts_20210324::container_internal::btree_container<absl::lts_20210324::container_internal::btree<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> > >::clear() at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x426af4b7: rpc::transport::fail_outstanding_futures() at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x43207e9c: net::base_transport::stop() at ??:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4276d968: rpc::reconnect_transport::stop()::$_1::operator()() const at reconnect_transport.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4276d7e3: seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<rpc::reconnect_transport::stop()::$_1&>(rpc::reconnect_transport::stop()::$_1&) at reconnect_transport.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4276d76f: auto seastar::futurize_invoke<rpc::reconnect_transport::stop()::$_1&>(rpc::reconnect_transport::stop()::$_1&) at reconnect_transport.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4276d6bd: _ZZN7seastar6futureIvE4thenIZN3rpc19reconnect_transport4stopEvE3$_1S1_EET0_OT_ENUlDpOT_E_clIJEEEDaSB_ at reconnect_transport.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4276d62b: _ZN7seastar20noncopyable_functionIFNS_6futureIvEEvEE17direct_vtable_forIZNS2_4thenIZN3rpc19reconnect_transport4stopEvE3$_1S2_EET0_OT_EUlDpOT_E_E4callEPKS4_ at reconnect_transport.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3480e16b: seastar::noncopyable_function<seastar::future<void> ()>::operator()() const at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3480dff3: decltype ((static_cast<seastar::noncopyable_function<seastar::future<void> ()>&>({parm#1}))()) std::__1::__invoke<seastar::noncopyable_function<seastar::future<void> ()>&>(seastar::noncopyable_function<seastar::future<void> ()>&) at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3480df1f: std::__1::invoke_result<seastar::noncopyable_function<seastar::future<void> ()>&>::type std::__1::invoke<seastar::noncopyable_function<seastar::future<void> ()>&>(seastar::noncopyable_function<seastar::future<void> ()>&) at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3480dd13: auto seastar::internal::future_invoke<seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate>(seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate&&) at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3480dc5c: seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}::operator()() const at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3480d9ec: void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&&) at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3480d7e6: seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3480d2fa: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose() at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4562275d: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at reactor.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x45628776: seastar::reactor::run_some_tasks() at reactor.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4562d2cf: seastar::reactor::do_run() at reactor.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x45794398: seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91::operator()() const at reactor.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x45793050: decltype ((static_cast<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&>({parm#1}))()) std::__1::__invoke<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&) at reactor.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x45792f0c: void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&) at reactor.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x45792e74: std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91>, void ()>::operator()() at reactor.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4578dea0: std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91>, void ()>::operator()() at reactor.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x346b4b15: std::__1::__function::__value_func<void ()>::operator()() const at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x345c66c8: std::__1::function<void ()>::operator()() const at application.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x454bdbb8: seastar::posix_thread::start_routine(void*) at posix.cc:?
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x9298: ?? ??:0
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x1006a2: ?? ??:0

@mmaslankaprv mmaslankaprv mentioned this issue Aug 22, 2022
Closed
@bharathv
Copy link
Contributor

#5730 fixes this? Seems the issue is consensus::_op_lock getting destroyed before a transport is releasing its units? With the patch, we release the units sooner.

@mmaslankaprv
Copy link
Member 

DEBUG 2022-08-22 06:38:43,929 [shard 0] rpc - transport.cc:237 - could not parse header from server: {host: docker-rp-24, port: 33145}
[Backtrace #0]
 0x345f54fd: std::exception_ptr::operator bool() const at application.cc:?
 0x348bf977: seastar::basic_semaphore<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::signal(unsigned long) at ??:?
 0x348bf8ae: seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::return_all() at ??:?
 0x348bf758: seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::~semaphore_units() at ??:?
 0x35564270: void std::__1::__destroy_at<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, 0>(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at application.cc:?
 0x3556420c: void std::__1::destroy_at<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, 0>(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at application.cc:?
 0x355641e0: void std::__1::allocator_traits<std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::destroy<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, void, void>(std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >&, seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at ??:?
 0x367aa20f: std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::__base_destruct_at_end(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at admin_server.cc:?
 0x367aa0f9: std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::__clear() at admin_server.cc:?
 0x367aa01d: std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::~vector() at admin_server.cc:?
 0x367a9f8c: seastar::lw_shared_ptr_no_esft<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >::~lw_shared_ptr_no_esft() at ??:?
 0x367a9f23: seastar::internal::lw_shared_ptr_accessors_no_esft<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >::dispose(seastar::lw_shared_ptr_counter_base*) at ??:?
 0x367a9d7f: seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >::operator=(seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >&&) at ??:?
 0x367a4bc5: seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >::operator=(decltype(nullptr)) at ??:?
 0x367a4405: seastar::foreign_ptr<seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > > >::destroy_on(seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >, unsigned int) at ??:?
 0x367a3f71: seastar::foreign_ptr<seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > > >::destroy(seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >, unsigned int) at ??:?
 0x367a38bb: seastar::foreign_ptr<seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > > >::~foreign_ptr() at ??:?
 0x4274ed4c: rpc::transport::entry::~entry() at ??:?
 0x4274ec0c: std::__1::default_delete<rpc::transport::entry>::operator()(rpc::transport::entry*) const at transport.cc:?
 0x4274e9e4: std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >::reset(rpc::transport::entry*) at transport.cc:?
 0x4274e7ec: std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >::~unique_ptr() at transport.cc:?
 0x4274e78c: std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >::~pair() at ??:?
 0x4274e730: void std::__1::__destroy_at<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >, 0>(std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at transport.cc:?
 0x4274e6cc: void std::__1::destroy_at<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >, 0>(std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at transport.cc:?
 0x4274e6a0: void std::__1::allocator_traits<std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > > >::destroy<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >, void, void>(std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >&, std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at ??:?
 0x4274e66a: void absl::lts_20210324::container_internal::map_slot_policy<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >::destroy<std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > > >(std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >*, absl::lts_20210324::container_internal::map_slot_type<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at ??:?
 0x4274e194: absl::lts_20210324::container_internal::common_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false, absl::lts_20210324::container_internal::map_slot_policy<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >::destroy(std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >*, absl::lts_20210324::container_internal::map_slot_type<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at ??:?
 0x4274cfc4: absl::lts_20210324::container_internal::btree_node<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> >::value_destroy_n(unsigned char, unsigned char, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >*) at ??:?
 0x4274c503: absl::lts_20210324::container_internal::btree_node<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> >::clear_and_delete(absl::lts_20210324::container_internal::btree_node<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> >*, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >*) at ??:?
 0x4274c2b2: absl::lts_20210324::container_internal::btree<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> >::clear() at ??:?
 0x42746828: absl::lts_20210324::container_internal::btree_container<absl::lts_20210324::container_internal::btree<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> > >::clear() at ??:?
 0x4271ab57: rpc::transport::fail_outstanding_futures() at ??:?
 0x4327353c: net::base_transport::stop() at ??:?
 0x427d9008: rpc::reconnect_transport::stop()::$_1::operator()() const at reconnect_transport.cc:?
 0x427d8e83: seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<rpc::reconnect_transport::stop()::$_1&>(rpc::reconnect_transport::stop()::$_1&) at reconnect_transport.cc:?
 0x427d8e0f: auto seastar::futurize_invoke<rpc::reconnect_transport::stop()::$_1&>(rpc::reconnect_transport::stop()::$_1&) at reconnect_transport.cc:?
 0x427d8d5d: _ZZN7seastar6futureIvE4thenIZN3rpc19reconnect_transport4stopEvE3$_1S1_EET0_OT_ENUlDpOT_E_clIJEEEDaSB_ at reconnect_transport.cc:?
 0x427d8ccb: _ZN7seastar20noncopyable_functionIFNS_6futureIvEEvEE17direct_vtable_forIZNS2_4thenIZN3rpc19reconnect_transport4stopEvE3$_1S2_EET0_OT_EUlDpOT_E_E4callEPKS4_ at reconnect_transport.cc:?
 0x3483e26b: seastar::noncopyable_function<seastar::future<void> ()>::operator()() const at application.cc:?
 0x3483e0f3: decltype ((static_cast<seastar::noncopyable_function<seastar::future<void> ()>&>({parm#1}))()) std::__1::__invoke<seastar::noncopyable_function<seastar::future<void> ()>&>(seastar::noncopyable_function<seastar::future<void> ()>&) at application.cc:?
 0x3483e01f: std::__1::invoke_result<seastar::noncopyable_function<seastar::future<void> ()>&>::type std::__1::invoke<seastar::noncopyable_function<seastar::future<void> ()>&>(seastar::noncopyable_function<seastar::future<void> ()>&) at application.cc:?
 0x3483de13: auto seastar::internal::future_invoke<seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate>(seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate&&) at application.cc:?
 0x3483dd5c: seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}::operator()() const at application.cc:?
 0x3483daec: void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&&) at application.cc:?
 0x3483d8e6: seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const at application.cc:?
 0x3483d3fa: seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose() at application.cc:?
 0x4568ddfd: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at reactor.cc:?
 0x45693e16: seastar::reactor::run_some_tasks() at reactor.cc:?
 0x4569896f: seastar::reactor::do_run() at reactor.cc:?
 0x457ffa38: seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91::operator()() const at reactor.cc:?
 0x457fe6f0: decltype ((static_cast<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&>({parm#1}))()) std::__1::__invoke<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&) at reactor.cc:?
 0x457fe5ac: void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&) at reactor.cc:?
 0x457fe514: std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91>, void ()>::operator()() at reactor.cc:?
 0x457f9540: std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91>, void ()>::operator()() at reactor.cc:?
 0x346e4c15: std::__1::__function::__value_func<void ()>::operator()() const at application.cc:?
 0x345f67c8: std::__1::function<void ()>::operator()() const at application.cc:?
 0x45529258: seastar::posix_thread::start_routine(void*) at posix.cc:?
 0x9298: ?? ??:0
 0x1006a2: ?? ??:0

freed by thread T1 (reactor-1) here:
[Backtrace #1]
 0x3436ca1d: operator delete(void*) at ??:?
 0x34c1ee61: seastar::shared_ptr_count_for<raft::group_manager>::~shared_ptr_count_for() at ??:?
 0x348b3fb5: seastar::shared_ptr<raft::group_manager>::~shared_ptr() at ??:?
 0x34c1dffa: seastar::shared_ptr<raft::group_manager>::operator=(seastar::shared_ptr<raft::group_manager>&&) at ??:?
 0x34c3c741: seastar::shared_ptr<raft::group_manager>::operator=(decltype(nullptr)) at ??:?
 0x34c3c494: seastar::sharded<raft::group_manager>::stop()::{lambda(seastar::future<void>)#1}::operator()(seastar::future<void>) const::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}::operator()() const at ??:?
 0x34c3b0b3: seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::sharded<raft::group_manager>::stop()::{lambda(seastar::future<void>)#1}::operator()(seastar::future<void>) const::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}&>(seastar::sharded<raft::group_manager>::stop()::{lambda(seastar::future<void>)#1}::operator()(seastar::future<void>) const::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}&) at ??:?
 0x34c3fb63: seastar::smp_message_queue::async_work_item<seastar::sharded<raft::group_manager>::stop()::{lambda(seastar::future<void>)#1}::operator()(seastar::future<void>) const::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>::run_and_dispose() at ??:?
 0x4568ddfd: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at reactor.cc:?
 0x45693e16: seastar::reactor::run_some_tasks() at reactor.cc:?
 0x4569896f: seastar::reactor::do_run() at reactor.cc:?
 0x457ffa38: seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91::operator()() const at reactor.cc:?
 0x457fe6f0: decltype ((static_cast<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&>({parm#1}))()) std::__1::__invoke<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&) at reactor.cc:?
 0x457fe5ac: void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&) at reactor.cc:?
 0x457fe514: std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91>, void ()>::operator()() at reactor.cc:?
 0x457f9540: std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91>, void ()>::operator()() at reactor.cc:?
 0x346e4c15: std::__1::__function::__value_func<void ()>::operator()() const at application.cc:?
 0x345f67c8: std::__1::function<void ()>::operator()() const at application.cc:?
 0x45529258: seastar::posix_thread::start_routine(void*) at posix.cc:?
 0x9298: ?? ??:0

previously allocated by thread T1 (reactor-1) here:
[Backtrace #2]
 0x3436c1bd: operator new(unsigned long) at ??:?
 0x34498162: seastar::shared_ptr<raft::group_manager> seastar::shared_ptr_make_helper<raft::group_manager, false>::make<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, seastar::internal::either_sharded_or_local<rpc::connection_cache>, seastar::internal::either_sharded_or_local<storage::api>, seastar::internal::either_sharded_or_local<raft::recovery_throttle> >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, seastar::internal::either_sharded_or_local<rpc::connection_cache>&&, seastar::internal::either_sharded_or_local<storage::api>&&, seastar::internal::either_sharded_or_local<raft::recovery_throttle>&&) at application.cc:?
 0x344980f2: seastar::shared_ptr<raft::group_manager> seastar::make_shared<raft::group_manager, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, seastar::internal::either_sharded_or_local<rpc::connection_cache>, seastar::internal::either_sharded_or_local<storage::api>, seastar::internal::either_sharded_or_local<raft::recovery_throttle> >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, seastar::internal::either_sharded_or_local<rpc::connection_cache>&&, seastar::internal::either_sharded_or_local<storage::api>&&, seastar::internal::either_sharded_or_local<raft::recovery_throttle>&&) at application.cc:?
 0x34497ae1: seastar::shared_ptr<raft::group_manager> seastar::sharded<raft::group_manager>::create_local_service<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, seastar::internal::either_sharded_or_local<rpc::connection_cache>, seastar::internal::either_sharded_or_local<storage::api>, seastar::internal::either_sharded_or_local<raft::recovery_throttle> >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, seastar::internal::either_sharded_or_local<rpc::connection_cache>&&, seastar::internal::either_sharded_or_local<storage::api>&&, seastar::internal::either_sharded_or_local<raft::recovery_throttle>&&) at application.cc:?
 0x34497529: seastar::sharded<raft::group_manager>::start<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> > >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >&&, std::__1::reference_wrapper<seastar::sharded<storage::api> >&&, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >)#1}::operator()(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >) const at application.cc:?
 0x34496416: decltype ((static_cast<seastar::sharded<raft::group_manager>::start<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> > >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >&&, std::__1::reference_wrapper<seastar::sharded<storage::api> >&&, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >)#1}>({parm#1}))(static_cast<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&>({parm#2}), static_cast<std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >&>({parm#2}), static_cast<seastar::scheduling_group&>({parm#2}), static_cast<std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >&>({parm#2}), static_cast<std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >&>({parm#2}), static_cast<application::wire_up_redpanda_services()::$_21&>({parm#2}), static_cast<seastar::sharded<rpc::connection_cache>&>({parm#2}), static_cast<seastar::sharded<storage::api>&>({parm#2}), static_cast<seastar::sharded<raft::recovery_throttle>&>({parm#2}))) std::__1::__invoke_constexpr<seastar::sharded<raft::group_manager>::start<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> > >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >&&, std::__1::reference_wrapper<seastar::sharded<storage::api> >&&, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >)#1}, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >&, seastar::scheduling_group&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >&, application::wire_up_redpanda_services()::$_21&, seastar::sharded<rpc::connection_cache>&, seastar::sharded<storage::api>&, seastar::sharded<raft::recovery_throttle>&>(seastar::sharded<raft::group_manager>::start<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> > >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >&&, std::__1::reference_wrapper<seastar::sharded<storage::api> >&&, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >)#1}&&, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >&, seastar::scheduling_group&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >&, application::wire_up_redpanda_services()::$_21&, seastar::sharded<rpc::connection_cache>&, seastar::sharded<storage::api>&, seastar::sharded<raft::recovery_throttle>&) at application.cc:?
 0x34495e89: decltype(auto) std::__1::__apply_tuple_impl<seastar::sharded<raft::group_manager>::start<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> > >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >&&, std::__1::reference_wrapper<seastar::sharded<storage::api> >&&, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >)#1}, std::__1::tuple<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, application::wire_up_redpanda_services()::$_21, seastar::sharded<rpc::connection_cache>&, seastar::sharded<storage::api>&, seastar::sharded<raft::recovery_throttle>&>&, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul>(seastar::sharded<raft::group_manager>::start<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> > >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >&&, std::__1::reference_wrapper<seastar::sharded<storage::api> >&&, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >)#1}&&, std::__1::tuple<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, application::wire_up_redpanda_services()::$_21, seastar::sharded<rpc::connection_cache>&, seastar::sharded<storage::api>&, seastar::sharded<raft::recovery_throttle>&>&, std::__1::__tuple_indices<0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul>) at application.cc:?
 0x344956d7: decltype(auto) std::__1::apply<seastar::sharded<raft::group_manager>::start<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> > >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >&&, std::__1::reference_wrapper<seastar::sharded<storage::api> >&&, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >)#1}, std::__1::tuple<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, application::wire_up_redpanda_services()::$_21, seastar::sharded<rpc::connection_cache>&, seastar::sharded<storage::api>&, seastar::sharded<raft::recovery_throttle>&>&>(seastar::sharded<raft::group_manager>::start<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> > >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >&&, std::__1::reference_wrapper<seastar::sharded<storage::api> >&&, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >)#1}&&, std::__1::tuple<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, application::wire_up_redpanda_services()::$_21, seastar::sharded<rpc::connection_cache>&, seastar::sharded<storage::api>&, seastar::sharded<raft::recovery_throttle>&>&) at application.cc:?
 0x34495499: seastar::sharded<raft::group_manager>::start<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> > >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >&&, std::__1::reference_wrapper<seastar::sharded<storage::api> >&&, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()() at application.cc:?
 0x3449ad0f: seastar::future<void> seastar::futurize<void>::invoke<seastar::sharded<raft::group_manager>::start<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> > >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >&&, std::__1::reference_wrapper<seastar::sharded<storage::api> >&&, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}&>(seastar::sharded<raft::group_manager>::start<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> > >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >&&, std::__1::reference_wrapper<seastar::sharded<storage::api> >&&, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}&) at application.cc:?
 0x3449a793: seastar::smp_message_queue::async_work_item<seastar::sharded<raft::group_manager>::start<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >, std::__1::reference_wrapper<seastar::sharded<storage::api> >, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> > >(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, seastar::scheduling_group&&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > const&, application::wire_up_redpanda_services()::$_21&&, std::__1::reference_wrapper<seastar::sharded<rpc::connection_cache> >&&, std::__1::reference_wrapper<seastar::sharded<storage::api> >&&, std::__1::reference_wrapper<seastar::sharded<raft::recovery_throttle> >&&)::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}>::run_and_dispose() at application.cc:?
 0x4568ddfd: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at reactor.cc:?
 0x45693e16: seastar::reactor::run_some_tasks() at reactor.cc:?
 0x4569896f: seastar::reactor::do_run() at reactor.cc:?
 0x457ffa38: seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91::operator()() const at reactor.cc:?
 0x457fe6f0: decltype ((static_cast<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&>({parm#1}))()) std::__1::__invoke<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&) at reactor.cc:?
 0x457fe5ac: void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91&) at reactor.cc:?
 0x457fe514: std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91>, void ()>::operator()() at reactor.cc:?
 0x457f9540: std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_91>, void ()>::operator()() at reactor.cc:?
 0x346e4c15: std::__1::__function::__value_func<void ()>::operator()() const at application.cc:?
 0x345f67c8: std::__1::function<void ()>::operator()() const at application.cc:?
 0x45529258: seastar::posix_thread::start_routine(void*) at posix.cc:?
 0x9298: ?? ??:0

@mmaslankaprv mmaslankaprv mentioned this issue Aug 23, 2022
Closed
@ZeDRoman ZeDRoman mentioned this issue Aug 23, 2022
Merged
5 tasks
@bharathv
Copy link
Contributor

Another instance: https://ci-artifacts.dev.vectorized.cloud/redpanda/0182cc67-74cd-4666-be11-f921b2460e19/vbuild/ducktape/results/2022-08-23--001/PartitionMoveInterruption/test_cancelling_partition_move_x_core/replication_factor=3.unclean_abort=True.recovery=restart_recovery/188/report.txt

std::exception_ptr::operator bool() const at application.cc:?
seastar::basic_semaphore<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::signal(unsigned long) at ??:?
seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::return_all() at ??:?
seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::~semaphore_units() at ??:?
void std::__1::__destroy_at<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, 0>(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at application.cc:?
void std::__1::destroy_at<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, 0>(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at application.cc:?
void std::__1::allocator_traits<std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::destroy<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, void, void>(std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >&, seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at ??:?
std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::__base_destruct_at_end(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at admin_server.cc:?
std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::__clear() at admin_server.cc:?
std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::~vector() at admin_server.cc:?
seastar::lw_shared_ptr_no_esft<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >::~lw_shared_ptr_no_esft() at ??:?
seastar::internal::lw_shared_ptr_accessors_no_esft<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >::dispose(seastar::lw_shared_ptr_counter_base*) at ??:?
seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >::operator=(seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >&&) at ??:?
seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >::operator=(decltype(nullptr)) at ??:?
seastar::foreign_ptr<seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > > >::destroy_on(seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >, unsigned int) at ??:?
seastar::foreign_ptr<seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > > >::destroy(seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > >, unsigned int) at ??:?
seastar::foreign_ptr<seastar::lw_shared_ptr<std::__1::vector<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > > > >::~foreign_ptr() at ??:?
rpc::transport::entry::~entry() at ??:?
std::__1::default_delete<rpc::transport::entry>::operator()(rpc::transport::entry*) const at transport.cc:?
std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >::reset(rpc::transport::entry*) at transport.cc:?
std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >::~unique_ptr() at transport.cc:?
std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >::~pair() at ??:?
void std::__1::__destroy_at<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >, 0>(std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at transport.cc:?
void std::__1::destroy_at<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >, 0>(std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at transport.cc:?
void std::__1::allocator_traits<std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > > >::destroy<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >, void, void>(std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >&, std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at ??:?
void absl::lts_20210324::container_internal::map_slot_policy<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >::destroy<std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > > >(std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >*, absl::lts_20210324::container_internal::map_slot_type<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at ??:?
absl::lts_20210324::container_internal::common_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false, absl::lts_20210324::container_internal::map_slot_policy<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >::destroy(std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >*, absl::lts_20210324::container_internal::map_slot_type<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > >*) at ??:?
absl::lts_20210324::container_internal::btree_node<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> >::value_destroy_n(unsigned char, unsigned char, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >*) at ??:?
absl::lts_20210324::container_internal::btree_node<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> >::clear_and_delete(absl::lts_20210324::container_internal::btree_node<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> >*, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >*) at ??:?
absl::lts_20210324::container_internal::btree<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> >::clear() at ??:?
absl::lts_20210324::container_internal::btree_container<absl::lts_20210324::container_internal::btree<absl::lts_20210324::container_internal::map_params<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> >, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> >, std::__1::less<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned long, rpc::sequence_tag, std::__1::integral_constant<bool, true> > const, std::__1::unique_ptr<rpc::transport::entry, std::__1::default_delete<rpc::transport::entry> > > >, 256, false> > >::clear() at ??:?
rpc::transport::fail_outstanding_futures() at ??:?
net::base_transport::stop() at ??:?
rpc::reconnect_transport::stop()::$_1::operator()() const at reconnect_transport.cc:?
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<rpc::reconnect_transport::stop()::$_1&>(rpc::reconnect_transport::stop()::$_1&) at reconnect_transport.cc:?
auto seastar::futurize_invoke<rpc::reconnect_transport::stop()::$_1&>(rpc::reconnect_transport::stop()::$_1&) at reconnect_transport.cc:?
_ZZN7seastar6futureIvE4thenIZN3rpc19reconnect_transport4stopEvE3$_1S1_EET0_OT_ENUlDpOT_E_clIJEEEDaSB_ at reconnect_transport.cc:?
_ZN7seastar20noncopyable_functionIFNS_6futureIvEEvEE17direct_vtable_forIZNS2_4thenIZN3rpc19reconnect_transport4stopEvE3$_1S2_EET0_OT_EUlDpOT_E_E4callEPKS4_ at reconnect_transport.cc:?
seastar::noncopyable_function<seastar::future<void> ()>::operator()() const at application.cc:?
decltype ((static_cast<seastar::noncopyable_function<seastar::future<void> ()>&>({parm#1}))()) std::__1::__invoke<seastar::noncopyable_function<seastar::future<void> ()>&>(seastar::noncopyable_function<seastar::future<void> ()>&) at application.cc:?
std::__1::invoke_result<seastar::noncopyable_function<seastar::future<void> ()>&>::type std::__1::invoke<seastar::noncopyable_function<seastar::future<void> ()>&>(seastar::noncopyable_function<seastar::future<void> ()>&) at application.cc:?
auto seastar::internal::future_invoke<seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate>(seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate&&) at application.cc:?
seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}::operator()() const at application.cc:?
void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&&) at application.cc:?
seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const at application.cc:?
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose() at application.cc:?
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at reactor.cc:?
seastar::reactor::run_some_tasks() at reactor.cc:?
seastar::reactor::do_run() at reactor.cc:?
seastar::reactor::run() at reactor.cc:?
seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at app-template.cc:?
seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at app-template.cc:?
application::run(int, char**) at ??:?
main at ??:?
?? ??:0
_start at ??:?

@andrwng andrwng mentioned this issue Aug 26, 2022
Merged
6 tasks
@mmaslankaprv mmaslankaprv mentioned this issue Oct 18, 2022
Merged
6 tasks
@jcsp jcsp mentioned this issue Oct 25, 2022
Merged
@vbotbuildovich vbotbuildovich mentioned this issue Oct 25, 2022
Closed
@vbotbuildovich vbotbuildovich mentioned this issue Nov 16, 2022
Closed
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dlex dlex

Labels
area/redpanda area/tests ci-disabled-test ci-failure kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

rpc/transport: release caller units when timeout occurs mmaslankaprv/redpanda
7 participants
@dotnwat @bharathv @rystsov @ztlpn @dlex @mmedenjak @mmaslankaprv