-
Notifications
You must be signed in to change notification settings - Fork 577
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Shard aware connections v1 #5
Shard aware connections v1 #5
Conversation
Signed-off-by: Noah Watkins <noah@vectorized.io>
Signed-off-by: Noah Watkins <noah@vectorized.io>
Changes the batch cache reclaim test to use the record batch builder because creating batches is now far stricter about batches being internally consistent. Signed-off-by: Noah Watkins <noah@vectorized.io>
Various storage updates: - Conversion of compression/decompression routines to use new record batch constructor to avoid record materializaiton. - Expanded context information in batch consumer logging messages - Bug fixes in segment appender leading to data loss - Extra consistency checks in record batch constructor * 'storage-update-v1' of github.com:dotnwat/v: storage: cache reclaim test should use consistent batches model: remove unused code storage: batch builder should use new batch ctor storage: fix typo in method name model: remove unused code model: add consistency checks to batch construction storage: ensure batch creation has consistent header model: add batch crc helper for header and data storage: use new batch ctor for segment reducer storage: use new batch memory format for compress storage: use new batch memory format for decompress batch_cache: use new style ctor for test batches storage: segment append can eagerly dispatch write log parser: display consumer ctx in parser log msgs storage: a print interface to batch consumers storage: ensure fallocate is disjoint from commit offset storage: inline segment appender chunk operations storage: ensure segment truncation is durable
Update to Seastar API_LEVEL 6 * 'update-seastar-api-v6' of github.com:BenPope/v: 3rdparty: Update to Seastar API_LEVEL 6 3rdparty: Update to Seastar API_LEVEL 5 3rdparty: Update to Seastar API_LEVEL 4
tools/bootstrap: Only install ubuntu-toolchain-test-r on older ubuntu Ubuntu 20+ already has g++-9 * 'restrict-ubuntu-toolchain-r-v1' of github.com:BenPope/v: bootstrap: Only install ubuntu-toolchain-test-r on older ubuntu
Signed-off-by: Noah Watkins <noah@vectorized.io>
CI Improvements * 'build-improvements-node-builder-v1' of github.com:BenPope/v: ci: Reuse the clang base image for building ci: exclude src/js/generate-entries.sh from dockerignore
Repalce pcg-cpp with abseil/random/internal implementaton CCache hit ratio goes from ~55% to 100%. * 'replace-pcg32-with-abseil-internal-v1' of github.com:BenPope/v: random: Use abseil/random/internal/pcg32
Exception thrown during reader compaction caused reader destructor assertion to be triggered as the reader wasn't closed. The reader is captured in lambda executed in `do_compact_segment_index.then(...` if exceptional future is returned from `do_compact_segment_index` continuation will not be executed hence the reader will stay open. Fixed the problem by creating the segment reader right before it is going to be used. Reader will be closed automatically after all the batches will be consumed This issue was revealed after recent segment_appender fixes from Noah. Signed-off-by: Michal Maslanka <michal@vectorized.io>
vtools doesn't pass a SHA for the base, which results in a build error. But we can reuse it from clang base, and then not copy llvm dir.
Add optional parameter to rpc::base_transport to pass SNI hostname parameter. It's not needed for all https endpoints but might be required for some.
- Also included is an abstract type 'coprocessor' emmulating the javascript API. Included are a few subclasses that modify a stream in a predictable way to be used to test the router
Introducing a new type of routing mechanism to route inbound records on coproc registered topics to the javascript/typescript copro engine. The router polls for data from v/storage on interested ntps, and forwards the data wrapped in enevelopes defined in v/coproc/types.h. The type recieved from the engine is an array of 2-tuples (represeting topic -> list of record batches) of data to produce onto materialized logs. * 'copro-router-v13' of github.com:graphcareful/v: coproc: Rename management_service to script_manager_service coproc/tests: Adding unit tests for the coproc::router coproc/tests: Simplified version of coprocessor engine for testing coproc/tests: Adding coproc/utilities module for common methods coproc/tests: copro/service unit tests + coproc harness rpc/test: Enhance rpc test fixture to start a sharded service redpanda: Replace active_mappings with copro::router config: Allow configurable coproc services ip/port coproc: Modify coproc/service to use the new router class coproc: router - sends batches to copro + writes response batches rpc: Initialize after base - compiler warning fix coproc/engine: New IDL to describe coprocessor engine/service reflection: Update async_adl<model::record_batch_reader> ssx: Add async_transform a non parallel version of parallel_xform reflection: Template specialization for async_adl::std::vector<T> coproc: Update errc with new error types coproc: Update coproc types cluster: Fixed use after free error with calls to async_adl<T>
* 'build-improvements-builder-v2' of github.com:BenPope/v: ci: Reuse the clang base image for building
--- since v2: - fix typos - improve error message
--- since v2: - fix typos
--- since v3: - fix path in build:public command, now it will create vectorized dependency in root js folder add compact-public.js, this file has a webpack script that compact all public folder into modules/public/ in one file, that will save /sdk package.json adds a webpack dependency and add build:public command
DPDK will be build by default for release builds. It can be disabled by passing `--enable-dpdk=false` as a vtools build argument Signed-off-by: Michal Maslanka <michal@vectorized.io>
Integrates chaos tests with ducktape
* 'feature/export-public-folder-v4' of github.com:andresaristizabal/v: coproc: add webpack script to compact public folder coproc: add index for exporting all public dependencies coproc: add util for creating record batch coproc: add wrapper for coprocessor class coproc: ignore vectorized public dependecy
--- since v3: - change template implementation for vectorized dependency, now it use a f-string instance to jinja2 - add comment about "why do we create vectorized dependency here?" since v5: - improve npm step in `vtools build go`
This reverts commit cd833a596a2a32a3ce8d1c4a9fa736898bd61006, reversing changes made to 7d1d6ade6e4d935dfa98fcbe23beffb618298d69.
This reverts commit 7d1d6ade6e4d935dfa98fcbe23beffb618298d69, reversing changes made to 89e222e124610bdf336737172a467001b58f4367.
Added max_consumable_offset field to consensus to make sure that we have a sound representation of an offset that may be read by consumers. Max consumable offset is an offset that is safe to be fetched by the consumers. This is similar to Kafka's HighWatermark. Max consumable offset is depends on the consistency level of messages replicated by the consensus instance and state of the log. Max consumable offset similarly to HighWaterMark is monotonic and can never move backward. Max consumable offset is always greater than log start offset and smaller than log dirty offset. Max consumable offset is updated in two scenarios - committed offset is updated (consistency_level=quorum) - when batch with was appended to the leader log with relaxed consistency We always update max_consumable offset with std::max(prev, possible_value) to guarantee its monotonicity. Signed-off-by: Michal Maslanka <michal@vectorized.io>
Enabling batch cache allow consumers to read up to 30K. When Kafka consumer receives batches it use last batch last offset + 1 as a start offset for the next fetch request. This request didn't passed redpanda validation since we were using log::committed_offset as a upper bound for validation. Fixed the problem by using newly introduced raft::max_consumable_offset which determines maximal offset visible to consumers. Fixes: [ch1683] Signed-off-by: Michal Maslanka <michal@vectorized.io>
Added test simulating produce/consume workload used in Hazelcast benchmark. Signed-off-by: Michal Maslanka <michal@vectorized.io>
Introduced consistent semantic to last consumable log offset. This patch set is fixing errors observed by Hazelcast. * 'max-consumable-offset-v3' of github.com:mmaslankaprv/v: k/tests: added consume/produce test for small batches k/fetch_request: fixed offset validation r/consensus: added max consumable offset
Adds a pre-vote step to Raft's leader election to avoid disruptions when a server rejoins a cluster. It checks if a majority of nodes is willing to vote for a candidate before incrementing its term, the change based on the 9.6 section on the raft thesis.
After a follower casts a vote it ignores new voting requests for a while. Since we use a vote request as prevote it may cause a prevote request to suppress a follow up vote. This change updates a node not to ignore a vote request in case it comes from a voted for node thus fixing he prevote blocking vote problem.
Adds a pre-vote step to Raft's leader election to avoid disruptions when a server rejoins a cluster. It checks if a majority of nodes is willing to vote for a candidate before incrementing its term, the change based on the 9.6 section on the raft dissertation.
We want a raft leader to step down when it's disconnected from peers to avoid availability hit in case it's still connected to the client.
This reverts commit 1d8ee3b4818f0d99ca4af540ce25b2d6324a1d0d, reversing changes made to 1c673058571e7285c16fff048108cd056cb9d6f8.
src/v/rpc/connection_cache.h
Outdated
@@ -92,7 +99,8 @@ inline ss::shard_id connection_cache::shard_for( | |||
// NOLINTNEXTLINE | |||
size_t h = universe[jump_consistent_hash(src_shard, vnodes)]; | |||
boost::hash_combine(h, std::hash<model::node_id>{}(n)); | |||
return jump_consistent_hash(h, total_shards); | |||
// use self node id to shift jump_consistent_hash_assignment | |||
return (jump_consistent_hash(h, total_shards) + self()) % total_shards; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i don't get this. why not simply do another line of
boost::hash_combine(h, self())
You want a better bit distribution than modulo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good idea, will combine with node_id
src/v/rpc/types.h
Outdated
@@ -172,6 +172,8 @@ struct server_configuration { | |||
std::optional<ss::tls::credentials_builder> credentials; | |||
metrics_disabled disable_metrics = metrics_disabled::no; | |||
ss::sstring name; | |||
std::optional<ss::server_socket::load_balancing_algorithm> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not simply default it to port based? rather than std::optional
Also, i think we should. expose this via the configuration.hh
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have to make it configurable as we do not want Kafka server to have shard aware connections as the client may use some random ports to connect to redpanda. This would cause uneven distribution of Kafka connections on shards.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
then, let's use the seastar default. My point was that we already have a default/optional value which is to use the default value. So let's default that so it is more explicit.
Seastar support different load balancing algorithms that determine which shard will handle incoming connection. Added configuration option to rpc::server allowing to configure the listener load balancing algorithm. Signed-off-by: Michal Maslanka <michal@vectorized.io>
Added current node_id shift to outgoing connections shard assignment. In order to make sure that connections to the same node created on different nodes are assigned to different shards added node_id based offset to connections shard assignment. This approach is required to make use of deterministic incoming connection to shard assignment on the target node. If there would be no shift all connections to given node would be handled on the same set of shards. Signed-off-by: Michal Maslanka <michal@vectorized.io>
Using port based shard connection assignment in RPC server. This makes incoming connection shard assignment deterministic. Signed-off-by: Michal Maslanka <michal@vectorized.io>
fc03832
to
beace57
Compare
gofmt -s -w
Regarding the test `serde_test` in `src/v/storage/tests/index_state_test.cc` $> ninja storage_log_index_rpunit $> bin/storage_log_index_rpunit Running 7 test cases... AddressSanitizer:DEADLYSIGNAL ================================================================= ==934285==ERROR: AddressSanitizer: stack-overflow on address 0x7fff2a209f58 (pc 0x558e9c2eab0e bp 0x7fff2a20a7a0 sp 0x7fff2a209f60 T0) #0 0x558e9c2eab0e in __asan_memmove /home/nwatkins/src/redpanda/vbuild/llvm/src/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:30:3 #1 0x558e9c448ec2 in std::__1::enable_if<(is_same<std::__1::remove_const<char const>::type, char>::value) && (is_trivially_copy_assignable<char>::value), char*>::type std::__1::__copy<char const, char>(char const*, char const*, char*) /home/nwatkins/src/redpanda/vbuild/llvm/in stall/bin/../include/c++/v1/algorithm:1735:9 #2 0x558e9c448e1e in char* std::__1::copy<char const*, char*>(char const*, char const*, char*) /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/algorithm:1748:16 #3 0x558e9c448c9a in std::__1::enable_if<__is_cpp17_random_access_iterator<char const*>::value, char*>::type std::__1::copy_n<char const*, unsigned long, char*>(char const*, unsigned long, char*) /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/algorithm:1 867:12 #4 0x558e9c674104 in iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4::operator()(char const*, unsigned long) const /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/bytes/iobuf.cc:145:13 #5 0x558e9c65aa69 in unsigned long details::io_iterator_consumer::consume<iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4>(unsigned long, iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4&&) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../sr c/v/bytes/details/io_iterator_consumer.h:141:45 #6 0x558e9c659af1 in iobuf_copy(details::io_iterator_consumer&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/bytes/iobuf.cc:143:12 #7 0x558e9c57eeb3 in iobuf_parser_base::peek(unsigned long) const /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/bytes/iobuf_parser.h:100:16 #8 0x558e9c5751b6 in serde::peek_version(iobuf_parser&) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/serde/serde.h:547:43 #9 0x558e9c55ae54 in storage::read_nested(iobuf_parser&, storage::index_state&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/storage/index_state.cc:257:9 #10 0x558e9c475297 in std::__1::decay<storage::index_state>::type serde::read_nested<storage::index_state>(iobuf_parser&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/serde/serde.h:475:5 #11 0x558e9c55aedb in storage::read_nested(iobuf_parser&, storage::index_state&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/storage/index_state.cc:258:14 #12 0x558e9c475297 in std::__1::decay<storage::index_state>::type serde::read_nested<storage::index_state>(iobuf_parser&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/serde/serde.h:475:5 #13 0x558e9c55aedb in storage::read_nested(iobuf_parser&, storage::index_state&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/storage/index_state.cc:258:14 #14 0x558e9c475297 in std::__1::decay<storage::index_state>::type serde::read_nested<storage::index_state>(iobuf_parser&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/serde/serde.h:475:5 #15 0x558e9c55aedb in storage::read_nested(iobuf_parser&, storage::index_state&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/storage/index_state.cc:258:14 ... === This makes the recursion go away, but the deserialized output is all zeros. So that probably isn't the solution. diff --git a/src/v/storage/index_state.cc b/src/v/storage/index_state.cc index 57808eae2..4aceb904f 100644 --- a/src/v/storage/index_state.cc +++ b/src/v/storage/index_state.cc @@ -255,7 +255,7 @@ iobuf index_state::checksum_and_serialize() { void read_nested(iobuf_parser& in, index_state& st, size_t bytes_left_limit) { if (serde::peek_version(in) > index_state::ondisk_version) { - st = serde::read_nested<index_state>(in, bytes_left_limit); + serde::read_nested<index_state>(in, st, bytes_left_limit); return; } Signed-off-by: Noah Watkins <noah@vectorized.io>
Regarding the test `serde_test` in `src/v/storage/tests/index_state_test.cc` $> ninja storage_log_index_rpunit $> bin/storage_log_index_rpunit Running 7 test cases... AddressSanitizer:DEADLYSIGNAL ================================================================= ==934285==ERROR: AddressSanitizer: stack-overflow on address 0x7fff2a209f58 (pc 0x558e9c2eab0e bp 0x7fff2a20a7a0 sp 0x7fff2a209f60 T0) #0 0x558e9c2eab0e in __asan_memmove /home/nwatkins/src/redpanda/vbuild/llvm/src/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:30:3 #1 0x558e9c448ec2 in std::__1::enable_if<(is_same<std::__1::remove_const<char const>::type, char>::value) && (is_trivially_copy_assignable<char>::value), char*>::type std::__1::__copy<char const, char>(char const*, char const*, char*) /home/nwatkins/src/redpanda/vbuild/llvm/in stall/bin/../include/c++/v1/algorithm:1735:9 #2 0x558e9c448e1e in char* std::__1::copy<char const*, char*>(char const*, char const*, char*) /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/algorithm:1748:16 #3 0x558e9c448c9a in std::__1::enable_if<__is_cpp17_random_access_iterator<char const*>::value, char*>::type std::__1::copy_n<char const*, unsigned long, char*>(char const*, unsigned long, char*) /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/algorithm:1 867:12 #4 0x558e9c674104 in iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4::operator()(char const*, unsigned long) const /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/bytes/iobuf.cc:145:13 #5 0x558e9c65aa69 in unsigned long details::io_iterator_consumer::consume<iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4>(unsigned long, iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4&&) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../sr c/v/bytes/details/io_iterator_consumer.h:141:45 #6 0x558e9c659af1 in iobuf_copy(details::io_iterator_consumer&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/bytes/iobuf.cc:143:12 #7 0x558e9c57eeb3 in iobuf_parser_base::peek(unsigned long) const /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/bytes/iobuf_parser.h:100:16 #8 0x558e9c5751b6 in serde::peek_version(iobuf_parser&) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/serde/serde.h:547:43 #9 0x558e9c55ae54 in storage::read_nested(iobuf_parser&, storage::index_state&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/storage/index_state.cc:257:9 #10 0x558e9c475297 in std::__1::decay<storage::index_state>::type serde::read_nested<storage::index_state>(iobuf_parser&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/serde/serde.h:475:5 #11 0x558e9c55aedb in storage::read_nested(iobuf_parser&, storage::index_state&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/storage/index_state.cc:258:14 #12 0x558e9c475297 in std::__1::decay<storage::index_state>::type serde::read_nested<storage::index_state>(iobuf_parser&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/serde/serde.h:475:5 #13 0x558e9c55aedb in storage::read_nested(iobuf_parser&, storage::index_state&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/storage/index_state.cc:258:14 #14 0x558e9c475297 in std::__1::decay<storage::index_state>::type serde::read_nested<storage::index_state>(iobuf_parser&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/serde/serde.h:475:5 #15 0x558e9c55aedb in storage::read_nested(iobuf_parser&, storage::index_state&, unsigned long) /home/nwatkins/src/redpanda/vbuild/debug/clang/../../../src/v/storage/index_state.cc:258:14 ... === This makes the recursion go away, but the deserialized output is all zeros. So that probably isn't the solution. diff --git a/src/v/storage/index_state.cc b/src/v/storage/index_state.cc index 57808eae2..4aceb904f 100644 --- a/src/v/storage/index_state.cc +++ b/src/v/storage/index_state.cc @@ -255,7 +255,7 @@ iobuf index_state::checksum_and_serialize() { void read_nested(iobuf_parser& in, index_state& st, size_t bytes_left_limit) { if (serde::peek_version(in) > index_state::ondisk_version) { - st = serde::read_nested<index_state>(in, bytes_left_limit); + serde::read_nested<index_state>(in, st, bytes_left_limit); return; } Signed-off-by: Noah Watkins <noah@vectorized.io>
Without this change there is a data race that can result in a heap-use-after-free. ``` ==3190349==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000257420 at pc 0x14995faf26e4 bp 0x7ffd3e04b710 sp 0x7ffd3e04b708 READ of size 8 at 0x60b000257420 thread T0 #0 0x14995faf26e3 in profile_flush_file_data /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:468:24 #1 0x14995faf0cef in profile_close_file /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:559:14 #2 0x14995fb0c086 in profile_release /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_init.c:514:13 #3 0x14995fa924c6 in k5_os_free_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/os/init_os_ctx.c:506:9 #4 0x14995f938b87 in krb5_free_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/krb/init_ctx.c:294:5 #5 0x14995ffc41c3 in krb5_gss_delete_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/krb5/delete_sec_context.c:87:9 #6 0x14995ff399a6 in gssint_delete_internal_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/mechglue/g_glue.c:606:15 #7 0x14995ff25450 in gss_delete_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/mechglue/g_delete_sec_context.c:91:11 #8 0x55e258b9c641 in security::gss::ctx_id::reset() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi.h:170:13 #9 0x55e258b9c408 in security::gss::ctx_id::~ctx_id() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi.h:165:17 #10 0x55e258b9c37c in security::gssapi_authenticator::impl::~impl() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:116:29 #11 0x55e258b9c23c in std::__1::default_delete<security::gssapi_authenticator::impl>::operator()(security::gssapi_authenticator::impl*) const /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__memory/unique_ptr.h:54:5 #12 0x55e258b65324 in std::__1::unique_ptr<security::gssapi_authenticator::impl, std::__1::default_delete<security::gssapi_authenticator::impl> >::reset(security::gssapi_authenticator::impl*) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__memory/unique_ptr.h:315:7 #13 0x55e258b5f302 in security::gssapi_authenticator::authenticate(seastar::basic_sstring<unsigned char, unsigned int, 31u, false>) (.resume) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:202:15 #14 0x55e258b11830 in std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<boost::outcome_v2::basic_result<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, void> > >::promise_type>::resume() const /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__coroutine/coroutine_handle.h:168:9 #15 0x55e258b11315 in seastar::internal::coroutine_traits_base<boost::outcome_v2::basic_result<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, void> > >::promise_type::run_and_dispose() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/rp_deps_install/include/seastar/core/coroutine.hh:78:20 #16 0x55e2594a1ead in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330:14 #17 0x55e2594a7d0e in seastar::reactor::run_some_tasks() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737:9 #18 0x55e2594ac86f in seastar::reactor::do_run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906:9 #19 0x55e2594aa3f8 in seastar::reactor::run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2789:16 #20 0x55e2591c0a7c in seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265:31 #21 0x55e2591be135 in seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156:12 #22 0x55e2470c46e4 in application::run(int, char**) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/redpanda/application.cc:323:16 #23 0x55e247081d16 in main /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/redpanda/main.cc:22:16 #24 0x14995cc29d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 #25 0x14995cc29e3f in __libc_start_main csu/../csu/libc-start.c:392:3 #26 0x55e246fc1844 in _start (/home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/bin/redpanda+0x3a678844) (BuildId: 0a81be2927716d279ef7fc96d20cc5d5dfbd1cb2) 0x60b000257420 is located 0 bytes inside of 104-byte region [0x60b000257420,0x60b000257488) freed by thread T0 here: #0 0x55e2470443e2 in free /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/src/compiler-rt/lib/asan/asan_malloc_linux.cpp:52:3 #1 0x14995faf3ed4 in profile_free_file_data /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:552:5 #2 0x14995faf3860 in profile_dereference_data_locked /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:509:9 #3 0x14995faf0b74 in profile_dereference_data /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:502:5 #4 0x14995faf3fc1 in profile_free_file /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:523:5 #5 0x14995faf0d38 in profile_close_file /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:562:5 #6 0x14995fb0c086 in profile_release /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_init.c:514:13 #7 0x14995fa924c6 in k5_os_free_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/os/init_os_ctx.c:506:9 #8 0x14995f938b87 in krb5_free_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/krb/init_ctx.c:294:5 #9 0x14995ffc41c3 in krb5_gss_delete_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/krb5/delete_sec_context.c:87:9 #10 0x14995ff399a6 in gssint_delete_internal_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/mechglue/g_glue.c:606:15 #11 0x14995ff25450 in gss_delete_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/mechglue/g_delete_sec_context.c:91:11 #12 0x55e258b9c641 in security::gss::ctx_id::reset() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi.h:170:13 #13 0x55e258b9c408 in security::gss::ctx_id::~ctx_id() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi.h:165:17 #14 0x55e258b9c37c in security::gssapi_authenticator::impl::~impl() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:116:29 #15 0x55e258b9c23c in std::__1::default_delete<security::gssapi_authenticator::impl>::operator()(security::gssapi_authenticator::impl*) const /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__memory/unique_ptr.h:54:5 #16 0x55e258b65324 in std::__1::unique_ptr<security::gssapi_authenticator::impl, std::__1::default_delete<security::gssapi_authenticator::impl> >::reset(security::gssapi_authenticator::impl*) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__memory/unique_ptr.h:315:7 #17 0x55e258b5f302 in security::gssapi_authenticator::authenticate(seastar::basic_sstring<unsigned char, unsigned int, 31u, false>) (.resume) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:202:15 #18 0x55e258b11830 in std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<boost::outcome_v2::basic_result<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, void> > >::promise_type>::resume() const /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__coroutine/coroutine_handle.h:168:9 #19 0x55e258b11315 in seastar::internal::coroutine_traits_base<boost::outcome_v2::basic_result<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, void> > >::promise_type::run_and_dispose() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/rp_deps_install/include/seastar/core/coroutine.hh:78:20 #20 0x55e2594a1ead in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330:14 #21 0x55e2594a7d0e in seastar::reactor::run_some_tasks() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737:9 #22 0x55e2594ac86f in seastar::reactor::do_run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906:9 #23 0x55e2594aa3f8 in seastar::reactor::run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2789:16 #24 0x55e2591c0a7c in seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265:31 #25 0x55e2591be135 in seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156:12 #26 0x55e2470c46e4 in application::run(int, char**) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/redpanda/application.cc:323:16 #27 0x55e247081d16 in main /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/redpanda/main.cc:22:16 #28 0x14995cc29d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 previously allocated by thread T4 here: #0 0x55e24704468e in malloc /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/src/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3 #1 0x14995faee48d in profile_make_prf_data /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:150:9 #2 0x14995faf0360 in profile_open_file /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:233:12 #3 0x14995fb0ab1e in profile_init_flags /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_init.c:190:22 #4 0x14995fa90bc0 in os_init_paths /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/os/init_os_ctx.c:387:18 #5 0x14995fa9090e in k5_os_init_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/os/init_os_ctx.c:438:18 #6 0x14995f936ece in krb5_init_context_profile /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/krb/init_ctx.c:209:14 #7 0x14995f936871 in krb5_init_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/krb/init_ctx.c:139:12 #8 0x14995fffc70b in krb5_gss_init_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/krb5/init_sec_context.c:1047:12 #9 0x14995ffa0719 in kg_accept_krb5 /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/krb5/accept_sec_context.c:694:12 #10 0x14995ff9e686 in krb5_gss_accept_sec_context_ext /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/krb5/accept_sec_context.c:1311:12 #11 0x14995ffaaf9c in krb5_gss_accept_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/krb5/accept_sec_context.c:1340:12 #12 0x14995ff05dbd in gss_accept_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/mechglue/g_accept_sec_context.c:266:15 #13 0x55e258b2b157 in security::gssapi_authenticator::impl::more(std::__1::basic_string_view<unsigned char, std::__1::char_traits<unsigned char> >) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:292:25 #14 0x55e258b2946c in security::gssapi_authenticator::impl::authenticate(seastar::basic_sstring<unsigned char, unsigned int, 31u, false>) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:218:16 #15 0x55e258b3b562 in security::gssapi_authenticator::authenticate(seastar::basic_sstring<unsigned char, unsigned int, 31u, false>)::$_0::operator()() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:195:25 #16 0x55e258b3a96a in ssx::impl::worker_task<security::gssapi_authenticator::authenticate(seastar::basic_sstring<unsigned char, unsigned int, 31u, false>)::$_0>::process(seastar::alien::instance&, unsigned int) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/ssx/thread_worker.h:67:37 #17 0x55e24764d1bc in ssx::impl::thread_worker::run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/ssx/thread_worker.h:159:20 #18 0x55e24764ca9f in ssx::impl::thread_worker::start()::'lambda'()::operator()() const /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/ssx/thread_worker.h:104:49 #19 0x55e24764c910 in decltype(static_cast<ssx::impl::thread_worker::start()::'lambda'()>(fp)()) std::__1::__invoke<ssx::impl::thread_worker::start()::'lambda'()>(ssx::impl::thread_worker::start()::'lambda'()&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/type_traits:3640:23 #20 0x55e24764c808 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ssx::impl::thread_worker::start()::'lambda'()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ssx::impl::thread_worker::start()::'lambda'()>&, std::__1::__tuple_indices<>) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/thread:282:5 #21 0x55e24764ad39 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ssx::impl::thread_worker::start()::'lambda'()> >(void*) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/thread:293:5 #22 0x14995cc94b42 in start_thread nptl/./nptl/pthread_create.c:442:8 Thread T4 created by T0 here: #0 0x55e24702db0c in pthread_create /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/src/compiler-rt/lib/asan/asan_interceptors.cpp:208:3 #1 0x55e24764ab0c in std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__threading_support:375:10 #2 0x55e24764a106 in std::__1::thread::thread<ssx::impl::thread_worker::start()::'lambda'(), void>(ssx::impl::thread_worker::start()::'lambda'()&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/thread:309:16 #3 0x55e247606d40 in ssx::impl::thread_worker::start() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/ssx/thread_worker.h:104:19 #4 0x55e247305329 in ssx::thread_worker::start() (.resume) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/ssx/thread_worker.h:199:15 #5 0x55e24756b680 in std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<void>::promise_type>::resume() const /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__coroutine/coroutine_handle.h:168:9 #6 0x55e24756b1a5 in seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/rp_deps_install/include/seastar/core/coroutine.hh:120:20 #7 0x55e2594a1ead in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330:14 #8 0x55e2594a7d0e in seastar::reactor::run_some_tasks() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737:9 #9 0x55e2594ac86f in seastar::reactor::do_run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906:9 #10 0x55e2594aa3f8 in seastar::reactor::run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2789:16 #11 0x55e2591c0a7c in seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265:31 #12 0x55e2591be135 in seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156:12 #13 0x55e2470c46e4 in application::run(int, char**) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/redpanda/application.cc:323:16 #14 0x55e247081d16 in main /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/redpanda/main.cc:22:16 #15 0x14995cc29d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 SUMMARY: AddressSanitizer: heap-use-after-free /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:468:24 in profile_flush_file_data Shadow bytes around the buggy address: 0x0c1680042e30: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa 0x0c1680042e40: fa fa fd fd fd fd fd fd fd fd fd fd fd fd fd fa 0x0c1680042e50: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd 0x0c1680042e60: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fd fd 0x0c1680042e70: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa =>0x0c1680042e80: fa fa fa fa[fd]fd fd fd fd fd fd fd fd fd fd fd 0x0c1680042e90: fd fa fa fa fa fa fa fa fa fa fd fd fd fd fd fd 0x0c1680042ea0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa 0x0c1680042eb0: fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa 0x0c1680042ec0: fa fa fa fa fa fa fd fd fd fd fd fd fd fd fd fd 0x0c1680042ed0: fd fd fd fd fa fa fa fa fa fa fa fa fd fd fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==3190349==ABORTING ``` Signed-off-by: Ben Pope <ben@redpanda.com>
This segfault is triggering as the test does something innocuous like logging a few string: ``` #0 0x7f875b0bbc1f in std::__1::basic_ostream<char, std::__1::char_traits<char> >::sentry::sentry(std::__1::basic_ostream<char, std::__1::char_traits<char> >&) (/vectorized/lib/libc++.so.1+0x65c1f) redpanda-data#1 0x5597c3979afc in std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) /vectorized/llvm/bin/../include/c++/v1/ostream:722:57 redpanda-data#2 0x5597c4079e44 in std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::operator<<<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, std::__1::basic_string_view<char, std::__1::char_traits<char> >) /vectorized/llvm/bin/../include/c++/v1/ostream:1057:12 redpanda-data#3 0x5597c602c9f3 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/log.cc:342:15 redpanda-data#4 0x5597c3c4ad20 in void seastar::logger::log<char const*, int, seastar::basic_sstring<char, unsigned int, 15u, true>&>(seastar::log_level, seastar::logger::format_info, char const*&&, int&&, seastar::basic_sstring<char, unsigned int, 15u, true>&) /vectorized/include/seastar/util/log.hh:231:17 redpanda-data#5 0x5597c3f23c51 in void seastar::logger::trace<char const*, int, seastar::basic_sstring<char, unsigned int, 15u, true>&>(seastar::logger::format_info, char const*&&, int&&, seastar::basic_sstring<char, unsigned int, 15u, true>&) /vectorized/include/seastar/util/log.hh:385:9 redpanda-data#6 0x5597c3f14b95 in cloud_roles::signature_v4::sign_header(boost::beast::http::header<true, boost::beast::http::basic_fields<std::__1::allocator<char> > >&, std::__1::basic_string_view<char, std::__1::char_traits<char> >) const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/signature.cc:382:5 redpanda-data#7 0x5597c3bf1dfb in cloud_roles::apply_aws_credentials::add_auth(boost::beast::http::header<true, boost::beast::http::basic_fields<std::__1::allocator<char> > >&) const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/apply_aws_credentials.cc:83:23 redpanda-data#8 0x5597c3b8f88d in cloud_roles::apply_credentials::add_auth(boost::beast::http::header<true, boost::beast::http::basic_fields<std::__1::allocator<char> > >&) const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/apply_credentials.h:47:23 redpanda-data#9 0x5597c3b8c578 in test_aws_headers::test_method() /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/tests/credential_tests.cc:50:17 ``` What is suspicious is that this test is using seastar's logger from a non-seastar reactor thread. Taking a look we do see some things like this_shard_id() being called, but other than that it doesn't look like it should be problematic. But, it's unclear if there is more lurking there. Changed the test to run things that log using seastar logger on a seastar thread. Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
The tee_wrapper helper was being used to set the ostream pointer for global logger to be an ostream instance that lived only on the stack, and not resetting it to something global like std::cerr after the test finished. This resulted in the following segfault in other tests that ran after the change but didn't make their own change to ostream pointer: ``` #0 0x7f875b0bbc1f in std::__1::basic_ostream<char, std::__1::char_traits<char> >::sentry::sentry(std::__1::basic_ostream<char, std::__1::char_traits<char> >&) (/vectorized/lib/libc++.so.1+0x65c1f) #1 0x5597c3979afc in std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) /vectorized/llvm/bin/../include/c++/v1/ostream:722:57 #2 0x5597c4079e44 in std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::operator<<<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, std::__1::basic_string_view<char, std::__1::char_traits<char> >) /vectorized/llvm/bin/../include/c++/v1/ostream:1057:12 #3 0x5597c602c9f3 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/log.cc:342:15 #4 0x5597c3c4ad20 in void seastar::logger::log<char const*, int, seastar::basic_sstring<char, unsigned int, 15u, true>&>(seastar::log_level, seastar::logger::format_info, char const*&&, int&&, seastar::basic_sstring<char, unsigned int, 15u, true>&) /vectorized/include/seastar/util/log.hh:231:17 #5 0x5597c3f23c51 in void seastar::logger::trace<char const*, int, seastar::basic_sstring<char, unsigned int, 15u, true>&>(seastar::logger::format_info, char const*&&, int&&, seastar::basic_sstring<char, unsigned int, 15u, true>&) /vectorized/include/seastar/util/log.hh:385:9 #6 0x5597c3f14b95 in cloud_roles::signature_v4::sign_header(boost::beast::http::header<true, boost::beast::http::basic_fields<std::__1::allocator<char> > >&, std::__1::basic_string_view<char, std::__1::char_traits<char> >) const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/signature.cc:382:5 #7 0x5597c3bf1dfb in cloud_roles::apply_aws_credentials::add_auth(boost::beast::http::header<true, boost::beast::http::basic_fields<std::__1::allocator<char> > >&) const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/apply_aws_credentials.cc:83:23 #8 0x5597c3b8f88d in cloud_roles::apply_credentials::add_auth(boost::beast::http::header<true, boost::beast::http::basic_fields<std::__1::allocator<char> > >&) const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/apply_credentials.h:47:23 #9 0x5597c3b8c578 in test_aws_headers::test_method() /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/tests/credential_tests.cc:50:17 ``` This change resets the ostream pointer in the wrapper destructor. Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
The tee_wrapper helper was being used to set the ostream pointer for global logger to be an ostream instance that lived only on the stack, and not resetting it to something global like std::cerr after the test finished. This resulted in the following segfault in other tests that ran after the change but didn't make their own change to ostream pointer: ``` #0 0x7f875b0bbc1f in std::__1::basic_ostream<char, std::__1::char_traits<char> >::sentry::sentry(std::__1::basic_ostream<char, std::__1::char_traits<char> >&) (/vectorized/lib/libc++.so.1+0x65c1f) #1 0x5597c3979afc in std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) /vectorized/llvm/bin/../include/c++/v1/ostream:722:57 redpanda-data#2 0x5597c4079e44 in std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::operator<<<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, std::__1::basic_string_view<char, std::__1::char_traits<char> >) /vectorized/llvm/bin/../include/c++/v1/ostream:1057:12 redpanda-data#3 0x5597c602c9f3 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/log.cc:342:15 redpanda-data#4 0x5597c3c4ad20 in void seastar::logger::log<char const*, int, seastar::basic_sstring<char, unsigned int, 15u, true>&>(seastar::log_level, seastar::logger::format_info, char const*&&, int&&, seastar::basic_sstring<char, unsigned int, 15u, true>&) /vectorized/include/seastar/util/log.hh:231:17 redpanda-data#5 0x5597c3f23c51 in void seastar::logger::trace<char const*, int, seastar::basic_sstring<char, unsigned int, 15u, true>&>(seastar::logger::format_info, char const*&&, int&&, seastar::basic_sstring<char, unsigned int, 15u, true>&) /vectorized/include/seastar/util/log.hh:385:9 redpanda-data#6 0x5597c3f14b95 in cloud_roles::signature_v4::sign_header(boost::beast::http::header<true, boost::beast::http::basic_fields<std::__1::allocator<char> > >&, std::__1::basic_string_view<char, std::__1::char_traits<char> >) const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/signature.cc:382:5 redpanda-data#7 0x5597c3bf1dfb in cloud_roles::apply_aws_credentials::add_auth(boost::beast::http::header<true, boost::beast::http::basic_fields<std::__1::allocator<char> > >&) const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/apply_aws_credentials.cc:83:23 redpanda-data#8 0x5597c3b8f88d in cloud_roles::apply_credentials::add_auth(boost::beast::http::header<true, boost::beast::http::basic_fields<std::__1::allocator<char> > >&) const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/apply_credentials.h:47:23 redpanda-data#9 0x5597c3b8c578 in test_aws_headers::test_method() /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/tests/credential_tests.cc:50:17 ``` This change resets the ostream pointer in the wrapper destructor. Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Made inter-node connections shard assignment deterministic.
Client side
Seastar in
posix-stack
for outgoing connections uses following port_assignment implementation:This algorithm makes outgoing port to be shard dependent.
Server side:
Incoming connections are assigned to shards using one of the following policies:
By default connections are distributed using simple round robin algorithm. In worst case scenario it can lead to the situation in which all active connections are handled on single cores. (Multiple re-connections are required).
We can leverage the fact that all the nodes in the cluster have unique ids and seastar outgoing port assignment being shard dependent to assign connection to shard deterministically on target node.
It is important to note that proposed solution is simple but it will provide perfect connection-shard assignment only on cluster with uniform number of shards per node.
Implemented solution
load_balancing_algorithm::port
on the server to assign connection to shard.