Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Oversized allocation: 3661824 bytes in storage::internal::compress_batch #11934

Closed
vshtokman opened this issue Jul 6, 2023 · 0 comments · Fixed by #12071
Closed

Oversized allocation: 3661824 bytes in storage::internal::compress_batch #11934

vshtokman opened this issue Jul 6, 2023 · 0 comments · Fixed by #12071
Assignees
Labels

Comments

@vshtokman
Copy link
Contributor

Artifact URL: https://username:passwordci-artifacts.dev.vectorized.cloud:443/vtools/8384/01892a3c-a1a4-4ec3-822c-2abcd1ace8a9/vbuild/output/ducktape-release-clang-amd64.tgz

Test details: tmp/ducktape/results/2023-07-06--001/ManyClientsTest/test_many_clients_pathological_compaction/187/RedpandaService-0-140380673610256/ip-172-31-57-103/redpanda.log

[shard 0] seastar_memory - oversized allocation: 3661824 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at 0x6e7c213 0x6af761b 0x6b07449 0x652ae4d 0x6527404 0x6525f55 0x64f5646 0x64f367a 0x643c6a9 0x642c92c 0x64336ab 0x294e6ea 0x6bc2cbf 0x6bc6377 0x6bc3526 0x6ac6590 0x6ac4988 0x20fd8e5 0x6ebd06b /opt/redpanda/lib/libc.so.6+0x2d58f /opt/redpanda/lib/libc.so.6+0x2d648 0x20f7624 log line
LLVMSymbolizer: error reading file: No such file or directory
WARNING:root:for url: https://<username:password>@ci-artifacts.dev.vectorized.cloud:443/vtools/8384/01892a3c-a1a4-4ec3-822c-2abcd1ace8a9/vbuild/output/ducktape-release-clang-amd64.tgz
WARNING:root:decoded: tmp/ducktape/results/2023-07-06--001/ManyClientsTest/test_many_clients_pathological_compaction/187/RedpandaService-0-140380673610256/ip-172-31-57-103/redpanda.log:2958:WARN  2023-07-06 10:04:58,465 [shard 0] seastar_memory - oversized allocation: 3661824 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at
[Backtrace #0]
{tmp/libexec/redpanda} 0x6e7c213: void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:60
 (inlined by) seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:86
 (inlined by) seastar::current_tasktrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:137
 (inlined by) seastar::current_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:170
{tmp/libexec/redpanda} 0x6af761b: seastar::memory::cpu_pages::warn_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:809
 (inlined by) seastar::memory::cpu_pages::check_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:859
 (inlined by) seastar::memory::cpu_pages::allocate_large(unsigned int, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:866
 (inlined by) seastar::memory::allocate_large(unsigned long, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1429
 (inlined by) seastar::memory::allocate_slowpath(unsigned long, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1547
{tmp/libexec/redpanda} 0x6b07449: seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1565
 (inlined by) malloc at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2050
{tmp/libexec/redpanda} 0x652ae4d: ZSTD_customMalloc at /v/build/v_deps_build/zstd-prefix/src/zstd/lib/compress/../common/allocations.h:30
 (inlined by) ZSTD_cwksp_create at /v/build/v_deps_build/zstd-prefix/src/zstd/lib/compress/zstd_cwksp.h:659
 (inlined by) ZSTD_resetCCtx_internal at /v/build/v_deps_build/zstd-prefix/src/zstd/lib/compress/zstd_compress.c:2118
{tmp/libexec/redpanda} 0x6527404: ZSTD_compressBegin_internal at /v/build/v_deps_build/zstd-prefix/src/zstd/lib/compress/zstd_compress.c:5026
 (inlined by) ZSTD_CCtx_init_compressStream2 at /v/build/v_deps_build/zstd-prefix/src/zstd/lib/compress/zstd_compress.c:6176
{tmp/libexec/redpanda} 0x6525f55: ZSTD_compressStream2 at /v/build/v_deps_build/zstd-prefix/src/zstd/lib/compress/zstd_compress.c:6234
{tmp/libexec/redpanda} 0x64f5646: compression::stream_zstd::do_compress(iobuf const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0cd8fef5f0cf9484c-1/redpanda/redpanda/src/v/compression/stream_zstd.cc:129
{tmp/libexec/redpanda} 0x64f367a: compression::stream_zstd::compress(iobuf const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0cd8fef5f0cf9484c-1/redpanda/redpanda/src/v/compression/stream_zstd.h:27
 (inlined by) compression::internal::zstd_compressor::compress(iobuf const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0cd8fef5f0cf9484c-1/redpanda/redpanda/src/v/compression/internal/zstd_compressor.h:20
 (inlined by) compression::compressor::compress(iobuf const&, model::compression) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0cd8fef5f0cf9484c-1/redpanda/redpanda/src/v/compression/compression.cc:45
{tmp/libexec/redpanda} 0x643c6a9: storage::internal::compress_batch(model::compression, model::record_batch const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0cd8fef5f0cf9484c-1/redpanda/redpanda/src/v/storage/parser_utils.cc:111
 (inlined by) storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0::operator()(model::record_batch&) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0cd8fef5f0cf9484c-1/redpanda/redpanda/src/v/storage/parser_utils.cc:101
 (inlined by) decltype(std::declval<storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&>()(std::declval<model::record_batch&>())) std::__1::__invoke[abi:v160004]<storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&, model::record_batch&>(storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&, model::record_batch&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:394
 (inlined by) decltype(auto) std::__1::__apply_tuple_impl[abi:v160004]<storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&, std::__1::tuple<model::record_batch>&, 0ul>(storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&, std::__1::tuple<model::record_batch>&, std::__1::__tuple_indices<0ul>) at /vectorized/llvm/bin/../include/c++/v1/tuple:1816
 (inlined by) decltype(auto) std::__1::apply[abi:v160004]<storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&, std::__1::tuple<model::record_batch>&>(storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&, std::__1::tuple<model::record_batch>&) at /vectorized/llvm/bin/../include/c++/v1/tuple:1825
 (inlined by) auto seastar::internal::do_with_impl<model::record_batch, storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0>(model::record_batch&&, storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&&) at /vectorized/include/seastar/core/do_with.hh:96
 (inlined by) seastar::future<model::record_batch> seastar::futurize<seastar::future<model::record_batch> >::invoke<seastar::future<model::record_batch> (*&)(model::record_batch&&, storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&&), model::record_batch, storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0>(seastar::future<model::record_batch> (*&)(model::record_batch&&, storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&&), model::record_batch&&, storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&&) at /vectorized/include/seastar/core/future.hh:1999
 (inlined by) auto seastar::futurize_invoke<seastar::future<model::record_batch> (*&)(model::record_batch&&, storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&&), model::record_batch, storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0>(seastar::future<model::record_batch> (*&)(model::record_batch&&, storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&&), model::record_batch&&, storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&&) at /vectorized/include/seastar/core/future.hh:2030
 (inlined by) auto seastar::do_with<model::record_batch, storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0>(model::record_batch&&, storage::internal::compress_batch(model::compression, model::record_batch&&)::$_0&&) at /vectorized/include/seastar/core/do_with.hh:131
 (inlined by) storage::internal::compress_batch(model::compression, model::record_batch&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0cd8fef5f0cf9484c-1/redpanda/redpanda/src/v/storage/parser_utils.cc:100
{tmp/libexec/redpanda} 0x642c92c: storage::internal::copy_data_segment_reducer::do_compaction(model::compression, model::record_batch) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0cd8fef5f0cf9484c-1/redpanda/redpanda/src/v/storage/compaction_reducers.cc:260
{tmp/libexec/redpanda} 0x64336ab: storage::internal::copy_data_segment_reducer::operator()(model::record_batch) (.resume) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0cd8fef5f0cf9484c-1/redpanda/redpanda/src/v/storage/compaction_reducers.cc:294
{tmp/libexec/redpanda} 0x294e6ea: std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<seastar::bool_class<seastar::stop_iteration_tag> >::promise_type>::resume[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__coroutine/coroutine_handle.h:169
 (inlined by) seastar::internal::coroutine_traits_base<seastar::bool_class<seastar::stop_iteration_tag> >::promise_type::run_and_dispose() at /vectorized/include/seastar/core/coroutine.hh:78
{tmp/libexec/redpanda} 0x6bc2cbf: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2659
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3122
{tmp/libexec/redpanda} 0x6bc6377: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3306
{tmp/libexec/redpanda} 0x6bc3526: seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3174
{tmp/libexec/redpanda} 0x6ac6590: seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265
{tmp/libexec/redpanda} 0x6ac4988: seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156
{tmp/libexec/redpanda} 0x20fd8e5: application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0cd8fef5f0cf9484c-1/redpanda/redpanda/src/v/redpanda/application.cc:346
{tmp/libexec/redpanda} 0x6ebd06b: main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0cd8fef5f0cf9484c-1/redpanda/redpanda/src/v/redpanda/main.cc:22
{/opt/redpanda/lib/libc.so.6} 0x2d58f: ?? at ??:0
{/opt/redpanda/lib/libc.so.6} 0x2d648: ?? at ??:0
{tmp/libexec/redpanda} 0x20f7624: _start at ??:0
@vshtokman vshtokman added kind/bug Something isn't working area/oversized-allocation labels Jul 6, 2023
@BenPope BenPope self-assigned this Jul 12, 2023
BenPope added a commit to BenPope/redpanda that referenced this issue Jul 12, 2023
vbotbuildovich pushed a commit to vbotbuildovich/redpanda that referenced this issue Jul 14, 2023
graphcareful pushed a commit to graphcareful/redpanda that referenced this issue Aug 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants