Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory leake with zlib compression optional. #629

Closed
superwood opened this issue Jun 7, 2015 · 2 comments
Closed

Memory leake with zlib compression optional. #629

superwood opened this issue Jun 7, 2015 · 2 comments
Labels
abandoned-or-aged-out

Comments

@superwood
Copy link

The size of rocksdb (RocksDB version: 3.11.0) is 300G . The memory used by rocksdb keeps going up over time while compacting. Our OOM killer kills the process when it hits 64GB. while optionals about compression reseted with snappy , every thing is ok.

the stack message is
#0 0x00000039f040e6fd in write () from /lib64/libpthread.so.0
#1 0x000000000053bace in rocksdb::(anonymous namespace)::PosixWritableFile::Flush (this=0x5eb2a00)

at util/env_posix.cc:790
#2 0x00000000005946ba in rocksdb::BlockBasedTableBuilder::Flush (this=Unhandled dwarf expression opcode 0xf3

) at table/block_based_table_builder.cc:584
#3 0x0000000000594870 in rocksdb::BlockBasedTableBuilder::Add (this=0x3cb9af400, key=..., value=...)

at table/block_based_table_builder.cc:546
---Type to continue, or q to quit---
#4 0x00000000005761bd in rocksdb::CompactionJob::ProcessKeyValueCompaction (this=0x7f373ddb6d00,

imm_micros=0x7f373ddb5f78, input=0x43a0ba1c0, is_compaction_v2=false) at db/compaction_job.cc:879
#5 0x00000000005795f2 in rocksdb::CompactionJob::Run (this=0x7f373ddb6d00) at db/compaction_job.cc:360
#6 0x00000000004c0335 in rocksdb::DBImpl::BackgroundCompaction (this=0x2a08000, madeProgress=0x7f373ddb714e,

job_context=0x7f373ddb7170, log_buffer=0x7f373ddb7330) at db/db_impl.cc:2395
#7 0x00000000004cbedf in rocksdb::DBImpl::BackgroundCallCompaction (this=0x2a08000) at db/db_impl.cc:2106
#8 0x000000000053b60f in BGThread (arg=Unhandled dwarf expression opcode 0xf3

) at util/env_posix.cc:1702
#9 rocksdb::(anonymous namespace)::PosixEnv::ThreadPool::BGThreadWrapper (arg=Unhandled dwarf expression opcode 0xf3

here is the options

[WARN] Options.create_if_missing: 1
[WARN] Options.paranoid_checks: 1
[WARN] Options.env: 0x839600
[WARN] Options.info_log: 0x29f8000
[WARN] Options.max_open_files: 5000
[WARN] Options.max_total_wal_size: 0
[WARN] Options.disableDataSync: 0
[WARN] Options.use_fsync: 0
[WARN] Options.max_log_file_size: 536870912
[WARN] Options.max_manifest_file_size: 18446744073709551615
[WARN] Options.log_file_time_to_roll: 0
[WARN] Options.keep_log_file_num: 1000
[WARN] Options.allow_os_buffer: 1
[WARN] Options.allow_mmap_reads: 0
[WARN] Options.allow_mmap_writes: 0
[WARN] Options.create_missing_column_families: 0
[WARN] Options.db_log_dir: ./log/
[WARN] Options.wal_dir: /ssd/data/morpheus/userprofile
[WARN] Options.table_cache_numshardbits: 4
[WARN] Options.delete_obsolete_files_period_micros: 21600000000
[WARN] Options.max_background_compactions: 2
[WARN] Options.max_background_flushes: 2
[WARN] Options.WAL_ttl_seconds: 0
[WARN] Options.WAL_size_limit_MB: 0
[WARN] Options.manifest_preallocation_size: 4194304
[WARN] Options.allow_os_buffer: 1
[WARN] Options.allow_mmap_reads: 0
[WARN] Options.allow_mmap_writes: 0
[WARN] Options.is_fd_close_on_exec: 1
[WARN] Options.stats_dump_period_sec: 600
[WARN] Options.advise_random_on_open: 1
[WARN] Options.db_write_buffer_size: 0
[WARN] Options.access_hint_on_compaction_start: NORMAL
[WARN] Options.use_adaptive_mutex: 0
[WARN] Options.rate_limiter: (nil)
[WARN] Options.bytes_per_sync: 0
[WARN] Options.wal_bytes_per_sync: 0
[WARN] Options.enable_thread_tracking: 0
Compression algorithms supported:
Snappy supported: 1
Zlib supported: 1
Bzip supported: 1
LZ4 supported: 1
Recovering from manifest
--------------- Options for
[WARN] Options.error_if_exists: 0
[WARN] Options.create_if_missing: 1
[WARN] Options.paranoid_checks: 1
[WARN] Options.env: 0x839600
[WARN] Options.info_log: 0x29f8000
[WARN] Options.max_open_files: 5000
[WARN] Options.max_total_wal_size: 0
[WARN] Options.disableDataSync: 0
[WARN] Options.use_fsync: 0
[WARN] Options.max_log_file_size: 536870912
[WARN] Options.max_manifest_file_size: 18446744073709551615
[WARN] Options.log_file_time_to_roll: 0
[WARN] Options.keep_log_file_num: 1000
[WARN] Options.allow_os_buffer: 1
[WARN] Options.allow_mmap_reads: 0
[WARN] Options.allow_mmap_writes: 0
[WARN] Options.create_missing_column_families: 0
[WARN] Options.db_log_dir: ./log/
[WARN] Options.wal_dir: /ssd/data/morpheus/userprofile
[WARN] Options.table_cache_numshardbits: 4
[WARN] Options.delete_obsolete_files_period_micros: 21600000000
[WARN] Options.max_background_compactions: 2
[WARN] Options.max_background_flushes: 2
[WARN] Options.WAL_ttl_seconds: 0
[WARN] Options.WAL_size_limit_MB: 0
[WARN] Options.manifest_preallocation_size: 4194304
[WARN] Options.allow_os_buffer: 1
[WARN] Options.allow_mmap_reads: 0
[WARN] Options.allow_mmap_writes: 0
[WARN] Options.is_fd_close_on_exec: 1
[WARN] Options.stats_dump_period_sec: 600
[WARN] Options.advise_random_on_open: 1
[WARN] Options.db_write_buffer_size: 0
[WARN] Options.access_hint_on_compaction_start: NORMAL
[WARN] Options.use_adaptive_mutex: 0
[WARN] Options.rate_limiter: (nil)
[WARN] Options.bytes_per_sync: 0
[WARN] Options.wal_bytes_per_sync: 0
[WARN] Options.enable_thread_tracking: 0
[WARN] Options.comparator: rocksdb.InternalKeyComparator:leveldb.BytewiseComparator
[WARN] Options.merge_operator: rocks_merge
[WARN] Options.compaction_filter: None
[WARN] Options.compaction_filter_factory: DefaultCompactionFilterFactory
[WARN] Options.compaction_filter_factory_v2: DefaultCompactionFilterFactoryV2
[WARN] Options.memtable_factory: HashLinkListRepFactory
[WARN] Options.table_factory: BlockBasedTable
[WARN] table_factory options:

[WARN] Options.write_buffer_size: 536870912
[WARN] Options.max_write_buffer_number: 4
[WARN] Options.compression[0]: NoCompression
[WARN] Options.compression[1]: Snappy
[WARN] Options.compression[2]: Snappy
[WARN] Options.compression[3]: Zlib
[WARN] Options.compression[4]: Zlib
[WARN] Options.compression[5]: Zlib
[WARN] Options.compression[6]: Zlib
[WARN] Options.prefix_extractor: rocksdb.Noop
[WARN] Options.num_levels: 7
[WARN] Options.min_write_buffer_number_to_merge: 2
[WARN] Options.purge_redundant_kvs_while_flush: 1
[WARN] Options.compression_opts.window_bits: -14
[WARN] Options.compression_opts.level: -1
[WARN] Options.compression_opts.strategy: 0
[WARN] Options.level0_file_num_compaction_trigger: 4
[WARN] Options.level0_slowdown_writes_trigger: 20
[WARN] Options.level0_stop_writes_trigger: 48
[WARN] Options.max_mem_compaction_level: 2
[WARN] Options.target_file_size_base: 134217728
[WARN] Options.target_file_size_multiplier: 1
[WARN] Options.max_bytes_for_level_base: 1073741824
[WARN] Options.level_compaction_dynamic_level_bytes: 0
[WARN] Options.max_bytes_for_level_multiplier: 10
[WARN] Options.max_bytes_for_level_multiplier_addtl[0]: 1
[WARN] Options.max_bytes_for_level_multiplier_addtl[1]: 1
[WARN] Options.max_bytes_for_level_multiplier_addtl[2]: 1
[WARN] Options.max_bytes_for_level_multiplier_addtl[3]: 1
[WARN] Options.max_bytes_for_level_multiplier_addtl[4]: 1
[WARN] Options.max_bytes_for_level_multiplier_addtl[5]: 1
[WARN] Options.max_bytes_for_level_multiplier_addtl[6]: 1
[WARN] Options.max_sequential_skip_in_iterations: 8
[WARN] Options.expanded_compaction_factor: 25
[WARN] Options.source_compaction_factor: 1
[WARN] Options.max_grandparent_overlap_factor: 10
[WARN] Options.arena_block_size: 53687091
[WARN] Options.soft_rate_limit: 0.00
[WARN] Options.hard_rate_limit: 0.00
[WARN] Options.rate_limit_delay_max_milliseconds: 1000
[WARN] Options.disable_auto_compactions: 0
[WARN] Options.purge_redundant_kvs_while_flush: 1
[WARN] Options.filter_deletes: 1
[WARN] Options.verify_checksums_in_compaction: 1
[WARN] Options.compaction_style: 0
[WARN] Options.compaction_options_universal.size_ratio: 1
[WARN] Options.compaction_options_universal.min_merge_width: 2
[WARN] Options.compaction_options_universal.max_merge_width: 4294967295
[WARN] Options.compaction_options_universal.max_size_amplification_percent: 200
[WARN] Options.compaction_options_universal.compression_size_percent: -1
[WARN] Options.compaction_options_fifo.max_table_files_size: 1073741824
[WARN] Options.table_properties_collectors:
[WARN] Options.inplace_update_support: 1
[WARN] Options.inplace_update_num_locks: 10000
@mdcallag
Copy link
Contributor

mdcallag commented Jun 7, 2015

Two things have confused me in the past about RocksDB memory usage. The first is that if metadata (index and bloom filter) blocks are not in the block cache then you can use more memory as you keep more files open when the database grows. You don't report the option ("cache_index_and_filter_blocks").

The other problem was a bug where the size of memory allocated per block in the block cache was much larger than the uncompressed block and this was a big issue for zlib, but not so much for snappy. With format_version=2 the problem is fixed.

What is the value of format_version option? It is printed in LOG. The default is 0 and the best value is 2. Alas, this is a data format change, so the default stays at 0.

Read the comments for it:
https://github.com/facebook/rocksdb/blob/master/include/rocksdb/table.h#L144

And the diff:
9ab5adf

@gfosco
Copy link
Contributor

gfosco commented Jan 10, 2018

Closing this via automation due to lack of activity. If discussion is still needed here, please re-open or create a new/updated issue.

@gfosco gfosco closed this as completed Jan 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
abandoned-or-aged-out
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gfosco @mdcallag @superwood