Possible memory leak when iterating over database with MergeOperator enabled

[vitalyisaev2]

We faced with memory leak in one of our applications, which is written in Go and uses RocksDB 5.13 via cgo interface (https://github.com/tecbot/gorocksdb). The application periodically iterates over whole RocksDB nstance and reads data sequentially key by key. It turns out that the memory allocated after every iteration is never freed.

Please visit this repo for the further information: https://github.com/vitalyisaev2/gorocksdb_merge_operator_issue
I will appreciate if somebody could point out, what's wrong with our database instance (database dump provided) or the application code.

[ajkr]

Also reported tecbot/gorocksdb#144.

I'll probably wait on them to rule out a problem with your Go application or the Go bindings, unless you want to try providing a repro using RocksDB directly in C or C++.

[vitalyisaev2]

@ajkr I've added C++ example and as far as I can see there is no memory leak in it.

Though I'm not totally sure if my implementations of MergeOperator in Go and C++ are equivalent: in Go it concatenates []byte slices via built-in append function, while in C++ it uses string::append (which I believe also implies memory allocations and copying). Providing that these operations are really equal, we can conclude that problem is somewhere in Go runtime.

Also I would like to ask you if there are any tools/methods/techniques to check the integrity of database structure. The data within a problematic dump is not corrupted, it just causes strange behavior of Go application.

[vitalyisaev2]

I still have no idea what has happened to the database, but if I trigger whole database compaction manually at the start of the application, the leak is no longer observed.

Like this:

db.CompactRange(gorocksdb.Range{})

I thinks that there is some non-obvious and strange relationship between state of the database and the behavior of the iterators accessed through the CGO wrappers.

[vitalyisaev2]

Sorry, but it is still an issue. Even after manual compaction, after several hours, the linear memory growth comes back.

[ajkr]

To check integrity: first use DB::Open(), which will process WALs and MANIFEST. It'll return non-OK status if there's any problem. Then, call DB::VerifyChecksum(), which will verify checksums in all blocks of all SSTs and will let you know about any corruptions.

[ajkr]

For your primary issue: I'm not sure I can be much help if the equivalent program written in C++ doesn't repro the issue. Maybe chase down the go binding maintainers? :)

[vitalyisaev2]

@ajkr thanks for the tips! Going to check database integrity first