Memory allocated in runtime.asmcgocall is not freed under some circumstances

[vitalyisaev2]

What version of Go are you using (go version)?

go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/isaev/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/isaev/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build093950829=/tmp/go-build -gno-record-gcc-switches"

Problem

I'm not sure if this is really Go runtime's issue, but I have been encouraged by @aclements to open a new issue after short discussion in #16843.

There is a small Go application that uses RocksDB via CGO wrapper library. RocksDB is actually written in C++, but also exposes plain C API. There are two ways of interaction between C++ and Go parts of the application:

1. Go -> C -> C++: Go just calls C API;
2. Go -> C -> C++ -> C -> Go: user-defined part of RocksDB - a MergeOperator - is implemented in Go, and then passed to C++ through function pointer tricks (consider 1, 2). This means that some Go call stacks are not terminated in the C++ part, but pass through C++ part to Go part again.

The problem is in linear memory comsumption growth that occurres at the time of database iteration. I can't find out if it's caused by some of Go runtime optimisations, or Linux just don't take back released memory, or is it a memory leak that happens somewhere between programming languages. The available tools seem not to be sufficient to investigate the reasons of this behavior.

In order to reproduce this issue, please follow the instructions.

Call the application, also turn on go tool pprof (or http://localhost:6060/debug/pprof/heap?debug=1) and htop, and you'll see that:

./mve-go real iterate

...
2018/09/10 19:07:01 Iteration finished 9
2018/09/10 19:07:01 Database stats:
2018/09/10 19:07:01 estimate-num-keys: 6260293
2018/09/10 19:07:01 cur-size-all-mem-tables: 728
2018/09/10 19:07:01 estimate-table-readers-mem: 13159991
2018/09/10 19:07:01 Waiting for a while in order to let runtime free memory
...

1. RSS is about 605 MB;
2. Go runtime's HeapSys is about 65 MB;
3. RocksDB database internal structures (estimate-table-readers-mem) are about 13 MB;

So the question is where are the remaining 605 - (65 + 13) = 527 MB. They are not released even after a half of an our after the iteration stop.

On the chart produced by massif-visualizer we see that something happens in runtime.asmcgocall, but it's hard to find out, what's exactly:

I will appreciate if someone could comment this issue from the point of Go runtime and CGO internals and give an advice about the tools and best practices of debugging.

[aclements]

I haven't used massif, but that looks very much like some Go code is calling C.malloc to allocate on the C heap and simply never freeing that memory. This memory isn't being allocated by runtime.asmcgocall, that's just the bridge for some Go code that's calling C.malloc. Unfortunately, the massif traceback is a bit messed up, so it's not clear what Go code is calling C.malloc (not too surprising considering this call stack jumps back and forth between two stacks).

However, picking out the bits that make some sense, it looks like some Go code called rockdb::DBIter::Next(), which eventually called gorocksdb_mergeoperator_full_merge, which called back into Go code, which then called C.malloc. Does that help with finding the culprit?

[aclements]

Given that this isn't a Go issue, I'm going to go ahead and close this issue, but we can continue discussing here.

[vitalyisaev2]

Thank you @aclements . I agree that something is leaking in RocksDB iterators, and I believe this happens in C++ code (maybe because of MergeOperator implemented in Go). Also, there are no too much options to control these allocations on the side of Go application. There are Free() methods on CGO wrapped structs, and I always call them. Perhaps I should reach gorocksdb maintainers with this issue.

[aclements]

@vitalyisaev2, it's not just that something is being allocated in C++ code. There is specifically some Go code that is specifically calling C.malloc. The allocation is not coming from C++.