Memory usage – default allocator replacement

[Wiezzel]

Recently, inspired by @nazar-pc I've decided to replace the default allocator for my rust-libp2p-based binaries, and it led to a significant reduction of memory footprint. Using jemallocator I have observed the following results:

• bootnode (only rust-libp2p code + main function): 170 MiB -> 75 MiB,
• logs-collector (rust-libp2p + some DB operations): 320 MiB -> 60 MiB,
• gateway (rust-libp2p + axum webserver): 640 MiB -> 80 MiB,
• scheduler (rust-libp2p + axum + some s3 operations): 670 MiB -> 120 MiB,
• worker (rust-libp2p + some data-heavy logic): 16 GiB -> 400-800 MiB.

This is a really big improvement. Similar results were achieved with mimalloc. Most binaries have some custom logic, but the bootnode is 99% rust-libp2p code. So it seems that perhaps we should have an official recommendation for rust-libp2p to swap out the default allocator?
@jxs

[nazar-pc]

I think it was always the unwritten recommendation for production apps.
It also has a huge impact when embedded databases (like RocksDb) are used, glibc behaves pretty bad sometimes and is very unwilling to release memory back to the kernel.

[Wiezzel]

I think it was always the unwritten recommendation for production apps.

Maybe let's make it a written recommendation then. Otherwise, newcomers might not get it.

[drHuangMHT]

I think Allocator is still nightly(like for 8 years)? Probably not appropriate now.

[Wiezzel]

WDYM? You can use custom allocators in stable Rust.

[dariusc93]

I think he means the Allocator api under nightly feature allocator_api while GlobalAlloc is in stable (which what allocators like jemalloc, etc., uses iirc)

[Wiezzel]

I have recently started observing that RAM usage goes up again, even though I'm using jemalloc now. Analyzing the application with heaptrack gave the following result:

peak heap memory consumption:
217,6kB after 00.081s
peak RSS (including heaptrack overhead):
3,5GB

I am puzzled by the huge difference between peak RSS and heap memory consumption. The heaptrack overhead shouldn't exceed a few hundred MB, so it looks like jemalloc is keeping a couple GB of unused memory and not returning it to the system. 🤔

[dariusc93]

Could you provide the heaptrack file? I wonder if it could be vecs and maps that need to let go of the memory. I know in the past when ive observed the memory, quic would be the main one that had high memory usage, with kad being next (though kad can still get pretty high when bootstrapping the public network without connection limits set). Excluding external deps, i wonder if we should also make it a practice of shrinking them in rust-libp2p, which might help.

[Wiezzel]

0GzT5.gz
There you go.