Investigate codebase for memory problems

[qdm12]

This is more of low hanging fruits that can be worth checking/fixing before investing more time in profiling and metrics, to fix the current memory issue we have (2021-11-03).

• Check usage of bytes.Buffer (local branch)
• Check usage of pools
• Check usage of copy
• Check buffered channels (using regex make\(chan .+, [1-9].+\))
• Check DeepCopy functions (search for deepcopy without case sensitivity)
• Check usage of sizedBufferPool - found a problem, fixed in fix(dot/network): fix memory allocations with sizedBufferPool #1963, although not a reason for OOM
• Check for arrays returned (copy, it's not a slice), use regex \[[a-zA-Z0-9]+\](byte|int|uint|uint32|uint64|int32|int64|string)+?
• Check packages
  • dot/network
  • lib/trie
  • dot/sync

[qdm12]

The current state as of today is:

• the trie works fine and does not leak (probably thanks to fix(trie): remove encoding buffers pool #2929 and trie: no in-memory caching of node encoding #2919)
• Node running the Kusama chain does not cleanup tries in the in-memory map of tries after syncing 1.5M blocks
• Node running the Polkadot chain seems to use ~10MB per day of sync (= ~1M blocks) so that should be fine
• On the polkadot chain, leaks now come from the badgerdb implementation (see Create internal/database/badger package and drop chaindb #2982) and the hashToRuntime map (see Clean Up Old Runtimes on Canonical Chain #2767)