Debug: write memory profile if heap exceeds threshold

[jtibshirani]

This PR adds adds a debugging flag to periodically check memory usage against a threshold. If it exceeds the threshold, then a memory profile like indexmemory.prof.1 is written to disk. No more than 10 profiles will be written.

I've already found this more useful than the existing -memprofile flag, so I removed that. It's hard to get insights using that flag, since it only takes a single profile per shard, forces GC, and forces parallelism to 1.

[jtibshirani]

Leaving this as a draft for now, since I need to double-check some the concurrency logic. And I'm not sure it will be useful now that we have GCP profiling for zoekt-git-index (#816).

I took it for a spin while indexing sgtest/megarepo locally, and did notice some interesting things like much higher alloc_space than inuse_space:

top alloc_space

Showing top 10 nodes out of 101
      flat  flat%   sum%        cum   cum%
 1472.87MB 24.87% 24.87%  1472.87MB 24.87%  github.com/go-git/go-git/v5/plumbing.(*MemoryObject).Write
 1174.62MB 19.84% 44.71%  1174.62MB 19.84%  bytes.growSlice
  551.09MB  9.31% 54.01%   551.09MB  9.31%  bufio.NewReaderSize (inline)
  536.84MB  9.07% 63.08%   536.84MB  9.07%  github.com/go-git/go-git/v5/plumbing/format/idxfile.(*MemoryIndex).genOffsetHash
  429.91MB  7.26% 70.34%   429.91MB  7.26%  github.com/sourcegraph/zoekt.(*postingsBuilder).newSearchableString
  302.38MB  5.11% 75.44%   302.38MB  5.11%  github.com/go-git/go-git/v5/plumbing/format/idxfile.readObjectNames
  268.23MB  4.53% 79.97%   452.75MB  7.65%  github.com/sourcegraph/go-ctags.(*ctagsProcess).Parse
  185.17MB  3.13% 83.10%  1215.74MB 20.53%  github.com/sourcegraph/zoekt/gitindex.prepareNormalBuild
   96.95MB  1.64% 84.74%  4876.95MB 82.36%  github.com/sourcegraph/zoekt/gitindex.indexGitRepo
   83.80MB  1.42% 86.15%    83.80MB  1.42%  github.com/sourcegraph/zoekt/gitindex.(*repoWalker).handleEntry

top inuse_space

Showing top 10 nodes out of 68
      flat  flat%   sum%        cum   cum%
  806.24MB 36.69% 36.69%   806.24MB 36.69%  bytes.growSlice
  529.86MB 24.11% 60.81%   529.86MB 24.11%  github.com/go-git/go-git/v5/plumbing/format/idxfile.(*MemoryIndex).genOffsetHash
  302.38MB 13.76% 74.57%   302.38MB 13.76%  github.com/go-git/go-git/v5/plumbing/format/idxfile.readObjectNames
  126.53MB  5.76% 80.33%   126.53MB  5.76%  github.com/sourcegraph/zoekt.(*postingsBuilder).newSearchableString
  101.30MB  4.61% 84.94%   101.30MB  4.61%  github.com/go-git/go-git/v5/plumbing.(*MemoryObject).Write
   89.88MB  4.09% 89.03%   642.24MB 29.23%  github.com/sourcegraph/zoekt/gitindex.prepareNormalBuild
   58.84MB  2.68% 91.71%    58.84MB  2.68%  github.com/go-git/go-git/v5/plumbing/format/idxfile.readOffsets
   49.35MB  2.25% 93.95%  1983.50MB 90.27%  github.com/sourcegraph/zoekt/gitindex.indexGitRepo
      30MB  1.37% 95.32%       30MB  1.37%  encoding/json.(*decodeState).literalStore
   29.50MB  1.34% 96.66%    29.50MB  1.34%  github.com/sourcegraph/zoekt/build.(*tagsToSections).Convert

[keegancsmith]

Our use of git is quite vanilla, I wonder if we should instead rely on just shelling out to git? Maybe there is a wrapper which just relies on things like git cat-file? Or we can get cody to implement a go-git.Storer which just does read operations via git cat-file.