Abandoning thousands of commits runs out of memory

[cormacrelf]

Description

Abandoning lots of commits at once seems to have unbounded memory use and cannot finish. This has bricked a repo of mine because you can't undo deleting a git remote, jj must abandon them all before it can do anything else.

Steps to Reproduce the Problem

1. git clone URL and jj git init --colocate
2. git remote add other_repo OTHERURL
3. jj git fetch --all-remotes
4. jj log so it picks it up.
5. git remote rm other_repo
6. jj log (it says abandoning 2000 commits that are no longer reachable, and hangs there)

Expected Behavior

The computer does not run out of memory

Actual Behavior

Huge freeze, jj using 8GB of RAM last time the computer managed to render a frame with htop visible. I managed to record a sample, and jj is rebasing all descendants, ad infinitum. It eventually got a SIGKILL.

Running again on a quiet system, memory use grows and shrinks in cycles -- 10, 11, 12, 13, 14, 11, 12, 13, 15, 12, 13, 14, 16, ... 17, ... 18GB. Now up to 26GB. Gets stuck in swap and then gets killed.

To me, it looks like the commits are abandoned one by one from the oldest onwards, and then every single time it abandons one, it rebases the rest. And does this all in-memory. If that is the case, is it possible to do them in reverse order? Or figure out which branches are going to need rebasing, and wait until all abandonments have completed before attempting to rebase them?

Specifications

• Platform: Linux
• Version: 0.20

[cormacrelf]

Tried this where the second remote is https://github.com/facebook/buck2 and it doesn't happen. I think you need a repo with a lot of merge commits, which facebook's exported git repos do not.

[cormacrelf]

There is a workaround:

jj log --ignore-working-copy -r 'remote_branches(glob:"*", "other_repo") & mutable()' --no-graph -T 'concat(commit_id, "\n")' > /tmp/axe.txt

< /tmp/axe.txt xargs -n10 jj --ignore-working-copy abandon

[yuja]

Maybe you had to use jj git remote remove at the step 5. If remote-tracking branches were removed externally by git, the next jj command would import the changes as if the deleted branches were pulled from the git remote.

Regarding memory usage, I think the main problem is that our caching layer doesn't yet have upper bounds to purge old commit/tree entries. It'll need a simple LRU or something.