[MOD-3251] Stop heartbeats before sending the container snapshot RPC

[mattnappo]

Heartbeats make a connection to the gRPC proxy (modal.sock) which causes the container to open a FD. This open FD causes gVisor to crash during runsc checkpoint. I ran the repro over 300 times with 0 flakes.

[luiscape]

Any way to verify or test?

[thundergolfer]

I think you need a pause-and-restart structure here. This looks like just pause, which means a restored container will be killed for not heartbeating.

[mattnappo]

Update: I have landed on a solution I am happy with, with almost no overhead (@thundergolfer and @luiscape I removed the bottleneck I spoke of). Here is a summary:

• I introduce a asyncio.Condition variable that guards a boolean called waiting_for_memory_snapshot (same as the old name).
• Before a ContainerCheckpoint or ContainerHeartbeat RPC, the Condition must be acquired, and the heartbeat will only run if the container is NOT waiting_for_memory_snapshot. This prevents heartbeat requests from executing while the checkpoint-restore process in in progress.
• If the function is a snapshotting function (function_def.is_checkpointing_fn), then heartbeats will only turn on after the entire checkpoint-restore process
• If it's not a snapshotting function, then heartbeats are enabled immediately, and will run during the container init phase.
• Remove the old ContainerIOManager::_waiting_for_memory_snapshot logic, as it was broken and did not prevent heartbeats from happening when they shouldn't

[mattnappo]

The following invariants hold:

• If snapshotting fn, then heartbeats run after the snap-restore phase
• If snapshotting fn, then heartbeats do not run before the snap-restore phase
• If not snapshotting fn, then heartbeats are always running

[luiscape]

See comment re disabling heartbeats.

[thundergolfer]

🚢