Deleting Antrea should do a complete cleanup

[antoninbas]

Describe the bug
When running kubectl delete -f antrea.yml, one would expect everything created by Antrea to be deleted, including the OVS bridge on each Node, along with the gateway interface. Failure do so may cause networking issues if someone decides to deploy another CNI after deleting Antrea.

Versions:
Antrea version:

===> Version information <===
VERSION: v0.2.0-dev
GIT_SHA: 4a446d4
GIT_TREE_STATE: clean
RELEASE_STATUS: unreleased
DOCKER_IMG_VERSION: v0.2.0-dev-4a446d4

Additional context

1. We cannot simply delete the bridge, etc. when the agent exists. For example, we may want to preserve connectivity during some Antrea upgrades. Note that currently this is not possible since all flows are deleted when the openvswitch daemon restarts. See these resources:

• email thread: https://mail.openvswitch.org/pipermail/ovs-discuss/2019-November/049448.html
• submitted patch: https://mail.openvswitch.org/pipermail/ovs-dev/2019-November/364811.html

2. @jianjuns and I discussed this cleanup issue in a separate PR: Fix Kind support (Linux hosts only) #137 (comment). He suggested having a cleanup daemonset dedicated to this purpose. See also these K8s issues:

• Pod or Job lifecycle lacks of a mechanism to define cleanup actions once you delete a pod kubernetes/kubernetes#35183
• [Proposal] deferContainers - Pod Termination Semantics kubernetes/community#483

[jianjuns]

I think we could either have a separate DaemonSet or add a ConfigMap option to indicate antrea-agent to do the cleanup (the two approaches are not very different from each other, but with the ConfigMap way agent could watch the option in runtime when we support runtime config changes later).

[antoninbas]

I'd like to start working on this. I like the DaemonSet approach more since it works even if there is an issue (e.g. buggy agent) or if Antrea has already been deleted from the cluster. Someone might have done kubectl delete -f antrea.yml already, and we need to be able to perform the cleanup without the Antrea agent / controller running.

My proposal is as follows:

• define a YAML manifest antrea-cleanup.yaml to take care of cleaning up artifacts created by Antrea
• the manifest would define one K8s Job in charge of performing the cleanup, plus necessary RBAC permissions
• the Job would be in charge of creating the cleanup DaemonSet, monitoring the status of the cleanup process and deleting the DaemonSet once cleanup is complete

A few considerations:

• there is no notion of a DaemonSet that runs to completion and doesn't restart Pods (see Run job on each node once to help with setup kubernetes/kubernetes#64623). But we can use a Controller and have each DaemonSet Pod return completion status to the Controller using CRDs.
• the Job should probably return an error if Antrea is still running (and recommend deleting Antrea with kubectl delete -f antrea.yml). Although another option is to have the Job also delete all K8s resources with label app=antrea. I'm open to both solutions.
• the DaemonSet needs to be aware of the Antrea agent configuration (as the OVS bridge name / gateway interface name may not be the default ones). This is a bit tricky: if Antrea is not running anymore, the ConfigMap is not available anymore. Rather than ask the user to modify a ConfigMap in the cleanup manifest (antrea-cleanup.yaml), I would rather modify the Antrea agent to have it save its config (antrea-agent.conf) on the host under /var/run/antrea/. I think this is a very non-intrusive change, very much in spirit with how we use /var/run/antrea/ for other things (e.g. OVSDB) that we do not need to persist across reboots.

@jianjuns @salv-orlando @tnqn any thoughts or concerns?

[jianjuns]

Questions:

1. Do you plan to use agent bin to do the clean up or a separate bin?
2. For shared resources like Controller CRD who will do the clean up?

[antoninbas]

1. No, I was planning to use different binaries (one for the DaemonSet and one for the "orchestrator", which will be run by the K8s Job and talk to the apiserver).
2. I think the "orchestrator" could take care of this easily, what do you think?

[jianjuns]

I was thinking about running the same agent and controller bins, to avoid introducing new bins. But I saw there are benefits of using a separate bin too, that could be executed standalone, including executed manually. In this sense, do you think it is good to make it part of CLI, and so user could run the CLI manually to clean up?
Also note on Windows, we could not clean up Node state with DS.

[jianjuns]

Just to add, it might be helpful to be able to clean up node after K8s is down.

[antoninbas]

I think it's better to introduce new bins as I like the idea of separating the cleanup logic from the core functionality (rather than adding a new mode of operation for existing binaries).

I definitely think we should have the ability to run the binary directly as a process. I would like to organize the code as a library that can be used by the CLI & or in a DaemonSet. Even though kubectl apply -f antrea-cleanup.yaml may not work on Windows, it's a nice K8s-native way to do cleanup that doesn't require to download any binaries manually.

[antoninbas]

Removing this from 0.3.0, as PR #307 still needs a lot of work.

[salv-orlando]

Unfortunately there weren't many review in the past few days. @antoninbas do you think we can still make a final push for 0.5.0 inclusion, or do we just move this issue to 0.6.0?

[antoninbas]

Pushing this out to next release

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days