Antrea supports setting network interface for inter-host communication

[Jexf]

Describe the problem/challenge you have

// GetNodeAddr gets the available IP address of a Node. GetNodeAddr will first try to get the NodeInternalIP, then try
// to get the NodeExternalIP.
// Note: Although K8s supports dual-stack, there is only a single Internal address per Node because of issue (
// kubernetes/kubernetes#91940 ). The Node might have multiple addresses after the issue is fixed, and one per address
// family. And we should change the return type at that time.
func GetNodeAddr(node *v1.Node) (net.IP, error) {
	addresses := make(map[v1.NodeAddressType]string)
	for _, addr := range node.Status.Addresses {
		addresses[addr.Type] = addr.Address
	}
	var ipAddrStr string
	if internalIP, ok := addresses[v1.NodeInternalIP]; ok {
		ipAddrStr = internalIP
	} else if externalIP, ok := addresses[v1.NodeExternalIP]; ok {
		ipAddrStr = externalIP
	} else {
		return nil, fmt.Errorf("Node %s has neither external ip nor internal ip", node.Name)
	}
	ipAddr := net.ParseIP(ipAddrStr)
	if ipAddr == nil {
		return nil, fmt.Errorf("<%v> is not a valid ip address", ipAddrStr)
	}
	return ipAddr, nil
}

Antrea gets NodeInternalIP or NodeExternalIP for inter-host communication, sometimes we need set special network interface for inter-host communication.

Describe the solution you'd like
Maybe need antrea-agent --config /etc/antrea/antrea-agent.conf --iface eno1 or antrea-agent --config /etc/antrea/antrea-agent.conf --iface-regex 172.16.100.0/24

Use iface or iface-regex, which config param is not empty, when setting up node inter-host communication ip addr, otherwise use NodeInternalIP or NodeExternalIP.

[tnqn]

@Jexf thanks for the suggestion. This is definitely a valid scenario Antrea should support. We received similar feature suggestion in #2344, though it's for windows.
@wenyingd has #2370 (not merged yet) which should covers the scenario. It adds an option "transportInterface" to agent configuration file. Would you check if it will work for you?

[Jexf]

Thanks for reply @tnqn, the patch #2370 adds an option "transportInterface" to config interface name, maybe we need add option "transportCIDRRange" for different interface name on each node.

[tnqn]

Good idea. @jianjuns @antoninbas what's your opinion?
@Jexf would you like to create a PR for it after #2370 is merged?

[jianjuns]

I did not get the idea. transportCIDRRange is for supporting different interface names on different Nodes? How it helps?

[antoninbas]

I did not get the idea. transportCIDRRange is for supporting different interface names on different Nodes? How it helps?

I think it's to identify the interface based on its subnet and not based on its name which can vary across Nodes.
I understand the problem we are trying to solve but this doesn't seem like a very elegant solution, especially when not all Nodes are on the same L2 network. Can you think of any other project using a solution like this one? Any alternatives we could consider?

[Jexf]

A k8s-antrea has three nodes:

NAME     STATUS   ROLES                  AGE   VERSION
tos-04   Ready    control-plane,master   15d   v1.21.2
tos-05   Ready    <none>                 15d   v1.21.2
tos-06   Ready    <none>                 15d   v1.21.2

The interface for inter-host communicatio on tos-04 is:

[root@tos-04 ~]# ip a s eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:bd:db:31 brd ff:ff:ff:ff:ff:ff
    inet 172.18.123.15/22 brd 172.18.123.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:febd:db31/64 scope link
       valid_lft forever preferred_lft forever

The interface for inter-host communicatio on tos-05 is:

[root@tos05 ~]# ip a s eno1
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 3c:ec:ef:4e:15:06 brd ff:ff:ff:ff:ff:ff
    inet 172.18.123.16/22 brd 172.18.123.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::3eec:efff:fe4e:1506/64 scope link
       valid_lft forever preferred_lft forever

We can update transportInterface to transportInterfaces, and set the transportInterfaces to eth1,eno1`, then antrea match the first interface from transportInterfaces list. It will work well with different interface name for inter-host communication. @tnqn @jianjuns

The subnet option for inter-host communication only work on L2 network. Maybe we can add an option with []subnet, then antrea will match the first interface from subnets list. @antoninbas

[jianjuns]

Interface or subnet list do not sound ideal either, as two Nodes can have the same set of interfaces or subnets, but want to choose different ones.

If we step back, do you think you can create two Antrea DaemonSets (each should select different Nodes) for the two Nodes in your example, with different transportInterface configuration? It is not very convenient but should work?