Add user document for Antrea native secondary network support

Signed-off-by: Jianjun Shen <shenj@vmware.com>

docs/antrea-ipam.md

@@ -24,7 +24,6 @@
     * [CNI IPAM configuration](#cni-ipam-configuration)
     * [Configuration with `NetworkAttachmentDefinition` CRD](#configuration-with-networkattachmentdefinition-crd)
   * [`IPPool` CRD](#ippool-crd)
-    * [Secondary Network creation with Multus](#secondary-network-creation-with-multus)
 <!-- TOC -->
 
 ## Running NodeIPAM within Antrea Controller
@@ -321,10 +320,9 @@ will get same IP after recreated.
 
 ## IPAM for Secondary Network
 
-With the AntreaIPAM feature, Antrea can allocate IPs for Pod secondary networks. At the
-moment, AntreaIPAM supports secondary networks managed by [Multus](https://github.com/k8snetworkplumbingwg/multus-cni),
-we will add support for [secondary networks managed by Antrea](feature-gates.md#secondarynetwork)
-in the future.
+With the AntreaIPAM feature, Antrea can allocate IPs for Pod secondary networks,
+including both [secondary networks managed by Antrea](secondary-network.md) and
+secondary networks managed by [Multus](cookbooks/multus).
 
 ### Prerequisites
 
@@ -470,13 +468,40 @@ spec:
     prefixLength: 64
 ```
 
-VLAN ID in the IP range subnet definition of `IPPool` CRD is not supported for
-secondary network IPAM.
+When used for Antrea VLAN secondary network, the VLAN set in an `IPPool` IP
+range will be passed to the VLAN secondary interface configuration. For example:
 
-### Secondary Network creation with Multus
+```yaml
+apiVersion: "crd.antrea.io/v1alpha2"
+kind: IPPool
+metadata:
+  name: ipv4-pool-1
+spec:
+  ipVersion: 4
+  ipRanges:
+  - cidr: "10.10.1.0/26"
+    gateway: "10.10.1.1"
+    prefixLength: 24
+    vlan: 100
+
+---
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: ipv4-net-1
+spec:
+  {
+      "cniVersion": "0.3.0",
+      "type": "antrea",
+      "networkType": "vlan",
+      "ipam": {
+          "type": "antrea",
+          "ippools": [ "ipv4-pool-1" ]
+      }
+  }
+```
+
+You can refer to the [Antrea secondary network document](secondary-network.md)
+for more information about Antrea secondary VLAN network configuration.
 
-To leverage Antrea for secondary network IPAM, Antrea must be used as the CNI
-for the Pods' primary network, while the secondary networks are implemented by
-other CNIs which are managed by Multus. The [Antrea + Multus guide](cookbooks/multus)
-talks about how to use Antrea with Multus, including the option of using Antrea
-IPAM for secondary networks.
+For other network types, the VLAN field will be ignored.

docs/secondary-network.md

@@ -0,0 +1,142 @@
+# Antrea Secondary Network Support
+
+Antrea can work with Multus, in which case Antrea is the primary CNI of the
+Kubernetes cluster and provisions the "primary" network interfaces of Pods;
+while Multus manages secondary networks and executes other CNIs to create
+secondary network interfaces of Pods. The [Antrea + Multus guide](cookbooks/multus)
+talks about how to use Antrea with Multus.
+
+Starting with Antrea v1.15, Antrea can also provision secondary network
+interfaces and connect them to VLAN networks. This document describes Antrea's
+native support of VLAN secondary networks.
+
+## Prerequisites
+
+Native secondary network support is still an alpha feature and is disabled by
+default. To use the feature, the `SecondaryNetwork` feature gate must be enabled
+in the `antrea-agent` configuration. If you need IPAM for the secondary
+interfaces, you should also enable the `AntreaIPAM` feature gate in both
+`antrea-agent` and `antrea-controller` configuration. At the moment, Antrea IPAM
+is only available IPAM option for secondary networks managed by Antrea. The
+`antrea-config` ConfigMap with the two feature gates enables is like the
+following:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: antrea-config
+  namespace: kube-system
+data:
+  antrea-controller.conf: |
+    featureGates:
+      AntreaIPAM: true
+  antrea-agent.conf: |
+    featureGates:
+      AntreaIPAM: true
+      SecondaryNetwork: true
+```
+
+Antrea leverages the `NetworkAttachmentDefinition` CRD from [Kubernetes Network
+Plumbing Working Group](https://github.com/k8snetworkplumbingwg/multi-net-spec)
+to define secondary networks. You can import the CRD to your cluster using the
+following command:
+
+```bash
+kubectl apply -f https://github.com/k8snetworkplumbingwg/network-attachment-definition-client/raw/master/artifacts/networks-crd.yaml
+```
+
+## Secondary OVS bridge configuration
+
+A VLAN secondary interface will be connected to a separate OVS bridge on the
+Node. You can specify the secondary OVS bridge configuration in the
+`antrea-agent` configuration, and `antrea-agent` will automatically craete the
+OVS bridge on based on the configuration. For example, the following
+configuration will create an OVS bridge named `br-secondary`, with a physical
+interface `eth1`.
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: antrea-config
+  namespace: kube-system
+data:
+  antrea-agent.conf: |
+    secondaryNetwork:
+    ovsBridges: [{"bridgeName": "br-secondary", "physicalInterfaces": ["eth1"]}]
+```
+
+At the moment, Antrea supports only a single OVS bridge for secondary networks,
+and supports only a single physical interface on the bridge. The physical
+interface cannot be the Node's management interface, otherwise a Node's
+management network connectivity can be broken after `antrea-agent` creates the
+OVS bridge and moves the management interface to the bridge.
+
+## Secondary VLAN network configuration
+
+A secondary VLAN network is defined by a NetworkAttachmentDefinition CR. For
+example, the following NetworkAttachmentDefinition defines a VLAN network
+`vlan100`.
+
+```yaml
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: vlan100
+spec:
+  config: '{
+      "cniVersion": "0.3.0",
+      "type": "antrea",
+      "networkType": "vlan",
+      "mtu": 1500,
+      "vlan": 100,
+      "ipam": {
+        "type": "antrea",
+        "ippools": ["vlan100-ipv4", "vlan100-ipv6"]
+      }
+    }'
+```
+
+`antrea-agent` will connect a Pod secondary interface of a VLAN network to the
+secondary OVS bridge on the Node. If a non-zero VLAN is speficied in the
+network's `config`, `antrea-agent` will configure the VLAN ID to the OVS port,
+so the interface's traffic will be isolated within the VLAN. And before the
+traffic is forwarded out the Node via the secondary bridge's physical interface,
+OVS will insert the VLAN tag to the packets.
+
+A few extra notes about the NetworkAttachmentDefinition `config` fields:
+
+* `type` - must be set to `antrea`.
+* `networkType` - the only supported network type is `VLAN` as of now.
+* `mtu` - defaults to 1500 is not set.
+* `vlan` - can be set to 0 or a valid VLAN ID (1 - 4094). Defaults to 0. Also
+note, VLAN can be speficied in the spec of an IPPool in the `ipam` section too;
+but `vlan` in NetworkAttachmentDefinition `config` will override the VLAN in
+IPPool(s) if both are set.
+* `ipam` - it is optional. If not set, the secondary interfaces created for the
+network won't have an IP address allocated. For more information about secondary
+network IPAM configuration, please refer to the [Antrea IPAM document](antrea-ipam.md#ipam-for-secondary-network).
+
+## Pod secondary interface configuration
+
+You can create a Pod with secondary network interfaces by adding the
+`k8s.v1.cni.cncf.io/networks` annotation to the Pod. The following example Pod
+includes two secondary interfaces, one in network `vlan100` which should be
+created in the same Namespace as the Pod, the other in network `vlan200` which
+is created in Namespace `default`.
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: sample-pod
+ labels:
+   app: antrea-secondary-network-demo
+ annotations:
+   k8s.v1.cni.cncf.io/networks: [vlan100, {name: vlan200, namespace: default, interface: eth200"]
+spec:
+ containers:
+ - name: toolbox
+   image: antrea/toolbox:latest
+```

hack/.notableofcontents

@@ -39,5 +39,6 @@ docs/octant-plugin-installation.md
 docs/os-issues.md
 docs/ovs-offload.md
 docs/prometheus-integration.md
+docs/secondary-network.md
 docs/security.md
 docs/traffic-encryption.md