diff --git a/docs/multicluster/api.md b/docs/multicluster/api.md
index f61a584d8b0..6ff2cdb3ef4 100644
--- a/docs/multicluster/api.md
+++ b/docs/multicluster/api.md
@@ -16,6 +16,8 @@ which is defined by Kubernetes upstream [KEP-1645](https://github.com/kubernetes
| `MemberClusterAnnounces` | v1alpha1 | v1.5.0 | N/A | N/A |
| `ResourceExports` | v1alpha1 | v1.5.0 | N/A | N/A |
| `ResourceImports` | v1alpha1 | v1.5.0 | N/A | N/A |
+| `Gateway` | v1alpha1 | v1.7.0 | N/A | N/A |
+| `ClusterInfoImport` | v1alpha1 | v1.7.0 | N/A | N/A |
## CRDs in `multicluster.x-k8s.io`
diff --git a/docs/multicluster/assets/sample-clusterset.svg b/docs/multicluster/assets/sample-clusterset.svg
new file mode 100644
index 00000000000..5418bd09bfa
--- /dev/null
+++ b/docs/multicluster/assets/sample-clusterset.svg
@@ -0,0 +1,565 @@
+
+
diff --git a/docs/multicluster/quick-start.md b/docs/multicluster/quick-start.md
new file mode 100644
index 00000000000..157fff4d09d
--- /dev/null
+++ b/docs/multicluster/quick-start.md
@@ -0,0 +1,200 @@
+# Antrea Multi-cluster Quick Start
+
+In this quick start guide, we will set up an Antrea Multi-cluster ClusterSet
+with two clusters. One cluster will serve as the leader of the ClusterSet, and
+meanwhile also join as a member cluster; another cluster will be a member only.
+
+The diagram below shows the two clusters and the ClusterSet to be created (for
+simplicity, the diagram just shows two Nodes for each cluster).
+
+
+
+## Preparation
+
+We assume an Antrea version >= `v1.7.0` is used in this guide, and the Antrea
+version is set to an environment variable `TAG`. For example, the following
+command sets the Antrea version to `v1.7.0`.
+
+```bash
+export TAG=v1.7.0
+```
+
+To use the latest version of Antrea Multi-cluster from the Antrea main branch,
+you can change the YAML manifest path to: `https://github.com/antrea-io/antrea/tree/main/multicluster/build/yamls/`
+when applying or downloading an Antrea YAML manifest.
+
+Antrea must be deployed in both cluster A and cluster B. To configure Antrea
+Multi-cluster Gateways, the `Multicluster` feature of `antrea-agent` must be
+feature enabled in both cluster A and B. Multi-cluster Gateways are required for
+routing multi-cluster Service traffic across the member clusters. Set the
+following configuration parameters in `antrea-agent.conf` of the Antrea
+deployment manifest to enable the `Multicluster` feature:
+
+```yaml
+antrea-agent.conf: |
+...
+ featureGates:
+...
+ Multicluster: true
+...
+ multicluster:
+ enable: true
+ namespace: ""
+```
+
+## Set up Leader and Member in Cluster A
+
+### Step 1 - deploy Antrea Multi-cluster Controllers for leader and member
+
+Run the following commands to deploy Multi-cluster Controller for the leader
+into Namespace `antrea-multicluster` (Namespace `antrea-multicluster` will be
+created by the commands), and Multi-cluster Controller for the member into
+Namepsace `kube-system`.
+
+```bash
+$kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-global.yml
+$kubectl create ns antrea-multicluster
+$curl -L https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-namespaced.yml > antrea-multicluster-leader-namespaced.yml
+$sed 's/changeme/antrea-multicluster/g' antrea-multicluster-leader-namespaced.yml | kubectl apply -f -
+$kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-member.yml
+```
+
+You can run the following command to verify the the leader and member
+`antrea-mc-controller` Pods are deployed and running:
+
+```bash
+$kubectl get all -A -l="component=antrea-mc-controller"
+NAMESPACE NAME READY STATUS RESTARTS AGE
+antrea-multicluster pod/antrea-mc-controller-cd7bf8f68-kh4kz 1/1 Running 0 50s
+kube-system pod/antrea-mc-controller-85dbf58b75-pjj48 1/1 Running 0 48s
+
+NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
+antrea-multicluster deployment.apps/antrea-mc-controller 1/1 1 1 50s
+kube-system deployment.apps/antrea-mc-controller 1/1 1 1 48s
+```
+
+### Step 2 - initialize ClusterSet
+
+Antrea provides several template YAML manifests to set up a ClusterSet quicker.
+You can run the following commands that use the template manifests to create a
+ClusterSet named `test-clusteraset` in the leader cluster and get a
+ServiceAccount token for the member clusters (both cluster A and B in our case)
+to access the leader cluster (cluster A in our case) apiserver.
+
+```bash
+$kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_clusterset_template.yaml
+$kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_leader_access_token_template.yaml
+$kubectl get secret leader-access-token -n antrea-multicluster -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e ' ca.crt' -e ' token:' -e '^type' -e ' namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' > leader-access-token.yml
+```
+
+The last command saves the ServiceAccount token to `leader-access-token.yml`
+which will be needed for member clusters to join the ClusterSet. Note, in this
+guide, we use a shared default ServiceAccount `antrea-mc-member-access-sa` for
+all member clusters. If you want to create a separate ServiceAccount for each
+member cluster for security considerations, you can follow the instructions in
+the [Multi-cluster User Guide](user-guide.md#set-up-access-to-leader-cluster).
+
+Next, run the following commands to make cluster A join the ClusterSet also as a
+member:
+
+```bash
+$kubectl apply -f leader-access-token.yml
+$curl -L https://raw.githubusercontent.com/antrea-io/antrea/v1.7.0/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml > multicluster_membercluster.yaml
+$sed -e 's/test-cluster-member/test-cluster-leader/g' -e 's//172.10.0.11/g' multicluster_membercluster.yaml | kubectl apply -f -
+```
+
+Here, `172.10.0.11` is the `kube-apiserver` IP of cluster A. You should replace
+it with the `kube-apiserver` IP of your leader cluster.
+
+### Step 3 - specify Multi-cluster Gateway Node
+
+Last, you need to choose a Node in cluster A to serve as the Multi-cluster
+Gateway. The Node should have an IP that is reachable from the cluster B's
+Gateway Node, so a tunnel can be created between the two Gateways. For more
+information about Multi-cluster Gatweay, please refer to the [Multi-cluster
+User Guide](user-guide.md#multi-cluster-gateway-configuration).
+
+Assuming K8s Node `node-a1` is selected for the Multi-cluster Gateway, run
+the following command to annotate the Node with
+`multicluster.antrea.io/gateway=true` (so Antrea can know it is the Gateway
+Node from the annotation):
+
+```bash
+$kubectl annotate node node-a1 multicluster.antrea.io/gateway=true
+```
+
+## Set up Cluster B
+
+Let us switch to cluster B. All the `kubectl` commands in the following steps
+should be run with the `kubeconfig` for cluster B.
+
+### Step 1 - deploy Antrea Multi-cluster Controller for member
+
+Run the following command to deploy the member Multi-cluster Controller into
+Namespace `kube-system`.
+
+```bash
+$kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-member.yml
+```
+
+You can run the following command to verify the `antrea-mc-controller` Pod is
+deployed and running:
+
+```bash
+$kubectl get all -A -l="component=antrea-mc-controller"
+NAMESPACE NAME READY STATUS RESTARTS AGE
+kube-system pod/antrea-mc-controller-85dbf58b75-pjj48 1/1 Running 0 40s
+
+NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
+kube-system deployment.apps/antrea-mc-controller 1/1 1 1 40s
+```
+
+### Step 2 - initialize ClusterSet
+
+Run the following commands to make cluster B join the ClusterSet:
+
+```bash
+$kubectl apply -f leader-access-token.yml
+$curl -L https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml > multicluster_membercluster.yaml
+$sed -e 's//172.10.0.11/g' multicluster_membercluster.yaml | kubectl apply -f -
+```
+
+`leader-access-token.yml` saves the leader cluster ServiceAccount token which
+was generated when initializing the ClusterSet in cluster A.
+
+### Step 3 - specify Multi-cluster Gateway Node
+
+Assuming K8s Node `node-b1` is chosen to be the Multi-cluster Gateway for cluster
+B, run the following command to annotate the Node:
+
+```bash
+$kubectl annotate node node-b1 multicluster.antrea.io/gateway=true
+```
+
+## What is Next
+
+So far, we set up an Antrea Multi-cluster ClusterSet with two clusters following
+the above sections of this guide. Next, you can start to consume the Antrea
+Multi-cluster features with the ClusterSet, including [Multi-cluster Services](user-guide.md#multi-cluster-service)
+and [ClusterNetworkPolicy Replication](user-guide.md#multi-cluster-clusternetworkpolicy-replication).
+Check the relevant Antrea Multi-cluster User Guide sections to learn more.
+
+If you want to add a new member cluster to your ClusterSet, you can follow the
+steps for cluster B to do so. But note, you will need the following two changes:
+
+1. You need to add the new mumber cluster to the ClusterSet in the leader
+cluster (cluster A). You can do that by adding the cluster ID of the new member
+to `multicluster_clusterset_template.yaml` and re-applying the manifest in
+cluster A.
+
+2. You need to update the member cluster ID in
+`multicluster_membercluster_template.yaml` to the cluster ID of the new member
+cluster in the step 2 of initializing ClusterSet. For example, you can run the
+following commands to the following to initialize the ClusterSet for a member
+cluster with ID `test-cluster-member2`:
+
+```bash
+$kubectl apply -f leader-access-token.yml
+$curl -L https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml > multicluster_membercluster.yaml
+$sed -e 's//172.10.0.11/g' -e 's/test-cluster-member/test-cluster-member2/g' multicluster_membercluster.yaml | kubectl apply -f -
+```
diff --git a/docs/multicluster/user-guide.md b/docs/multicluster/user-guide.md
index 13cae798e08..1fe51be3e13 100644
--- a/docs/multicluster/user-guide.md
+++ b/docs/multicluster/user-guide.md
@@ -2,136 +2,143 @@
Antrea Multi-cluster implements [Multi-cluster Service API](https://github.com/kubernetes/enhancements/tree/master/keps/sig-multicluster/1645-multi-cluster-services-api),
which allows users to create multi-cluster Services that can be accessed cross clusters in a
-ClusterSet. Antrea Multi-cluster also supports Antrea ClusterNetworkPolicy replication.
-Multi-cluster admins can define ClusterNetworkPolicies to be replicated across the entire
-ClusterSet, and enforced in all member clusters. Antrea Multi-cluster is introduced in
+ClusterSet. Antrea Multi-cluster also supports Antrea ClusterNetworkPolicy replication. A
+multi-cluster ClusterSet admin can define ClusterNetworkPolicies to be replicated across the
+entire ClusterSet and enforced in all member clusters. Antrea Multi-cluster is introduced in
Antrea v1.5.0, and the ClusterNetworkPolicy replication feature is supported since Antrea
-v1.6.0.
+v1.6.0. In Antrea v1.7.0, the Multi-cluster Gateway feature is added that supports routing
+Multi-cluster Service traffic through tunnels among clusters.
+
+## Quick Start
+
+Please refer to the [Quick Start Guide](./quick-start.md) to learn how to build a ClusterSet
+with two clusters quickly.
## Antrea Multi-cluster Installation
-### Prepare Antrea Multi-cluster Image
+### Preparation
-For Antrea Multi-cluster, there is only one image `antrea/antrea-mc-controller:latest`
-you can pull the image from Docker Hub by default, or if you'd like to build image locally,
-you can follow the following steps to get the image ready on your local clusters.
+We assume an Antrea version >= `v1.7.0` is used in this guide, and the Antrea
+version is set to an environment variable `TAG`. For example, the following
+command sets the Antrea version to `v1.7.0`.
-1. Go to `antrea` folder, run `make antrea-mc-controller`, and you will get a new image
- named `antrea/antrea-mc-controller:latest` locally.
-2. Run `docker save antrea/antrea-mc-controller:latest > antrea-mcs.tar` to save the image.
-3. Copy the image file `antrea-mcs.tar` to the Nodes of your local cluster.
-4. Run `docker load < antrea-mcs.tar` in each Node of your local cluster.
+```bash
+export TAG=v1.7.0
+```
-### Deploy Mulit-cluster Controller
+To use the latest version of Antrea Multi-cluster from the Antrea main branch,
+you can change the YAML manifest path to: `https://github.com/antrea-io/antrea/tree/main/multicluster/build/yamls/`
+when applying or downloading an Antrea YAML manifest.
-In a ClusterSet, there is one leader cluster and two or more member clusters. You can run the
-leader controller in either a dedicated cluster or one of the member clusters. Please refer
-to [Installation in Dedicated Leader Cluster](#installation-in-dedicated-leader-cluster)
-and [Installation in the Member Cluster](#installation-in-the-member-cluster)
-to deploy leader and member controllers in separate clusters, or you can refer to
-[Installation in a Shared Cluster](#installation-in-a-shared-cluster) to learn how to
-run both member and leader controllers in one cluster.
+Multi-cluster Services require an Antrea Multi-cluster Gateway to be set up in
+each member cluster, so the Multi-cluster Service traffic can be routed across
+clusters by the Gateways. To support Multi-cluster Gateways, `antrea-agent` must
+be deployed with the `Multicluster` feature enabled in a member cluster. You can
+set the following configuration parameters in `antrea-agent.conf` of the Antrea
+deployment manifest to enable the `Multicluster` feature:
-#### Deployment in a Dedicated Leader Cluster
+```yaml
+antrea-agent.conf: |
+...
+ featureGates:
+...
+ Multicluster: true
+...
+ multicluster:
+ enable: true
+ namespace: ""
+```
-1. Run the following commands to apply Multi-cluster CRDs in the leader cluster.
+### Deploy Antrea Mulit-cluster Controller
- * For any given release `` (e.g. `v1.5.0`).
+A Multi-cluster ClusterSet is comprised of a single leader cluster and at least
+two member clusters. Antrea Multi-cluster Controller needs to be deployed in the
+leader and all member clusters. A cluster can serve as the leader, and meanwhile
+also be a member cluster of the ClusterSet. To deploy Multi-cluster Controller
+in a dedicated leader cluster, please refer to [Deploy in a Dedicated Leader
+cluster](#deploy-in-a-dedicated-leader-cluster). To deploy Multi-cluster
+Controller in a member cluster, please refer to [Deploy in a Member Cluster](#deploy-in-a-member-cluster).
+To deploy Multi-cluster Controller in a dual-role cluster, please refer to
+[Deploy Leader and Member in One Cluster](#deploy-leader-and-member-in-one-cluster).
- ```bash
- kubectl apply -f https://github.com/antrea-io/antrea/releases/download//antrea-multicluster-leader-global.yml
- ```
+#### Deploy in a Dedicated Leader Cluster
- * To deploy the latest version (checkout and built from the `main` branch).
+1. Run the following command to import Multi-cluster CRDs in the leader cluster:
```bash
- kubectl apply -f multicluster/build/yamls/antrea-multicluster-leader-global.yml
+ kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-global.yml
```
-2. Install Multi-cluster Controller in the leader cluster. Since Multi-cluster Controller is
-running as namespaced deployment, you should create a Namespace first, then apply the
-manifest with new Namespace.
+2. Install Multi-cluster Controller in the leader cluster. Since Multi-cluster
+ Controller runs as a namespaced Deployment, you should create the Namespace
+ first, and then apply the deployment manifest with the Namespace.
-* For any given release `` (e.g. `v1.5.0`).
-
```bash
- kubectl create ns antrea-mcs-ns
- curl -L https://github.com/antrea-io/antrea/releases/download/v1.5.0/antrea-multicluster-leader-namespaced.yml > antrea-multicluster-leader-namespaced.yml
- sed 's/changeme/antrea-mcs-ns/g' antrea-multicluster-leader-namespaced.yml | kubectl apply -f -
+ kubectl create ns antrea-multicluster
+ curl -L https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-namespaced.yml > antrea-multicluster-leader-namespaced.yml
+ sed 's/changeme/antrea-multicluster/g' antrea-multicluster-leader-namespaced.yml | kubectl apply -f -
```
-
-* To deploy the latest version (checkout and built from the `main` branch).
-
- ```bash
- kubectl create ns antrea-mcs-ns
- multicluster/hack/generate-manifest.sh -l antrea-mcs-ns | kubectl apply -f -
- ```
-
-#### Deployment in a Member Cluster
-
-You can run the following commands to install Multi-cluster Controller to all
-member clusters. The command will run the controller in the "member" mode in the `kube-system`
-Namespace by default.
-* For any given release `` (e.g. `v1.5.0`).
+#### Deploy in a Member Cluster
-```bash
-kubectl apply -f https://github.com/antrea-io/antrea/releases/download//antrea-multicluster-member.yml
-```
-
-* To deploy the latest version (checkout and built from the `main` branch).
+You can run the following command to install Multi-cluster Controller in a
+member cluster. The command will run the controller in the "member" mode in the
+`kube-system` Namespace. If you want to use a different Namespace other than
+`kube-system`, you can edit `antrea-multicluster-member.yml` and change
+`kube-system` to the desired Namespace.
```bash
-kubectl apply -f multicluster/build/yamls/antrea-multicluster-member.yml
+kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-member.yml
```
-#### Deploy Leader and Member in one Cluster
+#### Deploy Leader and Member in One Cluster
-There is no deployment dependency between member and leader clusters if you are using a cluster
-as a dedicated leader cluster. But if you'd like to run both leader and member controllers in
-one cluster, below is the required deployment sequence:
+We need to run two instances of Multi-cluster Controller in the dual-role
+cluster, one in leader mode and another in member mode.
-1. Follow the step in section [Installation in the Member Cluster](#deployment-in-the-member-cluster)
-to install the member controller first.
-2. Follow the step 2 only in section [Installation in Dedicated Leader Cluster](#deployment-in-dedicated-leader-cluster)
-to install the leader controller. The global CRDs have been installed when you deploy the member controller.
+1. Follow the steps in section [Deploy in a Dedicated Leader Cluster](#deploy-in-a-dedicated-leader-cluster)
+ to deploy the leader controller and import the Multi-cluster CRDs.
+2. Follow the steps in section [Deploy in a Member Cluster](#deploy-in-a-member-cluster)
+ to deploy the member controller.
-### ClusterSet Creation
+### Create ClusterSet
An Antrea Multi-cluster ClusterSet should include at least one leader cluster
-and two member clusters. As an example, in the following sections we will create a ClusterSet with
-ID `test-clusterset` which has two member clusters with cluster ID `test-cluster-east`,
-`test-cluster-west` and one leader cluster with ID `test-cluster-north`.
+and two member clusters. As an example, in the following sections we will create
+a ClusterSet `test-clusterset` which has two member clusters with cluster ID
+`test-cluster-east` and `test-cluster-west` respectively, and one leader cluster
+with ID `test-cluster-north`.
#### Set up Access to Leader Cluster
-We first need to set up access to the leader cluster's API server for all member clusters.
-We recommend creating one ServiceAccount for each member for fine-grained access control.
+We first need to set up access to the leader cluster's API server for all member
+clusters. We recommend creating one ServiceAccount for each member for
+fine-grained access control.
-1. Apply the following yaml in the leader cluster to set up access for
-`test-cluster-east`.
+1. Apply the following YAML manifest in the leader cluster to set up access for
+ `test-cluster-east`:
```yml
apiVersion: v1
kind: ServiceAccount
metadata:
name: member-east-access-sa
- namespace: antrea-mcs-ns
+ namespace: antrea-multicluster
---
apiVersion: v1
kind: Secret
metadata:
- name: member-east-access-token
- namespace: antrea-mcs-ns
- annotations:
- kubernetes.io/service-account.name: member-east-access-sa
+ name: member-east-access-token
+ namespace: antrea-multicluster
+ annotations:
+ kubernetes.io/service-account.name: member-east-access-sa
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: member-east-access-rolebinding
- namespace: antrea-mcs-ns
+ namespace: antrea-multicluster
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
@@ -139,34 +146,37 @@ We recommend creating one ServiceAccount for each member for fine-grained access
subjects:
- kind: ServiceAccount
name: member-east-access-sa
- namespace: antrea-mcs-ns
+ namespace: antrea-multicluster
```
-2. Copy the access token into the member cluster `test-cluster-east`. E.g.
+2. Generate the access token file from the leader cluster, and create a Secret
+ with the token in member cluster `test-cluster-east`, e.g.:
```bash
- kubectl get secret member-east-access-token -n antrea-mcs-ns -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e ' ca.crt' -e ' token:' -e '^type' -e ' namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-mcs-ns/kube-system/g' > member-east-access-token.yml
+ # Generate the file 'member-east-access-token.yml' from your leader cluster
+ kubectl get secret member-east-access-token -n antrea-multicluster -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e ' ca.crt' -e ' token:' -e '^type' -e ' namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' > member-east-access-token.yml
+ # Apply 'member-east-access-token.yml' to the member cluster.
+ kubectl apply -f member-east-access-token.yml --kubeconfig=/path/to/kubeconfig-of-member-test-cluster-east
```
-3. Replace all `east` to `west` and repeat step 1/2 for the other member cluster `test-cluster-west`
-to create the token secret and copy the token.
-
-#### Set up ClusterSet
+3. Replace all `east` to `west` and repeat step 1/2 for the other member cluster
+ `test-cluster-west`.
-All clusters in the ClusterSet need to use `ClusterClaim` to claim itself as a member
-of a ClusterSet. A leader cluster will define `ClusterSet` which includes leader and
-member clusters.
+#### Initialize ClusterSet
-* Create below `ClusterClaim` and `ClusterSet` in the member cluster `test-cluster-east`.
+In all clusters, a `ClusterSet` CR must be created to define the ClusterSet, and
+two `ClusterClaim` CRs must be created to claim the ClusterSet and claim the
+cluster is a member of the ClusterSet.
-Note: Update `server: "https://172.18.0.2:6443"` in `ClusterSet` resource to
-the correct the leader cluster API address. You can refer to [multicluster_membercluster_template.yaml](../../multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml).
+- Create `ClusterClaim` and `ClusterSet` in member cluster `test-cluster-east`
+with the following YAML manifest (you can also refer to
+[multicluster_membercluster_template.yaml](../../multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml)):
```yaml
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterClaim
metadata:
- name: east-membercluster-id
+ name: id.k8s.io
namespace: kube-system
name: id.k8s.io
value: test-cluster-east
@@ -174,36 +184,36 @@ value: test-cluster-east
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterClaim
metadata:
- name: clusterset-id
+ name: clusterset.k8s.io
namespace: kube-system
-name: clusterSet.k8s.io
+name: clusterset.k8s.io
value: test-clusterset
---
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterSet
metadata:
- name: test-clusterset
- namespace: kube-system
+ name: test-clusterset
+ namespace: kube-system
spec:
- leaders:
- - clusterID: test-cluster-north
- secret: "member-east-access-token"
- server: "https://172.18.0.2:6443"
- members:
- - clusterID: test-cluster-east
- namespace: antrea-mcs-ns
+ leaders:
+ - clusterID: test-cluster-north
+ secret: "member-east-access-token"
+ server: "https://172.18.0.1:6443"
+ members:
+ - clusterID: test-cluster-east
+ namespace: antrea-multicluster
```
-* Create `ClusterClaim` and `ClusterSet` in the member cluster `test-cluster-west`.
+Note: update `server: "https://172.18.0.1:6443"` in the `ClusterSet` spec to the
+correct leader cluster API server address.
-Note: Update `server: "https://172.18.0.2:6443"` in `ClusterSet` resource to
-the correct leader cluster API address.
+- Create `ClusterClaim` and `ClusterSet` in member cluster `test-cluster-west`:
```yaml
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterClaim
metadata:
- name: west-membercluster-id
+ name: id.k8s.io
namespace: kube-system
name: id.k8s.io
value: test-cluster-west
@@ -211,108 +221,83 @@ value: test-cluster-west
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterClaim
metadata:
- name: clusterset-id
+ name: clusterset.k8s.io
namespace: kube-system
-name: clusterSet.k8s.io
+name: clusterset.k8s.io
value: test-clusterset
---
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterSet
metadata:
- name: test-clusterset
- namespace: kube-system
+ name: test-clusterset
+ namespace: kube-system
spec:
- leaders:
- - clusterID: test-cluster-north
- secret: "member-west-access-token"
- server: "https://172.18.0.2:6443"
- members:
- - clusterID: test-cluster-west
- namespace: antrea-mcs-ns
+ leaders:
+ - clusterID: test-cluster-north
+ secret: "member-west-access-token"
+ server: "https://172.18.0.1:6443"
+ members:
+ - clusterID: test-cluster-west
+ namespace: antrea-multicluster
```
-* Create `ClusterClaim` and `ClusterSet` in the leader cluster `test-cluster-north`. A sample is like below, you can also refer to [multicluster_clusterset_template.yaml](../../multicluster/config/samples/clusterset_init/multicluster_clusterset_template.yaml).
+- Create `ClusterClaim` and `ClusterSet` in the leader cluster
+`test-cluster-north` with the following YAML manifest (you can also refer to
+[multicluster_clusterset_template.yaml](../../multicluster/config/samples/clusterset_init/multicluster_clusterset_template.yaml)):
```yaml
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterClaim
metadata:
- name: leadercluster-id
- namespace: antrea-mcs-ns
+ name: id.k8s.io
+ namespace: antrea-multicluster
name: id.k8s.io
value: test-cluster-north
---
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterClaim
metadata:
- name: clusterset-id
- namespace: antrea-mcs-ns
-name: clusterSet.k8s.io
+ name: clusterset.k8s.io
+ namespace: antrea-multicluster
+name: clusterset.k8s.io
value: test-clusterset
---
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterSet
metadata:
- name: test-clusterset
- namespace: antrea-mcs-ns
+ name: test-clusterset
+ namespace: antrea-multicluster
spec:
- leaders:
- - clusterID: test-cluster-north
- members:
- - clusterID: test-cluster-east
- serviceAccount: "member-east-access-sa"
- - clusterID: test-cluster-west
- serviceAccount: "member-west-access-sa"
- namespace: antrea-mcs-ns
+ leaders:
+ - clusterID: test-cluster-north
+ members:
+ - clusterID: test-cluster-east
+ serviceAccount: "member-east-access-sa"
+ - clusterID: test-cluster-west
+ serviceAccount: "member-west-access-sa"
+ namespace: antrea-multicluster
```
-When you also want to make the leader cluster `test-cluster-north` as a member in the
-ClusterSet, so it can export and import resources from other member clusters, make
-sure you follow the steps in [Installation in a Shared Cluster](#installation-in-a-shared-cluster),
-and repeat the step [Setting up Access to Leader Cluster](#setting-up-access-to-leader-cluster)
-as well to create the token secret and copy the token first.
+In the leader cluster, the `ClusterSet` spec should include all member clusters
+of the ClusterSet.
-A sample yaml is like below:
+#### Initialize ClusterSet for a Dual-role Cluster
-```yml
-apiVersion: v1
-kind: ServiceAccount
-metadata:
- name: member-north-access-sa
- namespace: antrea-mcs-ns
----
-apiVersion: v1
-kind: Secret
-metadata:
- name: member-north-access-token
- namespace: antrea-mcs-ns
- annotations:
- kubernetes.io/service-account.name: member-north-access-sa
-type: kubernetes.io/service-account-token
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
- name: member-north-access-rolebinding
- namespace: antrea-mcs-ns
-roleRef:
- apiGroup: rbac.authorization.k8s.io
- kind: Role
- name: antrea-mc-member-cluster-role
-subjects:
- - kind: ServiceAccount
- name: member-north-access-sa
- namespace: antrea-mcs-ns
-```
+If you want to make the leader cluster `test-cluster-north` also a member
+cluster of the ClusterSet, make sure you follow the steps in [Deploy Leader and
+Member in One Cluster](#deploy-leader-and-member-in-one-cluster) and repeat the
+steps in [Set up Access to Leader Cluster](#set-up-access-to-leader-cluster) as
+well (don't forget replace all `east` to `north` when you repeat the steps).
-Then create below `ClusterClaim` and `ClusterSet` in the cluster `test-cluster-north` in the
-`kube-system` Namespace where the member controller is deployed.
+Then create the `ClusterClaim` and `ClusterSet` CRs in cluster
+`test-cluster-north` in the `kube-system` Namespace (where the member
+Multi-cluster Controller runs):
```yaml
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterClaim
metadata:
- name: north-membercluster-id
+ name: id.k8s.io
namespace: kube-system
name: id.k8s.io
value: test-cluster-north
@@ -320,49 +305,128 @@ value: test-cluster-north
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterClaim
metadata:
- name: clusterset-id
+ name: clusterset.k8s.io
namespace: kube-system
-name: clusterSet.k8s.io
+name: clusterset.k8s.io
value: test-clusterset
---
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterSet
metadata:
- name: test-clusterset
- namespace: kube-system
+ name: test-clusterset
+ namespace: kube-system
spec:
- leaders:
- - clusterID: test-cluster-north
- secret: "member-north-access-token"
- server: "https://172.18.0.2:6443"
- members:
- - clusterID: test-cluster-north
- namespace: antrea-mcs-ns
+ leaders:
+ - clusterID: test-cluster-north
+ secret: "member-north-access-token"
+ server: "https://172.18.0.1:6443"
+ members:
+ - clusterID: test-cluster-north
+ namespace: antrea-multicluster
```
-Then update the ClusterSet `test-clusterset` definition as below to include itself
-as a member cluster in the Namespace `antrea-mcs-ns` where the leader controller is
-running in the cluster `test-cluster-north`.
+Last, update the ClusterSet `test-clusterset` in Namepsace `antrea-multicluster`
+(where the leader Multi-cluster Controller runs) to include `test-cluster-north`
+as a member cluster of the ClusterSet:
```yaml
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterSet
metadata:
- name: test-clusterset
- namespace: antrea-mcs-ns
+ name: test-clusterset
+ namespace: antrea-multicluster
spec:
- leaders:
- - clusterID: test-cluster-north
- members:
- - clusterID: test-cluster-east
- serviceAccount: "member-east-access-sa"
- - clusterID: test-cluster-west
- serviceAccount: "member-west-access-sa"
- - clusterID: test-cluster-north
- serviceAccount: "member-north-access-sa"
- namespace: antrea-mcs-ns
+ leaders:
+ - clusterID: test-cluster-north
+ members:
+ - clusterID: test-cluster-east
+ serviceAccount: "member-east-access-sa"
+ - clusterID: test-cluster-west
+ serviceAccount: "member-west-access-sa"
+ - clusterID: test-cluster-north
+ serviceAccount: "member-north-access-sa"
+ namespace: antrea-multicluster
```
+## Multi-cluster Gateway Configuration
+
+After a member cluster joins a ClusterSet, and the `Multicluster` feature is
+enabled for `antrea-agent`, one K8s Node in the member cluster can be specified
+to serve as the Multi-cluster Gateway of the cluster. An annotation -
+`multicluster.antrea.io/gateway=true` - should be added to the Node to tell
+Antrea it is the Gateway Node. For example, you can run the following command to
+annotate Node `node-1` as the Multi-cluster Gateway:
+
+```bash
+$kubectl annotate node node-1 multicluster.antrea.io/gateway=true
+```
+
+Multi-cluster Controller in the member cluster will detect the Gateway Node, and
+create a `Gateway` CR with the same name as the Node. You can check the `Gateway`
+with: `kubectl get gateway node-1 -o yaml`, which should show the Gateway
+information like:
+
+```yaml
+apiVersion: multicluster.crd.antrea.io/v1alpha1
+kind: Gateway
+metadata:
+ name: node-1
+ namespace: kube-system
+gatewayIP: 10.17.27.55
+internalIP: 10.17.27.55
+```
+
+`internalIP` of the Gateway is used for the tunnels between the Gateway Node and
+other Nodes in the local cluster, while `gatewayIP` is used for the tunnels to
+remote Gateways of other member clusters. By default, Multi-cluster Controller
+will use the K8s Node `InternalIP` of the Gateway Node as the `gatewayIP`. If
+you want to use `ExternalIP` of the Gateway Node instead, you can change the
+configuration option `gatewayIPPrecedence` in ConfigMap
+`antrea-mc-controller-config-***` to value `public`, when you deploy the member
+Multi-cluster Controller. When selecting the Multi-cluster Gateway Node, you
+need to make sure the resulted `gatewayIP` can be reached from the remote
+Gateways.
+
+After the `Gateway` CR is created, Multi-cluster Controller will be responsible
+for exporting the cluster's network information to the leader cluster including
+the Gateway IPs and `serviceCIDR` (the cluster's Service ClusterIP range).
+Multi-cluster Controller will try to discover `serviceCIDR` automatically, but
+you can also define the `serviceCIDR` manually in the Antrea Multi-cluster
+ConfigMap `antrea-mc-controller-config-***`.
+
+The Multi-cluster resource export/import pipeline will replicate the exported
+cluster network information to all member clusters in the ClusterSet. For
+example, in other member clusters, you can see a `ClusterInfoImport` CR with
+name `test-cluster-east-clusterinfo` is created for `test-cluster-east` with
+its network information. You can check the `ClusterInfoImport` with command:
+`kubectl get clusterinfoimport test-cluster-east-clusterinfo -o yaml`, which
+should show information like:
+
+```yaml
+apiVersion: multicluster.crd.antrea.io/v1alpha1
+kind: ClusterInfoImport
+metadata:
+ name: test-cluster-east-clusterinfo
+ namespace: kube-system
+spec:
+ clusterID: test-cluster-east
+ gatewayInfos:
+ - gatewayIP: 10.17.27.55
+ serviceCIDR: 110.96.0.0/20
+```
+
+Make sure you repeat the same steps to assign a Gateway Node in all member
+clusters, then `antrea-agent` will set up Geneve tunnels among the Gateway Nodes
+of member clusters based on the local `Gateway` and the `ClusterInfoImport`
+resources, and route Multi-cluster Service traffic across clusters through the
+tunnels. `antrea-agent` on regular Nodes will route cross-cluster traffic from
+local Pods to the Gateway Node of the member cluster.
+
+Once you confirm that all `Gateway` and `ClusterInfoImport` resources are
+created correctly, you can follow the [Multi-cluster Service](#multi-cluster-service)
+section to create Multi-cluster Services and verify cross-cluster Service
+access.
+
## Multi-cluster Service
After you set up a ClusterSet properly, you can simply create a `ServiceExport` resource
@@ -407,16 +471,16 @@ If there is any new change from the exported Service or Endpoints, the derived m
resources will be updated accordingly. A few cases below are worth to note:
1. When there is only one Service ResourceExport, Antrea Multi-cluster Controller will converge
-the change and reflect the update in correspoding ResourceImport. Otherwise, controller will skip
-converging the update until users correct it to match the Service definition in correspoding
-ResourceImport.
+ the change and reflect the update in correspoding ResourceImport. Otherwise, controller will skip
+ converging the update until users correct it to match the Service definition in correspoding
+ ResourceImport.
2. When a member cluster has already exported a Service, e.g.: `default/nginx` with TCP
-Port `80`, then other member clusters can only export the same Service with the same Ports
-definition including port names. Otherwise, Antrea Multi-cluster Controller will skip converting
-the mismatched ResourceExport into the corresponding ResourceImport until users correct it.
+ Port `80`, then other member clusters can only export the same Service with the same Ports
+ definition including port names. Otherwise, Antrea Multi-cluster Controller will skip converting
+ the mismatched ResourceExport into the corresponding ResourceImport until users correct it.
3. When a member cluster's Service ResourceExport has not been converged successfully
-due to forementioned mismatch issue, Antrea Multi-cluster Controller will also skip converging
-the corresponding Endpoints ResourceExport until users correct it.
+ due to forementioned mismatch issue, Antrea Multi-cluster Controller will also skip converging
+ the corresponding Endpoints ResourceExport until users correct it.
## Multi-cluster ClusterNetworkPolicy Replication
@@ -429,7 +493,7 @@ Antrea ClusterNetworkPolicy (ACNP), refer to [this document](../antrea-network-p
To achieve such ACNP replication across clusters, admins can, in the acting leader cluster of a
Multi-cluster deployment, create a ResourceExport of kind `AntreaClusterNetworkPolicy` which contains
the ClusterNetworkPolicy spec they wish to be replicated. The ResourceExport should be created in the
-Namespace which implements the Common Area of the ClusterSet. In future releases, some additional tooling
+Namespace which implements the Common Area of the ClusterSet. In future releases, some additional tooling
may become available to automate the creation of such ResourceExport and make ACNP replication easier.
```yaml
@@ -437,26 +501,26 @@ apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ResourceExport
metadata:
name: strict-namespace-isolation-for-test-clusterset
- namespace: antrea-mcs-ns # Namespace that implements Common Area of test-clusterset
+ namespace: antrea-multicluster # Namespace that implements Common Area of test-clusterset
spec:
- kind: AntreaClusterNetworkPolicy
- name: strict-namespace-isolation # In each importing cluster, an ACNP of name antrea-mc-strict-namespace-isolation will be created with the spec below
+ kind: AntreaClusterNetworkPolicy
+ name: strict-namespace-isolation # In each importing cluster, an ACNP of name antrea-mc-strict-namespace-isolation will be created with the spec below
clusternetworkpolicy:
priority: 1
tier: securityops
appliedTo:
- - namespaceSelector: {} # Selects all Namespaces in the member cluster
+ - namespaceSelector: {} # Selects all Namespaces in the member cluster
ingress:
- action: Pass
from:
- - namespaces:
- match: Self # Skip drop rule for traffic from Pods in the same Namespace
- - podSelector:
- matchLabels:
- k8s-app: kube-dns # Skip drop rule for traffic from the core-dns components
+ - namespaces:
+ match: Self # Skip drop rule for traffic from Pods in the same Namespace
+ - podSelector:
+ matchLabels:
+ k8s-app: kube-dns # Skip drop rule for traffic from the core-dns components
- action: Drop
from:
- - namespaceSelector: {} # Drop from Pods from all other Namespaces
+ - namespaceSelector: {} # Drop from Pods from all other Namespaces
```
The above sample spec will create an ACNP in each member cluster which implements strict namespace
@@ -472,7 +536,7 @@ ResourceImport of the original ResourceExport:
kubectl describe resourceimport -A
---
Name: strict-namespace-isolation-antreaclusternetworkpolicy
-Namespace: antrea-mcs-ns
+Namespace: antrea-multicluster
API Version: multicluster.crd.antrea.io/v1alpha1
Kind: ResourceImport
Spec:
@@ -503,6 +567,17 @@ Events:
Warning ACNPImportFailed 2m11s resourceimport-controller ACNP Tier random does not exist in the importing cluster test-cluster-west
```
+## Build Antrea Multi-cluster Image
+
+If you'd like to build Antrea Multi-cluster Docker image locally, you can follow
+the following steps:
+
+1. Go to your local `antrea` source tree, run `make antrea-mc-controller`, and you will get a new image
+ named `antrea/antrea-mc-controller:latest` locally.
+2. Run `docker save antrea/antrea-mc-controller:latest > antrea-mcs.tar` to save the image.
+3. Copy the image file `antrea-mcs.tar` to the Nodes of your local cluster.
+4. Run `docker load < antrea-mcs.tar` in each Node of your local cluster.
+
## Known Issue
We recommend user to reinstall or update Antrea Multi-cluster controllers through `kubectl apply`.
diff --git a/hack/.notableofcontents b/hack/.notableofcontents
index 720e2f957e8..268f083128d 100644
--- a/hack/.notableofcontents
+++ b/hack/.notableofcontents
@@ -29,6 +29,7 @@ docs/maintainers/updating-ovs-windows.md
docs/multicluster/api.md
docs/multicluster/antctl.md
docs/multicluster/architecture.md
+docs/multicluster/quick-start.md
docs/multicluster/upgrade.md
docs/multicluster/user-guide.md
docs/network-requirements.md
diff --git a/multicluster/apis/multicluster/v1alpha1/clusterclaim_types.go b/multicluster/apis/multicluster/v1alpha1/clusterclaim_types.go
index c48b3c67b46..5699b8e4cb1 100644
--- a/multicluster/apis/multicluster/v1alpha1/clusterclaim_types.go
+++ b/multicluster/apis/multicluster/v1alpha1/clusterclaim_types.go
@@ -24,7 +24,7 @@ const (
// Identify this cluster.
WellKnownClusterClaimID = "id.k8s.io"
// Identify a clusterSet that this cluster is member of.
- WellKnownClusterClaimClusterSet = "clusterSet.k8s.io"
+ WellKnownClusterClaimClusterSet = "clusterset.k8s.io"
)
// +genclient
diff --git a/multicluster/config/samples/clusterset_init/multicluster_clusterset_template.yaml b/multicluster/config/samples/clusterset_init/multicluster_clusterset_template.yaml
index 915ecad32e8..faae8f4c770 100644
--- a/multicluster/config/samples/clusterset_init/multicluster_clusterset_template.yaml
+++ b/multicluster/config/samples/clusterset_init/multicluster_clusterset_template.yaml
@@ -1,30 +1,30 @@
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterClaim
metadata:
- name: leadercluster-id
- namespace: changeme
+ name: id.k8s.io
+ namespace: antrea-multicluster
name: id.k8s.io
value: test-cluster-leader
---
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterClaim
metadata:
- name: clusterset-id
- namespace: changeme
-name: clusterSet.k8s.io
+ name: clusterset.k8s.io
+ namespace: antrea-multicluster
+name: clusterset.k8s.io
value: test-clusterset
---
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterSet
metadata:
name: test-clusterset
- namespace: changeme
+ namespace: antrea-multicluster
spec:
leaders:
- clusterID: test-cluster-leader
members:
- - clusterID: test-cluster-east
+ - clusterID: test-cluster-member
serviceAccount: antrea-mc-member-access-sa
- - clusterID: test-cluster-west
+ - clusterID: test-cluster-leader
serviceAccount: antrea-mc-member-access-sa
- namespace: antrea-mcs-ns
+ namespace: antrea-multicluster
diff --git a/multicluster/config/samples/clusterset_init/multicluster_leader_access_token_template.yaml b/multicluster/config/samples/clusterset_init/multicluster_leader_access_token_template.yaml
index a75d5d33e3d..030302c84df 100644
--- a/multicluster/config/samples/clusterset_init/multicluster_leader_access_token_template.yaml
+++ b/multicluster/config/samples/clusterset_init/multicluster_leader_access_token_template.yaml
@@ -2,7 +2,7 @@ apiVersion: v1
kind: Secret
metadata:
name: leader-access-token
- namespace: changeme
+ namespace: antrea-multicluster
annotations:
kubernetes.io/service-account.name: antrea-mc-member-access-sa
type: kubernetes.io/service-account-token
diff --git a/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml b/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml
index 9510db551a7..2c9543dbafa 100644
--- a/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml
+++ b/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml
@@ -1,17 +1,17 @@
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterClaim
metadata:
- name: east-membercluster-id
+ name: id.k8s.io
namespace: kube-system
name: id.k8s.io
-value: test-cluster-east
+value: test-cluster-member
---
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterClaim
metadata:
- name: clusterset-id
+ name: clusterset.k8s.io
namespace: kube-system
-name: clusterSet.k8s.io
+name: clusterset.k8s.io
value: test-clusterset
---
apiVersion: multicluster.crd.antrea.io/v1alpha1
@@ -25,5 +25,6 @@ spec:
secret: leader-access-token
server: https://:6443
members:
- - clusterID: test-cluster-east
- namespace: antrea-mcs-ns
+ - clusterID: test-cluster-leader
+ - clusterID: test-cluster-member
+ namespace: antrea-multicluster
diff --git a/multicluster/config/samples/multicluster_v1alpha1_clusterclaim.yaml b/multicluster/config/samples/multicluster_v1alpha1_clusterclaim.yaml
index 81a0f1f4dac..238a0fd3395 100644
--- a/multicluster/config/samples/multicluster_v1alpha1_clusterclaim.yaml
+++ b/multicluster/config/samples/multicluster_v1alpha1_clusterclaim.yaml
@@ -11,5 +11,5 @@ kind: ClusterClaim
metadata:
name: clusterset-id
namespace: default
-name: clusterSet.k8s.io
+name: clusterset.k8s.io
value: test-clusterset
diff --git a/multicluster/test/integration/suite_test.go b/multicluster/test/integration/suite_test.go
index 781bbe0d767..6d100522eb6 100644
--- a/multicluster/test/integration/suite_test.go
+++ b/multicluster/test/integration/suite_test.go
@@ -201,7 +201,7 @@ func configureClusterSet() {
Namespace: LeaderNamespace,
Name: "clusterset-id",
},
- Name: "clusterSet.k8s.io",
+ Name: "clusterset.k8s.io",
Value: clusterSetID,
}
clusterSet := &mcsv1alpha1.ClusterSet{
diff --git a/multicluster/test/yamls/clusterset.yml b/multicluster/test/yamls/clusterset.yml
index b4f71257dab..49fc9fe30f7 100644
--- a/multicluster/test/yamls/clusterset.yml
+++ b/multicluster/test/yamls/clusterset.yml
@@ -11,7 +11,7 @@ kind: ClusterClaim
metadata:
name: clusterset-id
namespace: antrea-mcs-ns
-name: clusterSet.k8s.io
+name: clusterset.k8s.io
value: test-clusterset
---
apiVersion: multicluster.crd.antrea.io/v1alpha1
diff --git a/multicluster/test/yamls/east-member-cluster.yml b/multicluster/test/yamls/east-member-cluster.yml
index 36b36c7f183..92e393ec6e2 100644
--- a/multicluster/test/yamls/east-member-cluster.yml
+++ b/multicluster/test/yamls/east-member-cluster.yml
@@ -11,7 +11,7 @@ kind: ClusterClaim
metadata:
name: clusterset-id
namespace: kube-system
-name: clusterSet.k8s.io
+name: clusterset.k8s.io
value: test-clusterset
---
apiVersion: multicluster.crd.antrea.io/v1alpha1
diff --git a/multicluster/test/yamls/west-member-cluster.yml b/multicluster/test/yamls/west-member-cluster.yml
index 8b523116a17..be0da64ec44 100644
--- a/multicluster/test/yamls/west-member-cluster.yml
+++ b/multicluster/test/yamls/west-member-cluster.yml
@@ -11,7 +11,7 @@ kind: ClusterClaim
metadata:
name: clusterset-id
namespace: kube-system
-name: clusterSet.k8s.io
+name: clusterset.k8s.io
value: test-clusterset
---
apiVersion: multicluster.crd.antrea.io/v1alpha1
+
+## Preparation
+
+We assume an Antrea version >= `v1.7.0` is used in this guide, and the Antrea
+version is set to an environment variable `TAG`. For example, the following
+command sets the Antrea version to `v1.7.0`.
+
+```bash
+export TAG=v1.7.0
+```
+
+To use the latest version of Antrea Multi-cluster from the Antrea main branch,
+you can change the YAML manifest path to: `https://github.com/antrea-io/antrea/tree/main/multicluster/build/yamls/`
+when applying or downloading an Antrea YAML manifest.
+
+Antrea must be deployed in both cluster A and cluster B. To configure Antrea
+Multi-cluster Gateways, the `Multicluster` feature of `antrea-agent` must be
+feature enabled in both cluster A and B. Multi-cluster Gateways are required for
+routing multi-cluster Service traffic across the member clusters. Set the
+following configuration parameters in `antrea-agent.conf` of the Antrea
+deployment manifest to enable the `Multicluster` feature:
+
+```yaml
+antrea-agent.conf: |
+...
+ featureGates:
+...
+ Multicluster: true
+...
+ multicluster:
+ enable: true
+ namespace: ""
+```
+
+## Set up Leader and Member in Cluster A
+
+### Step 1 - deploy Antrea Multi-cluster Controllers for leader and member
+
+Run the following commands to deploy Multi-cluster Controller for the leader
+into Namespace `antrea-multicluster` (Namespace `antrea-multicluster` will be
+created by the commands), and Multi-cluster Controller for the member into
+Namepsace `kube-system`.
+
+```bash
+$kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-global.yml
+$kubectl create ns antrea-multicluster
+$curl -L https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-namespaced.yml > antrea-multicluster-leader-namespaced.yml
+$sed 's/changeme/antrea-multicluster/g' antrea-multicluster-leader-namespaced.yml | kubectl apply -f -
+$kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-member.yml
+```
+
+You can run the following command to verify the the leader and member
+`antrea-mc-controller` Pods are deployed and running:
+
+```bash
+$kubectl get all -A -l="component=antrea-mc-controller"
+NAMESPACE NAME READY STATUS RESTARTS AGE
+antrea-multicluster pod/antrea-mc-controller-cd7bf8f68-kh4kz 1/1 Running 0 50s
+kube-system pod/antrea-mc-controller-85dbf58b75-pjj48 1/1 Running 0 48s
+
+NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
+antrea-multicluster deployment.apps/antrea-mc-controller 1/1 1 1 50s
+kube-system deployment.apps/antrea-mc-controller 1/1 1 1 48s
+```
+
+### Step 2 - initialize ClusterSet
+
+Antrea provides several template YAML manifests to set up a ClusterSet quicker.
+You can run the following commands that use the template manifests to create a
+ClusterSet named `test-clusteraset` in the leader cluster and get a
+ServiceAccount token for the member clusters (both cluster A and B in our case)
+to access the leader cluster (cluster A in our case) apiserver.
+
+```bash
+$kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_clusterset_template.yaml
+$kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_leader_access_token_template.yaml
+$kubectl get secret leader-access-token -n antrea-multicluster -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e ' ca.crt' -e ' token:' -e '^type' -e ' namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' > leader-access-token.yml
+```
+
+The last command saves the ServiceAccount token to `leader-access-token.yml`
+which will be needed for member clusters to join the ClusterSet. Note, in this
+guide, we use a shared default ServiceAccount `antrea-mc-member-access-sa` for
+all member clusters. If you want to create a separate ServiceAccount for each
+member cluster for security considerations, you can follow the instructions in
+the [Multi-cluster User Guide](user-guide.md#set-up-access-to-leader-cluster).
+
+Next, run the following commands to make cluster A join the ClusterSet also as a
+member:
+
+```bash
+$kubectl apply -f leader-access-token.yml
+$curl -L https://raw.githubusercontent.com/antrea-io/antrea/v1.7.0/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml > multicluster_membercluster.yaml
+$sed -e 's/test-cluster-member/test-cluster-leader/g' -e 's/