-
Notifications
You must be signed in to change notification settings - Fork 363
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update multi-cluster user guide and templates #3853
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,200 @@ | ||
# Antrea Multi-cluster Quick Start | ||
|
||
In this quick start guide, we will set up an Antrea Multi-cluster ClusterSet | ||
with two clusters. One cluster will serve as the leader of the ClusterSet, and | ||
meanwhile also join as a member cluster; another cluster will be a member only. | ||
|
||
The diagram below shows the two clusters and the ClusterSet to be created (for | ||
simplicity, the diagram just shows two Nodes for each cluster). | ||
|
||
<img src="assets/sample-clusterset.svg" width="1000" alt="Antrea Multi-cluster Example ClusterSet"> | ||
|
||
## Preparation | ||
|
||
We assume an Antrea version >= `v1.7.0` is used in this guide, and the Antrea | ||
version is set to an environment variable `TAG`. For example, the following | ||
command sets the Antrea version to `v1.7.0`. | ||
|
||
```bash | ||
export TAG=v1.7.0 | ||
``` | ||
|
||
To use the latest version of Antrea Multi-cluster from the Antrea main branch, | ||
you can change the YAML manifest path to: `https://github.com/antrea-io/antrea/tree/main/multicluster/build/yamls/` | ||
when applying or downloading an Antrea YAML manifest. | ||
|
||
Antrea must be deployed in both cluster A and cluster B. To configure Antrea | ||
Multi-cluster Gateways, the `Multicluster` feature of `antrea-agent` must be | ||
feature enabled in both cluster A and B. Multi-cluster Gateways are required for | ||
routing multi-cluster Service traffic across the member clusters. Set the | ||
following configuration parameters in `antrea-agent.conf` of the Antrea | ||
deployment manifest to enable the `Multicluster` feature: | ||
|
||
```yaml | ||
antrea-agent.conf: | | ||
... | ||
featureGates: | ||
... | ||
Multicluster: true | ||
... | ||
multicluster: | ||
enable: true | ||
namespace: "" | ||
``` | ||
|
||
## Set up Leader and Member in Cluster A | ||
|
||
### Step 1 - deploy Antrea Multi-cluster Controllers for leader and member | ||
|
||
Run the following commands to deploy Multi-cluster Controller for the leader | ||
into Namespace `antrea-multicluster` (Namespace `antrea-multicluster` will be | ||
created by the commands), and Multi-cluster Controller for the member into | ||
Namepsace `kube-system`. | ||
|
||
```bash | ||
$kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-global.yml | ||
$kubectl create ns antrea-multicluster | ||
$curl -L https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-namespaced.yml > antrea-multicluster-leader-namespaced.yml | ||
$sed 's/changeme/antrea-multicluster/g' antrea-multicluster-leader-namespaced.yml | kubectl apply -f - | ||
$kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-member.yml | ||
``` | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It might be good to add a command to check the deployment results that can show both leader & member mc-controller Deployments and Pods. Does something like work: What should be the "xyz". There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. sure, I will add it, it can be There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, better to have a sample output. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
You can run the following command to verify the the leader and member | ||
`antrea-mc-controller` Pods are deployed and running: | ||
|
||
```bash | ||
$kubectl get all -A -l="component=antrea-mc-controller" | ||
NAMESPACE NAME READY STATUS RESTARTS AGE | ||
antrea-multicluster pod/antrea-mc-controller-cd7bf8f68-kh4kz 1/1 Running 0 50s | ||
kube-system pod/antrea-mc-controller-85dbf58b75-pjj48 1/1 Running 0 48s | ||
|
||
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE | ||
antrea-multicluster deployment.apps/antrea-mc-controller 1/1 1 1 50s | ||
kube-system deployment.apps/antrea-mc-controller 1/1 1 1 48s | ||
``` | ||
|
||
### Step 2 - initialize ClusterSet | ||
|
||
Antrea provides several template YAML manifests to set up a ClusterSet quicker. | ||
You can run the following commands that use the template manifests to create a | ||
ClusterSet named `test-clusteraset` in the leader cluster and get a | ||
ServiceAccount token for the member clusters (both cluster A and B in our case) | ||
to access the leader cluster (cluster A in our case) apiserver. | ||
|
||
```bash | ||
$kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_clusterset_template.yaml | ||
$kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_leader_access_token_template.yaml | ||
$kubectl get secret leader-access-token -n antrea-multicluster -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e ' ca.crt' -e ' token:' -e '^type' -e ' namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' > leader-access-token.yml | ||
``` | ||
|
||
The last command saves the ServiceAccount token to `leader-access-token.yml` | ||
which will be needed for member clusters to join the ClusterSet. Note, in this | ||
guide, we use a shared default ServiceAccount `antrea-mc-member-access-sa` for | ||
all member clusters. If you want to create a separate ServiceAccount for each | ||
member cluster for security considerations, you can follow the instructions in | ||
the [Multi-cluster User Guide](user-guide.md#set-up-access-to-leader-cluster). | ||
|
||
Next, run the following commands to make cluster A join the ClusterSet also as a | ||
member: | ||
|
||
```bash | ||
$kubectl apply -f leader-access-token.yml | ||
$curl -L https://raw.githubusercontent.com/antrea-io/antrea/v1.7.0/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml > multicluster_membercluster.yaml | ||
$sed -e 's/test-cluster-member/test-cluster-leader/g' -e 's/<LEADER_CLUSTER_IP>/172.10.0.11/g' multicluster_membercluster.yaml | kubectl apply -f - | ||
``` | ||
|
||
Here, `172.10.0.11` is the `kube-apiserver` IP of cluster A. You should replace | ||
it with the `kube-apiserver` IP of your leader cluster. | ||
|
||
### Step 3 - specify Multi-cluster Gateway Node | ||
|
||
Last, you need to choose a Node in cluster A to serve as the Multi-cluster | ||
Gateway. The Node should have an IP that is reachable from the cluster B's | ||
Gateway Node, so a tunnel can be created between the two Gateways. For more | ||
information about Multi-cluster Gatweay, please refer to the [Multi-cluster | ||
User Guide](user-guide.md#multi-cluster-gateway-configuration). | ||
|
||
Assuming K8s Node `node-a1` is selected for the Multi-cluster Gateway, run | ||
the following command to annotate the Node with | ||
`multicluster.antrea.io/gateway=true` (so Antrea can know it is the Gateway | ||
Node from the annotation): | ||
|
||
```bash | ||
$kubectl annotate node node-a1 multicluster.antrea.io/gateway=true | ||
``` | ||
|
||
## Set up Cluster B | ||
|
||
Let us switch to cluster B. All the `kubectl` commands in the following steps | ||
should be run with the `kubeconfig` for cluster B. | ||
|
||
### Step 1 - deploy Antrea Multi-cluster Controller for member | ||
|
||
Run the following command to deploy the member Multi-cluster Controller into | ||
Namespace `kube-system`. | ||
|
||
```bash | ||
$kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-member.yml | ||
``` | ||
|
||
You can run the following command to verify the `antrea-mc-controller` Pod is | ||
deployed and running: | ||
|
||
```bash | ||
$kubectl get all -A -l="component=antrea-mc-controller" | ||
NAMESPACE NAME READY STATUS RESTARTS AGE | ||
kube-system pod/antrea-mc-controller-85dbf58b75-pjj48 1/1 Running 0 40s | ||
|
||
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE | ||
kube-system deployment.apps/antrea-mc-controller 1/1 1 1 40s | ||
``` | ||
|
||
### Step 2 - initialize ClusterSet | ||
|
||
Run the following commands to make cluster B join the ClusterSet: | ||
|
||
```bash | ||
$kubectl apply -f leader-access-token.yml | ||
$curl -L https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml > multicluster_membercluster.yaml | ||
$sed -e 's/<LEADER_CLUSTER_IP>/172.10.0.11/g' multicluster_membercluster.yaml | kubectl apply -f - | ||
``` | ||
|
||
`leader-access-token.yml` saves the leader cluster ServiceAccount token which | ||
was generated when initializing the ClusterSet in cluster A. | ||
|
||
### Step 3 - specify Multi-cluster Gateway Node | ||
|
||
Assuming K8s Node `node-b1` is chosen to be the Multi-cluster Gateway for cluster | ||
B, run the following command to annotate the Node: | ||
|
||
```bash | ||
$kubectl annotate node node-b1 multicluster.antrea.io/gateway=true | ||
``` | ||
|
||
## What is Next | ||
|
||
So far, we set up an Antrea Multi-cluster ClusterSet with two clusters following | ||
the above sections of this guide. Next, you can start to consume the Antrea | ||
Multi-cluster features with the ClusterSet, including [Multi-cluster Services](user-guide.md#multi-cluster-service) | ||
and [ClusterNetworkPolicy Replication](user-guide.md#multi-cluster-clusternetworkpolicy-replication). | ||
Check the relevant Antrea Multi-cluster User Guide sections to learn more. | ||
|
||
If you want to add a new member cluster to your ClusterSet, you can follow the | ||
steps for cluster B to do so. But note, you will need the following two changes: | ||
|
||
1. You need to add the new mumber cluster to the ClusterSet in the leader | ||
cluster (cluster A). You can do that by adding the cluster ID of the new member | ||
to `multicluster_clusterset_template.yaml` and re-applying the manifest in | ||
cluster A. | ||
|
||
2. You need to update the member cluster ID in | ||
`multicluster_membercluster_template.yaml` to the cluster ID of the new member | ||
cluster in the step 2 of initializing ClusterSet. For example, you can run the | ||
following commands to the following to initialize the ClusterSet for a member | ||
cluster with ID `test-cluster-member2`: | ||
|
||
```bash | ||
$kubectl apply -f leader-access-token.yml | ||
$curl -L https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml > multicluster_membercluster.yaml | ||
$sed -e 's/<LEADER_CLUSTER_IP>/172.10.0.11/g' -e 's/test-cluster-member/test-cluster-member2/g' multicluster_membercluster.yaml | kubectl apply -f - | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to apply antrea-multicluster-leader-global.yml for a dual-role cluster? In the user guide, earlier you wrote it is necessary, as antrea-multicluster-member.yml already import all CRDs.
If we do need to apply all YAMLs and the order does not matter, I would apply leader YAMLs first, which is easier to understand. @luolanzone
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, we removed those unnecessary CRDs which was both in leader and member yamls, so we need to apply antrea-multicluster-leader-global.yml for a dual-role cluster, I will move it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done, the diagram is also updated.