Add Gateway, ClusterInfo exchange controllers #3463

luolanzone · 2022-03-17T03:22:45Z

Add a Node controller to watch Node changes in member cluster. It will do following things:
- When the Node is annotated with multicluster.antrea.io/gateway=true, it will
  create a new Gateway CR like below:
```
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: Gateway
metadata:
  name: k8s-node-1
  namespace: kube-system
gatewayIP: 172.16.27.224
internalIP: 172.16.27.224
```
- When the Node is annotated with both multicluster.antrea.io/gateway=true and multicluster.antrea.io/gateway-ip="10.10.10.10" ('10.10.10.10' is an IP example), the existing Gateway CR's gatewayIP will be updated like below:
```
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: Gateway
metadata:
  name: k8s-node-1
  namespace: kube-system
gatewayIP: 10.10.10.10
internalIP: 172.16.27.224
```
When the Node's annotation multicluster.antrea.io/gateway=true is removed, the corresponding Gateway will be deleted.
Add a Gateway controller to watch Gateway events in member cluster. It will do following things:
- When a Gateway is created, a new ClusterInfo type of ResourceExport will be created in leader cluster, a sample yaml is like below:
```
apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ResourceExport
metadata:
  name: test-cluster-west-kube-system-clusterinfo
  namespace: antrea-mcs-ns
spec:
  clusterID: test-cluster-west
  clusterinfo:
    clusterID: test-cluster-west
    gatewayInfos:
    - gatewayIP: 10.10.10.10
    serviceCIDR: 10.19.0.0/18
  kind: ClusterInfo
  name: test-cluster-west
  namespace: kube-system
```
- When a Gateway is updated or one of many Gateways are deleted, the corresponding ResourceExport will be updated in leader cluster.
  Please note, we support one Gateway only for now, so the last created Gateway will be wrapped into ResourceExport if there are multiple Gateways.
- When the last Gateway is deleted, the corresponding ResourceExport will be deleted in leader cluster.
There are two new fields serviceCIDR and gatewayIPPrecedence are added in MultiClusterConfig.
- By default, MC controller will detect ClusterIP range automatically. If admin sets the serviceCIDR config manually, it will use the value of serviceCIDR in the config.
- By default, MC controller will choose InternalIP of a Node as the GatewayIP, if admin sets gatewayIPPrecedence config as public, it will use ExternalIP as GatewayIP.
Add a new ClusterInfo kind handler to convert any new or updated ClusterInfo kind of ResourceExports into ResourceImports.

Add a new ClusterInfo importer to watch any new or updated ClusterInfo kind of ResourceImport events. It will create or update a ClusterInfoImport locally.
A sample ClusterInfoImport is like below:

apiVersion: multicluster.crd.antrea.io/v1alpha1
kind: ClusterInfoImport
metadata:
  name: test-cluster-west-kube-system-clusterinfo
  namespace: kube-system
spec:
  clusterID: test-cluster-west
  gatewayInfos:
  - gatewayIP: 10.10.10.10
  serviceCIDR: 10.19.0.0/18

Signed-off-by: Lan Luo luola@vmware.com

luolanzone · 2022-03-17T03:28:30Z

Hi @jianjuns I am still working on refining the codes and unit test, Could you help to take a look at the overview in summary? maybe a quick glance over codes, thanks!

codecov-commenter · 2022-03-17T04:35:21Z

Codecov Report

Merging #3463 (2e185a9) into main (65134a6) will decrease coverage by 7.65%.
The diff coverage is 52.83%.

@@            Coverage Diff             @@
##             main    #3463      +/-   ##
==========================================
- Coverage   64.63%   56.97%   -7.66%     
==========================================
  Files         279      398     +119     
  Lines       39733    55915   +16182     
==========================================
+ Hits        25680    31858    +6178     
- Misses      12059    21555    +9496     
- Partials     1994     2502     +508

Flag	Coverage Δ
integration-tests	`38.22% <ø> (?)`
kind-e2e-tests	`52.85% <ø> (-0.27%)`	⬇️
unit-tests	`43.87% <52.83%> (+0.12%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
multicluster/cmd/multicluster-controller/member.go	`0.00% <0.00%> (ø)`
...ulticluster/cmd/multicluster-controller/options.go	`9.30% <0.00%> (-1.51%)`	⬇️
...lers/multicluster/commonarea/remote_common_area.go	`26.01% <0.00%> (-0.11%)`	⬇️
multicluster/controllers/multicluster/test_data.go	`100.00% <ø> (ø)`
...uster/controllers/multicluster/controller_utils.go	`27.55% <10.41%> (-19.01%)`	⬇️
...ter/controllers/multicluster/gateway_controller.go	`43.82% <43.82%> (ø)`
...llers/multicluster/member_clusterset_controller.go	`10.13% <48.93%> (+10.13%)`	⬆️
...ticluster/commonarea/remote_common_area_manager.go	`59.61% <50.00%> (-0.79%)`	⬇️
...luster/controllers/multicluster/node_controller.go	`62.50% <62.50%> (ø)`
...rs/multicluster/commonarea/clusterinfo_importer.go	`63.82% <63.82%> (ø)`
... and 151 more

jianjuns

Do you have a design doc? I could not understand the functions of TunnelEndpoints and its attributes, just from the code comments (and probably you should add more comments in code too).

multicluster/apis/multicluster/v1alpha1/tunnelendpoint_types.go

luolanzone · 2022-03-22T08:37:57Z

Hi @jianjuns I have drafted a new google doc with data path design https://docs.google.com/document/d/1Lmm-Vh67QXiyGDftwJfEPS0kpK8oEVZsAjtcLVGrhfw/edit, Could you help to take a look? thanks.
I will address your comments and work on unit test.

multicluster/apis/multicluster/v1alpha1/tunnelendpoint_types.go

luolanzone · 2022-04-14T06:15:59Z

/test-multicluster-e2e

multicluster/apis/multicluster/v1alpha1/tunnelendpoint_types.go

multicluster/apis/multicluster/v1alpha1/multiclusterconfig_types.go

multicluster/apis/multicluster/v1alpha1/resourceexport_types.go

multicluster/apis/multicluster/v1alpha1/tunnelendpointimport_types.go

luolanzone · 2022-04-21T07:11:35Z

@jianjuns Could you help to review again? I have refined codes based on latest design. thanks.

multicluster/controllers/multicluster/member_clusterset_controller.go

luolanzone · 2022-05-17T06:14:49Z

/test-multicluster-e2e

multicluster/controllers/multicluster/commonarea/clusterinfo_importer_test.go

multicluster/controllers/multicluster/serviceexport_controller.go

multicluster/controllers/multicluster/stale_controller.go

luolanzone · 2022-05-18T06:05:23Z

/test-multicluster-e2e

luolanzone · 2022-05-18T07:12:04Z

/test-multicluster-e2e

multicluster/controllers/multicluster/node_controller.go

luolanzone · 2022-05-19T02:22:27Z

/test-multicluster-e2e

jianjuns

LGTM. @tnqn might have further comments.

multicluster/controllers/multicluster/node_controller.go

multicluster/controllers/multicluster/member_clusterset_controller.go

* Add a Node controller to watch Node changes in member cluster. It will do following things: * When the Node is annotated with `multicluster.antrea.io/gateway=true`, it will create a new Gateway CR like below: ```yaml apiVersion: multicluster.crd.antrea.io/v1alpha1 kind: Gateway metadata: name: k8s-node-1 namespace: kube-system gatewayIP: 172.16.27.224 internalIP: 172.16.27.224 ``` * When the Node is annotated with both `multicluster.antrea.io/gateway=true` and `multicluster.antrea.io/gateway-ip="10.10.10.10"` ('10.10.10.10' is an IP example), the existing Gateway CR's gatewayIP will be updated like below: ```yaml apiVersion: multicluster.crd.antrea.io/v1alpha1 kind: Gateway metadata: name: k8s-node-1 namespace: kube-system gatewayIP: 10.10.10.10 internalIP: 172.16.27.224 ``` * When the Node's annotation `multicluster.antrea.io/gateway=true` is removed, the corresponding Gateway will be deleted. * Add a Gateway controller to watch Gateway events in member cluster. It will do following things: * When a Gateway is created, a new ClusterInfo type of ResourceExport will be created in leader cluster, a sample yaml is like below: ```yaml apiVersion: multicluster.crd.antrea.io/v1alpha1 kind: ResourceExport metadata: name: test-cluster-west-kube-system-clusterinfo namespace: antrea-mcs-ns spec: clusterID: test-cluster-west clusterinfo: clusterID: test-cluster-west gatewayInfos: - gatewayIP: 10.10.10.10 serviceCIDR: 10.19.0.0/18 kind: ClusterInfo name: test-cluster-west namespace: kube-system ``` * When a Gateway is updated or one of many Gateways are deleted, the corresponding ResourceExport will be updated in leader cluster. Please note, we support one Gateway only for now, so the last created Gateway will be wrapped into ResourceExport if there are multiple Gateways. * When the last Gateway is deleted, the corresponding ResourceExport will be deleted in leader cluster. * There are two new fields `serviceCIDR` and `gatewayIPPrecedence` are added in `MultiClusterConfig`. - By default, MC controller will detect ClusterIP range automatically. If admin sets the `serviceCIDR` config manually, it will use the value of `serviceCIDR` in the config. - By default, MC controller will choose InternalIP of a Node as the GatewayIP, if admin sets `gatewayIPPrecedence` config as `public`, it will use ExternalIP as GatewayIP. * Add a new ClusterInfo kind handler to convert any new or updated ClusterInfo kind of ResourceExports into ResourceImports. * Add a new ClusterInfo importer to watch any new or updated ClusterInfo kind of ResourceImport events. It will create or update a ClusterInfoImport locally. A sample ClusterInfoImport is like below: ```yaml apiVersion: multicluster.crd.antrea.io/v1alpha1 kind: ClusterInfoImport metadata: name: test-cluster-west-kube-system-clusterinfo namespace: kube-system spec: clusterID: test-cluster-west gatewayInfos: - gatewayIP: 10.10.10.10 serviceCIDR: 10.19.0.0/18 ``` Signed-off-by: Lan Luo <luola@vmware.com>

luolanzone · 2022-05-20T04:49:51Z

/test-multicluster-e2e

tnqn

LGTM

tnqn · 2022-05-20T04:51:44Z

/skip-all

luolanzone · 2022-05-20T05:34:17Z

/test-multicluster-e2e

luolanzone · 2022-05-20T05:59:47Z

/test-multicluster-e2e

luolanzone · 2022-05-20T06:23:29Z

Hi @tnqn there was an e2e stale data issue in the environment, I cleaned them up and rerun job is passed, Could you help to move forward? thanks.

luolanzone marked this pull request as draft March 17, 2022 03:26

luolanzone changed the title ~~Add TunnelEndpoint&Import CRDs and controllers~~ [WIP]Add TunnelEndpoint&Import CRDs and controllers Mar 17, 2022

luolanzone added the area/multi-cluster Issues or PRs related to multi cluster. label Mar 17, 2022

jianjuns reviewed Mar 18, 2022

View reviewed changes

luolanzone force-pushed the tunnel-endpoint branch 4 times, most recently from a8634c9 to debb541 Compare March 25, 2022 08:56

jianjuns reviewed Mar 31, 2022

View reviewed changes

multicluster/apis/multicluster/v1alpha1/tunnelendpoint_types.go Outdated Show resolved Hide resolved

multicluster/apis/multicluster/v1alpha1/tunnelendpoint_types.go Outdated Show resolved Hide resolved

luolanzone force-pushed the tunnel-endpoint branch 5 times, most recently from b047dbb to 5dd5619 Compare April 7, 2022 10:03

luolanzone marked this pull request as ready for review April 8, 2022 06:33

luolanzone changed the title ~~[WIP]Add TunnelEndpoint&Import CRDs and controllers~~ Add TunnelEndpoint&Import CRDs and controllers Apr 8, 2022

luolanzone requested review from Dyanngg and hjiajing April 8, 2022 06:33

luolanzone mentioned this pull request Apr 8, 2022

Multicluster dataplane change for Service access #3603

Merged

luolanzone force-pushed the tunnel-endpoint branch from 5dd5619 to e4bdfe4 Compare April 14, 2022 06:15

jianjuns reviewed Apr 14, 2022

View reviewed changes

jianjuns reviewed Apr 15, 2022

View reviewed changes

luolanzone force-pushed the tunnel-endpoint branch 2 times, most recently from 2d49415 to cd9edcd Compare April 21, 2022 07:06

luolanzone changed the title ~~Add TunnelEndpoint&Import CRDs and controllers~~ Add GatewayNode, ClusterInfo exchange process Apr 21, 2022

jianjuns reviewed May 16, 2022

View reviewed changes

multicluster/controllers/multicluster/member_clusterset_controller.go Outdated Show resolved Hide resolved

multicluster/controllers/multicluster/member_clusterset_controller.go Outdated Show resolved Hide resolved

luolanzone force-pushed the tunnel-endpoint branch from ede610e to 004393b Compare May 17, 2022 06:14

tnqn reviewed May 17, 2022

View reviewed changes

luolanzone force-pushed the tunnel-endpoint branch from 004393b to 75d8352 Compare May 18, 2022 05:58

luolanzone force-pushed the tunnel-endpoint branch from 75d8352 to 73ec2c0 Compare May 18, 2022 06:42

jianjuns reviewed May 18, 2022

View reviewed changes

multicluster/controllers/multicluster/node_controller.go Outdated Show resolved Hide resolved

multicluster/controllers/multicluster/node_controller.go Show resolved Hide resolved

luolanzone force-pushed the tunnel-endpoint branch 2 times, most recently from bb915c5 to 2e185a9 Compare May 19, 2022 02:13

jianjuns reviewed May 19, 2022

View reviewed changes

multicluster/controllers/multicluster/node_controller.go Outdated Show resolved Hide resolved

multicluster/controllers/multicluster/node_controller.go Outdated Show resolved Hide resolved

multicluster/controllers/multicluster/node_controller.go Outdated Show resolved Hide resolved

tnqn reviewed May 19, 2022

View reviewed changes

multicluster/controllers/multicluster/member_clusterset_controller.go Outdated Show resolved Hide resolved

luolanzone force-pushed the tunnel-endpoint branch 2 times, most recently from 446142c to a4c1986 Compare May 19, 2022 15:36

luolanzone force-pushed the tunnel-endpoint branch from a4c1986 to a24474f Compare May 20, 2022 04:49

tnqn approved these changes May 20, 2022

View reviewed changes

tnqn added action/release-note Indicates a PR that should be included in release notes. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API. kind/feature Categorizes issue or PR as related to a new feature. labels May 20, 2022

tnqn merged commit 4f15ba1 into antrea-io:main May 20, 2022

luolanzone deleted the tunnel-endpoint branch May 24, 2022 06:13

luolanzone mentioned this pull request Jun 15, 2022

Release 1.7.0 #3895

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Gateway, ClusterInfo exchange controllers #3463

Add Gateway, ClusterInfo exchange controllers #3463

luolanzone commented Mar 17, 2022 •

edited

Loading

luolanzone commented Mar 17, 2022 •

edited

Loading

codecov-commenter commented Mar 17, 2022 •

edited

Loading

jianjuns left a comment

luolanzone commented Mar 22, 2022

luolanzone commented Apr 14, 2022

luolanzone commented Apr 21, 2022

luolanzone commented May 17, 2022

luolanzone commented May 18, 2022

luolanzone commented May 18, 2022

luolanzone commented May 19, 2022

jianjuns left a comment

luolanzone commented May 20, 2022

tnqn left a comment

tnqn commented May 20, 2022

luolanzone commented May 20, 2022

luolanzone commented May 20, 2022

luolanzone commented May 20, 2022

Add Gateway, ClusterInfo exchange controllers #3463

Add Gateway, ClusterInfo exchange controllers #3463

Conversation

luolanzone commented Mar 17, 2022 • edited Loading

luolanzone commented Mar 17, 2022 • edited Loading

codecov-commenter commented Mar 17, 2022 • edited Loading

Codecov Report

jianjuns left a comment

Choose a reason for hiding this comment

luolanzone commented Mar 22, 2022

luolanzone commented Apr 14, 2022

luolanzone commented Apr 21, 2022

luolanzone commented May 17, 2022

luolanzone commented May 18, 2022

luolanzone commented May 18, 2022

luolanzone commented May 19, 2022

jianjuns left a comment

Choose a reason for hiding this comment

luolanzone commented May 20, 2022

tnqn left a comment

Choose a reason for hiding this comment

tnqn commented May 20, 2022

luolanzone commented May 20, 2022

luolanzone commented May 20, 2022

luolanzone commented May 20, 2022

luolanzone commented Mar 17, 2022 •

edited

Loading

luolanzone commented Mar 17, 2022 •

edited

Loading

codecov-commenter commented Mar 17, 2022 •

edited

Loading