clusterip service not work after running big service(1000 endpoints) with antrea-proxy enabled along with kube-proxy

[luwang-vmware]

Describe the bug
I enabled antrea-proxy along with kube-proxy in our k8s clusters and create big services with 250 endpoints, 1000 endpoints and 5000 endpoints. After the testing, the clusterip service functionality is broken.

To Reproduce

1. create 500 worker nodes cluster in TKGM1.3 with antrea-proxy enabled
2. download clusterloader2 source code and run the scripts of dep_svc_l.yaml, dep_svc_xl.yaml and dep_svc_exl.yaml by sequential.

Expected
clusterip service still works.

Actual behavior
after the testing, I found that the grafana can not show the dashboard, due to grafana failed to access the prometheus clusterip service. The clusterip service function is broken.
I can succeed to access the pod's ip but failed to curl its service ip.

root@systest-runner:/tmp[40430]# kubectl  get pods -n monitoring -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP             NODE                                NOMINATED NODE   READINESS GATES
grafana-69c5568879-hd2sm              1/1     Running   0          18h   100.96.213.3   workload-21-md-0-7cf7d99f68-snf2m   <none>           <none>
prometheus-k8s-0                      3/3     Running   1          18h   100.96.17.3    workload-21-md-0-7cf7d99f68-57jcs   <none>           <none>
prometheus-operator-7bf997fc6-vlk89   1/1     Running   0          18h   100.97.179.3   workload-21-md-0-7cf7d99f68-bxb6b   <none>           <none>
root@systest-runner:/tmp[40430]# kubectl  get svc -n monitoring
NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                       AGE
grafana               NodePort    100.64.30.207    <none>        3000:32066/TCP                18h
master                ClusterIP   None             <none>        6443/TCP,2379/TCP,10250/TCP   18h
prometheus-k8s        NodePort    100.66.117.106   <none>        9090:31820/TCP                18h
prometheus-operated   ClusterIP   None             <none>        9090/TCP                      18h
prometheus-operator   ClusterIP   None             <none>        8080/TCP                      18h

/ $ curl http://100.96.17.3:9090
<a href="/graph">Found</a>.

/ $ curl http://100.66.117.106:9090
curl: (28) Failed to connect to 100.66.117.106 port 9090: Operation timed out

#Versions:
Please provide the following information:

• Antrea version (Docker image tag). v0.11.3
• Kubernetes version (use kubectl version). If your Kubernetes components have [different versions] v1.20.4
  (https://kubernetes.io/docs/setup/release/version-skew-policy/), please provide the version for all of them.
• Container runtime: which runtime are you using (e.g. containerd, cri-o, docker) and which version are you using? containerd
• Linux kernel version on the Kubernetes Nodes (uname -r). 5.4.0-66-generic
• If you chose to compile the Open vSwitch kernel module manually instead of using the kernel module built into the Linux kernel, which version of the OVS kernel module are you using? Include the output of modinfo openvswitch for the Kubernetes Nodes.

Additional context
Add any other context about the problem here, such as Antrea logs, kubelet logs, etc.

kubectl  exec -it antrea-agent-2r7rt -n kube-system -c antrea-ovs --  ovs-ofctl dump-groups br-int
NXST_GROUP_DESC reply (xid=0x2):
kubectl  exec -it antrea-agent-2r7rt -n kube-system -c antrea-ovs -- ovs-ofctl dump-flows br-int --no-stats --names > out2.log

Checking the antrea error log in the /var/log/antrea in each worker node, we could see the errors as following

E0413 15:30:09.010772       1 proxier.go:280] Error when installing Endpoints groups: error when installing Service Endpoints Group: message is canceled because of disconnection from the Switch
E0413 15:30:09.012386       1 proxier.go:275] Error when installing Endpoints flows: message is canceled because of disconnection from the Switch
E0413 15:30:10.077826       1 client.go:670] Error when replaying cached group 5: message is canceled because of disconnection from the Switch
E0413 15:30:10.077861       1 client.go:670] Error when replaying cached group 1: message is canceled because of disconnection from the Switch
E0413 15:30:10.077871       1 client.go:670] Error when replaying cached group 8: message is canceled because of disconnection from the Switch
E0413 15:30:10.077879       1 client.go:670] Error when replaying cached group 7: message is canceled because of disconnection from the Switch
E0413 15:30:10.077976       1 client.go:670] Error when replaying cached group 27: message is canceled because of disconnection from the Switch
E0413 15:30:10.077985       1 client.go:670] Error when replaying cached group 9: message is canceled because of disconnection from the Switch
E0413 15:30:10.077994       1 client.go:670] Error when replaying cached group 3: message is canceled because of disconnection from the Switch
E0413 15:30:10.078001       1 client.go:670] Error when replaying cached group 6: message is canceled because of disconnection from the Switch
E0413 15:30:10.078008       1 client.go:670] Error when replaying cached group 2: message is canceled because of disconnection from the Switch
E0413 15:30:12.819763       1 client.go:663] Error when replaying cached flows: one message in bundle failed
E0413 15:30:13.342754       1 client.go:663] Error when replaying cached flows: one message in bundle failed
E0413 15:30:13.439770       1 client.go:663] Error when replaying cached flows: one message in bundle failed
E0413 15:30:13.681335       1 client.go:663] Error when replaying cached flows: one message in bundle failed
E0413 15:30:14.100804       1 client.go:663] Error when replaying cached flows: one message in bundle failed
E0413 15:30:14.453300       1 client.go:663] Error when replaying cached flows: one message in bundle failed
E0413 15:30:14.604515       1 client.go:663] Error when replaying cached flows: one message in bundle failed
E0413 15:30:14.731602       1 client.go:663] Error when replaying cached flows: one message in bundle failed
E0413 15:30:14.770858       1 client.go:663] Error when replaying cached flows: one message in bundle failed

Whole bundle of test cases and log refer to tarball.
log.tar.gz

(Please consider pasting long output into a GitHub gist or any other pastebin.)

[luwang-vmware]

Tested that 500 endpoints per service(deployment) works fine. 1000 endpoints per service could consistently repo this issue.

[tnqn]

Thanks @luwang-vmware for finding it and providing a way to reproduce it consistently! We will prioritize fixing this issue.

cc @jianjuns @antoninbas

[hongliangl]

I have made a test with ovs. I used the below script to generate a group with X buckets, and I use the generated command to run in antrea-ovs container.

#!/bin/bash
FILE="ovs-group.sh"
LAST=$1
echo "#!/bin/bash" > $FILE
echo "ovs-ofctl add-group br-int group_id=88,type=select,\\" >> $FILE

for (( i=0; i<`expr $LAST - 1`; i++ ))
do
      echo "bucket=bucket_id:$i,weight:100,actions=load:0xaf40204\-\>NXM_NX_REG3[],load:0x50\-\>NXM_NX_REG4[0..15],load:0x2\-\>NXM_NX_REG4[16..18],load:0x1\-\>NXM_NX_REG0[19],resubmit\(,42\),\\" >> $FILE
done
      echo "bucket=bucket_id:`expr $LAST - 1`,weight:100,actions=load:0xaf40204\-\>NXM_NX_REG3[],load:0x50\-\>NXM_NX_REG4[0..15],load:0x2\-\>NXM_NX_REG4[16..18],load:0x1\-\>NXM_NX_REG0[19],resubmit\(,42\)" >> $FILE

kubectl cp $FILE -n kube-system antrea-agent-rjll6:/root -c antrea-ovs

• When the number of buckets < 512, it works
• When the number of buckets >= 512, the output is stuck, and long long time later, the output is below.

root@k8s-node-control-plane:~# wc -l ovs-group.sh && bash ovs-group.sh
514 1000-items.sh
ovs-ofctl: talking to unix:/var/run/openvswitch/br-int.mgmt (End of file)

I made a test to verify that if the action in table 41 can be set with multiple groups, like this: ovs-ofctl add-flow br-int table=41,priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.102.219.80,tp_dst=80,actions=group:7,group:8.

This command is legal, but the test indicated that the group 8 never hits.

@tnqn @jianjuns @antoninbas

[antoninbas]

I think you are just reaching the max OpenFlow message size which is 64K bytes.
Without getting too accurate :), 64K bytes / 512 buckets is 128 bytes. Given that each bucket has a 16-byte header and that you also have 5 actions per bucket, you pretty much get to 128 bytes.
You can run your ovs-ofctl command with --verbose=dbg to get more information. You should also specify -O OpenFlow13 to avoid getting unrelated error messages.

At the end of the day, this needs to be fixed in Antrea: we need to stop setting all the buckets using one group mod message, as we quickly reach the max message size. My guess is that libOpenflow will map our API calls to a single message, hence the problem. We can set the max number of buckets per message to something like 128 or 256, and use several insert-buckets in the same bundle to achieve the same result.

[hongliangl]

Thanks @antoninbas @tnqnf. I first thought of the idea of ​​solving this problem by modifying function func (self *Group) Install() error in github/contiv/ofnet/ofctrl/fgraphGroup.go like this below.

func (self *Group) Install() error {
	command := openflow13.OFPGC_ADD
	if self.isInstalled {
		command = openflow13.OFPGC_MODIFY
	}
	groupMod := self.getGroupModMessage(command)

	if len(groupMod.Buckets) <= maxBucketsInsert {
		if err := self.Switch.Send(groupMod); err != nil {
			return err
		}
	} else {
		insertBuckets := groupMod.Buckets[maxBucketsInsert:]
		groupMod.Buckets = groupMod.Buckets[:maxBucketsInsert]
		if err := self.Switch.Send(groupMod); err != nil {
			return err
		}

		groupMod.Command = uint16(openflow13.OFPGC_INSERT_BUCKET)
		insertBucketsNum := len(insertBuckets)
		for i := 0; i < insertBucketsNum; i += maxBucketsInsert {
			start, end := i, (i+1)*maxBucketsInsert
			if end > insertBucketsNum {
				end = insertBucketsNum
			}

			groupMod.Buckets = insertBuckets[start:end]
			if err := self.Switch.Send(groupMod); err != nil {
				return err
			}
		}
	}

	// Mark it as installed
	self.isInstalled = true

	return nil
}

After discussed with @tnqn , he suggested that the group buckets should be added/inserted in a bundle way. I'm working on this now.

[hongliangl]

Unfortunately, guys, I'm stuck with bundle issue. There are 2 issues:

1. I didn't find the reason that why when added a group in bundle, deleting the group with groupID failed, but deleting it in bundle way can work.
2. INSERT-BUCKET doesn't work as expected. I write a simple test in PR Extends the Endpoints support from 500 to 800, extra ones will be dropped in AntreaProxy #2101 https://github.com/vmware-tanzu/antrea/blob/b320a4dd63b8b51f9510ff74b1734897ae58663e/test/integration/agent/openflow_test.go#L487, the test has 4 endpoint, and I set the max number of inserting buckets is 3 for temp. The last one should be added by INSERT-BUCKET, but it failed. The output is below:

=== RUN   TestGroup
I0416 18:35:50.426456    2012 entry.go:314] Initialize connection or re-connect to /var/run/openvswitch/br01.mgmt.
I0416 18:35:51.427242    2012 entry.go:314] Connected to socket /var/run/openvswitch/br01.mgmt
I0416 18:35:51.427587    2012 entry.go:359] New connection..
I0416 18:35:51.427848    2012 entry.go:314] Send hello with OF version: 4
I0416 18:35:51.427953    2012 entry.go:359] Received Openflow 1.3 Hello message
I0416 18:35:51.428613    2012 entry.go:314] Received ofp1.3 Switch feature response: {Header:{Version:4 Type:6 Length:32 Xid:3} DPID:00:00:f6:81:90:ef:37:48 Buffers:0 NumTables:254 AuxilaryId:0 pad:[0 0] Capabilities:79 Actions:0 Ports:[]}
I0416 18:35:51.428717    2012 entry.go:359] Openflow Connection for new switch: 00:00:f6:81:90:ef:37:48
I0416 18:35:51.429269    2012 ofctrl_bridge.go:221] OFSwitch is connected: 00:00:f6:81:90:ef:37:48
E0416 18:35:51.433889    2012 entry.go:314] Received Vendor error: OFPBFC_MSG_FAILED on ONFT_BUNDLE_CONTROL message
    openflow_test.go:543:
                Error Trace:    openflow_test.go:543
                Error:          Received unexpected error:
                                error when installing Service Endpoints Group: one message in bundle failed
                Test:           TestGroup
                Messages:       no error should return when installing groups for Service
--- FAIL: TestGroup (1.02s)
FAIL
E0416 18:35:51.434699    2012 entry.go:314] Received Vendor error: NXBRC_UNENCODABLE_ERROR on ONFT_BUNDLE_ADD_MESSAGE message
I0416 18:35:51.434753    2012 entry.go:359] Controller is delete
coverage: 3.1% of statements in github.com/vmware-tanzu/antrea/pkg/...
FAIL    github.com/vmware-tanzu/antrea/test/integration/agent   1.062s
testing: warning: no tests to run
PASS
coverage: 1.2% of statements in github.com/vmware-tanzu/antrea/pkg/...
ok      github.com/vmware-tanzu/antrea/test/integration/ovs     0.043s  coverage: 1.2% of statements in github.com/vmware-tanzu/antrea/pkg/... [no tests to run]
FAIL

Could anyone give me some suggestions? Thanks.

[antoninbas]

I wanted to take a look at your code, but really couldn't make sense of the following:

	for i := insertBucketNum; i < endpointsNum; i += insertBucketNum {
		group := c.bridge.CreateGroup(fakeGroupId).ResetBuckets()
		c.bridge.DeleteGroup(fakeGroupId)
		group.SetGroupId(groupID)
		groups = append(groups, group)
	}

Why is there a fakeGroupId and why are we deleting it several times?

I think the interface needs to change so that AntreaProxy can provide a list of endpoints / buckets to insert and a list of endpoints / buckets to remove. The we generate the appropriate insertions & deletions. We shouldn't be deleting the group and re-adding it every time.

[tnqn]

Agree with @antoninbas's question. @hongliangl For your second question, I can see the code in the PR installs a group twice: one is by bucketBuilder.Done, another by bundle, which might be the root cause. BTW, Could you raise code related issue in the PR in the future so there is a context and the discussion can be tracked properly?

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days