[IPv6] Support flow exporter #1541

lzhecheng · 2020-11-12T07:24:22Z

No description provided.

antrea-bot · 2020-11-12T07:24:35Z

Thanks for your PR.
Unit tests and code linters are run automatically every time the PR is updated.
E2e, conformance and network policy tests can only be triggered by a member of the vmware-tanzu organization. Regular contributors to the project should join the org.

The following commands are available:

/test-e2e: to trigger e2e tests.
/skip-e2e: to skip e2e tests.
/test-conformance: to trigger conformance tests.
/skip-conformance: to skip conformance tests.
/test-all-features-conformance: to trigger conformance tests with all alpha features enabled.
/skip-all-features-conformance: to skip conformance tests with all alpha features enabled.
/test-whole-conformance: to trigger all conformance tests on linux.
/skip-whole-conformance: to skip all conformance tests on linux.
/test-networkpolicy: to trigger networkpolicy tests.
/skip-networkpolicy: to skip networkpolicy tests.
/test-windows-conformance: to trigger windows conformance tests.
/skip-windows-conformance: to skip windows conformance tests.
/test-windows-networkpolicy: to trigger windows networkpolicy tests.
/skip-windows-networkpolicy: to skip windows networkpolicy tests.
/test-hw-offload: to trigger ovs hardware offload test.
/skip-hw-offload: to skip ovs hardware offload test.
/test-all: to trigger all tests (except whole conformance).
/skip-all: to skip all tests (except whole conformance).

codecov-io · 2020-11-12T09:02:52Z

Codecov Report

Merging #1541 (7dea211) into master (9d3d10b) will increase coverage by 0.99%.
The diff coverage is 58.73%.

@@            Coverage Diff             @@
##           master    #1541      +/-   ##
==========================================
+ Coverage   63.31%   64.30%   +0.99%     
==========================================
  Files         170      181      +11     
  Lines       14250    15427    +1177     
==========================================
+ Hits         9023     9921     +898     
- Misses       4292     4472     +180     
- Partials      935     1034      +99

Flag	Coverage Δ
e2e-tests	`47.77% <38.24%> (?)`
kind-e2e-tests	`52.60% <51.84%> (-2.79%)`	⬇️
unit-tests	`40.46% <21.37%> (-0.81%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
cmd/antrea-agent/agent.go	`0.00% <0.00%> (ø)`
.../agent/apiserver/handlers/networkpolicy/handler.go	`58.33% <ø> (ø)`
...gent/controller/noderoute/node_route_controller.go	`61.04% <ø> (+14.58%)`	⬆️
pkg/agent/stats/collector.go	`97.72% <ø> (ø)`
pkg/antctl/antctl.go	`100.00% <ø> (ø)`
pkg/antctl/transform/controllerinfo/transform.go	`0.00% <ø> (ø)`
pkg/antctl/transform/version/transform.go	`44.82% <ø> (ø)`
pkg/controller/networkpolicy/tier.go	`90.00% <ø> (ø)`
pkg/features/antrea_features.go	`16.66% <ø> (ø)`
pkg/ovs/openflow/ofctrl_builder.go	`60.94% <0.00%> (-1.23%)`	⬇️
... and 91 more

lxiaopei · 2020-11-12T11:05:49Z

/test-e2e /test-conformance /test-networkpolicy

srikartati

My main comment is about destinationClusterIPv6. Otherwise, it looks good to me.

build/yamls/antrea-aks.yml

pkg/agent/flowexporter/exporter/exporter.go

srikartati

@lzhecheng Currently we consider only the connections in Antrea connection zone of conntrack table and ignore connections in the default zone.

There is a separate connection zone for IPv6. Don't we need to consider that connection zone if the stack is IPv6? Wondering if the flow exporter e2e test is passing with IPv6 stack or not.

lzhecheng · 2020-11-13T08:42:36Z

@lzhecheng Currently we consider only the connections in Antrea connection zone of conntrack table and ignore connections in the default zone.

There is a separate connection zone for IPv6. Don't we need to consider that connection zone if the stack is IPv6? Wondering if the flow exporter e2e test is passing with IPv6 stack or not.

I think I have it?
https://github.com/vmware-tanzu/antrea/pull/1541/files#diff-
fa3c4bf704e90acceacb369eed2fe23b739bd365e8d41ff087becab848371924R244

lzhecheng · 2020-11-13T14:02:00Z

/test-windows-conformance
/test-windows-networkpolicy

srikartati

Thanks for making the changes.
LGTM.

srikartati · 2020-11-13T17:45:01Z

/test-ipv6-all
/test-ipv6-only-all

srikartati · 2020-11-13T17:45:53Z

/test-e2e
/test-conformance
/test-networkpolicy

cmd/antrea-agent/agent.go

lzhecheng · 2020-11-16T01:49:45Z

/test-ipv6-e2e
/test-windows-networkpolicy

lzhecheng · 2020-12-04T10:24:04Z

/test-e2e
/test-ipv6-e2e
/test-ipv6-only-e2e

lzhecheng · 2020-12-04T12:01:10Z

/test-all
/test-ipv6-all
/test-ipv6-only-all

lzhecheng · 2020-12-04T14:02:51Z

/test-conformance
/test-ipv6-e2e
/test-ipv6-networkpolicy

srikartati · 2020-12-04T20:39:57Z

pkg/agent/flowexporter/exporter/exporter.go


 	return sentBytes, nil
 }

-func (exp *flowExporter) sendDataSet(dataSet ipfix.IPFIXSet, record flowexporter.FlowRecord, templateID uint16) error {
+func (exp *flowExporter) sendDataSet(dataSet ipfix.IPFIXSet, record flowexporter.FlowRecord, templateID uint16, isIPv6 bool) error {


Do we need the args templateID and isIPv6? I thinkrecord flowexporter.FlowRecord and exp *flowExporter should have these already.

Yes. Removed.

srikartati · 2020-12-04T20:46:03Z

pkg/agent/flowexporter/exporter/exporter.go

@@ -195,9 +227,15 @@ func (exp *flowExporter) sendFlowRecords() error {
 	return nil
 }

-func (exp *flowExporter) sendTemplateSet(templateSet ipfix.IPFIXSet, templateID uint16) (int, error) {
+func (exp *flowExporter) sendTemplateSet(templateSet ipfix.IPFIXSet, templateID uint16, isIPv6 bool) (int, error) {


templateID is not needed as we assign the templateID to exp *flowExporter in L179 and L189 and can use that inside the function based on isIPv6. Is my understanding correct?

You are right. Updated.

srikartati · 2020-12-04T20:53:47Z

pkg/agent/flowexporter/exporter/exporter.go

+				ie.Value = record.Conn.TupleOrig.DestinationAddress
+			} else {
+				// Same as destinationClusterIPv4.
+				ie.Value = net.IP{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}


nit: simplified version net.IP.Parse("::0")

It seems "::" works as well. Updated.

srikartati · 2020-12-04T20:55:10Z

pkg/agent/flowexporter/types.go

@@ -70,4 +70,6 @@ type FlowRecord struct {
 	PrevBytes          uint64
 	PrevReversePackets uint64
 	PrevReverseBytes   uint64
+	V4Enabled          bool


Just need one field isIPv6 as one flow record corresponds to either v4 or v6.

srikartati · 2020-12-04T20:58:27Z

pkg/agent/flowexporter/flowrecords/flow_records.go

@@ -35,9 +35,32 @@ func NewFlowRecords(connStore *connections.ConnectionStore) *FlowRecords {
 }

 // BuildFlowRecords builds the flow record map from connection map in connection store
-func (fr *FlowRecords) BuildFlowRecords() error {
+func (fr *FlowRecords) BuildFlowRecords(v4Enabled bool, v6Enabled bool) error {


We do not need the args. From IP in key, we can say if it is v4 or v6.

lzhecheng · 2020-12-07T05:21:44Z

/test-e2e
/test-ipv6-e2e
/test-ipv6-only-e2e

lzhecheng · 2020-12-07T06:24:57Z

/test-all
/test-ipv6-all
/test-ipv6-only-all

srikartati · 2020-12-07T07:29:24Z

pkg/agent/flowexporter/flowrecords/flow_records.go

+		fr.recordsMap[key] = record
+		return nil
+	}
+
 	// fr.addOrUpdateFlowRecord method does not return any error, hence no error handling required.


We could change the comment that still says the method name of flow record rather than function name.

Indeed. fr.addOrUpdateFlowRecord -> addOrUpdateFlowRecord

srikartati

Thanks for addressing comments.
LGTM.

lzhecheng · 2020-12-07T23:52:21Z

/test-all
/test-ipv6-all
/test-ipv6-only-all

lzhecheng · 2020-12-08T00:43:02Z

/test-conformance

lzhecheng · 2020-12-08T01:52:01Z

/test-e2e

lzhecheng · 2020-12-08T02:49:38Z

/test-ipv6-only-conformance

lzhecheng · 2020-12-08T03:42:33Z

/test-ipv6-only-conformance

antoninbas

couple nits, otherwise LGTM

antoninbas · 2020-12-08T03:58:08Z

pkg/agent/flowexporter/connections/connections.go

+// Poll calls into conntrackDumper interface to dump conntrack flows. In a dual-stack setup, there will be two connsLens
+// get returned.


suggestion: Poll calls into conntrackDumper interface to dump conntrack flows. It returns the number of connections for each address family, as a slice. In dual-stack clusters, the slice will contain 2 values (number of IPv4 connections first, then number of IPv6 connections).

antoninbas · 2020-12-08T03:58:39Z

pkg/agent/flowexporter/exporter/exporter.go

 	IANAReverseInfoElements = []string{
 		"reversePacketTotalCount",
 		"reverseOctetTotalCount",
 		"reversePacketDeltaCount",
 		"reverseOctetDeltaCount",
 	}
-	AntreaInfoElements = []string{
+	AntreaInfoElementsCommon = []string{


does not need to be exported I think... could be antreaInfoElementsCommon

Indeed. Updated.

antoninbas · 2020-12-08T03:59:46Z

pkg/agent/flowexporter/exporter/exporter_test.go

+
+func getElemList(ianaIE []string, antreaIE []string) []*ipfixentities.InfoElementWithValue {
+	// Following consists of all elements that are in IANAInfoElements and AntreaInfoElements (globals)
+	// Need only element name and other are dummys


s/other are dummys/other fields are set to dummy values

Co-authored-by: Antonin Bas <antonin.bas@gmail.com> Co-authored-by: srikartati <stati@vmware.com>

lzhecheng · 2020-12-08T04:46:18Z

/test-all
/test-ipv6-all
/test-ipv6-only-all

lzhecheng · 2020-12-08T05:02:55Z

/test-conformance
/test-networkpolicy

lzhecheng · 2020-12-08T07:06:14Z

Thank you a lot for the effort to review! @antoninbas @srikartati
Merging this PR.

Co-authored-by: Antonin Bas <antonin.bas@gmail.com> Co-authored-by: srikartati <stati@vmware.com>

lzhecheng requested review from srikartati, zyiou and jianjuns November 12, 2020 07:24

vmwclabot added the cla-not-required label Nov 12, 2020

lzhecheng mentioned this pull request Nov 12, 2020

[IPv6] Support flow exporter #1444

Closed

lzhecheng force-pushed the ipv6-fx branch 2 times, most recently from 019d81a to d14e98a Compare November 12, 2020 08:45

lzhecheng force-pushed the ipv6-fx branch from d14e98a to f9fba1d Compare November 12, 2020 09:07

srikartati reviewed Nov 12, 2020

View reviewed changes

build/yamls/antrea-aks.yml Outdated Show resolved Hide resolved

pkg/agent/flowexporter/exporter/exporter.go Outdated Show resolved Hide resolved

lzhecheng force-pushed the ipv6-fx branch 2 times, most recently from ebc6fbf to 6ed7a7c Compare November 13, 2020 07:40

srikartati reviewed Nov 13, 2020

View reviewed changes

lzhecheng force-pushed the ipv6-fx branch 4 times, most recently from 6580cfa to 1e173eb Compare November 13, 2020 10:10

lzhecheng requested a review from srikartati November 13, 2020 10:12

srikartati previously approved these changes Nov 13, 2020

View reviewed changes

antoninbas reviewed Nov 13, 2020

View reviewed changes

cmd/antrea-agent/agent.go Outdated Show resolved Hide resolved

antoninbas added this to the Antrea v0.12.0 release milestone Nov 13, 2020

srikartati mentioned this pull request Nov 15, 2020

[Flow-Aggregator] Implementation of flow aggregator build system and structure #1558

Merged

wenyingd mentioned this pull request Nov 16, 2020

IPv6 support - including dual stack #136

Closed

lzhecheng force-pushed the ipv6-fx branch 2 times, most recently from 09d082e to 6a50de8 Compare December 4, 2020 10:21

srikartati reviewed Dec 4, 2020

View reviewed changes

lzhecheng force-pushed the ipv6-fx branch 3 times, most recently from d4de2f4 to a579a20 Compare December 7, 2020 04:57

srikartati reviewed Dec 7, 2020

View reviewed changes

lzhecheng force-pushed the ipv6-fx branch from a579a20 to 7a36512 Compare December 7, 2020 07:57

srikartati previously approved these changes Dec 7, 2020

View reviewed changes

antoninbas reviewed Dec 8, 2020

View reviewed changes

[IPv6] Support flow exporter

7dea211

Co-authored-by: Antonin Bas <antonin.bas@gmail.com> Co-authored-by: srikartati <stati@vmware.com>

lzhecheng dismissed srikartati’s stale review via 7dea211 December 8, 2020 04:08

lzhecheng force-pushed the ipv6-fx branch from 7a36512 to 7dea211 Compare December 8, 2020 04:08

antoninbas approved these changes Dec 8, 2020

View reviewed changes

lzhecheng merged commit 501bd0b into antrea-io:master Dec 8, 2020

lzhecheng deleted the ipv6-fx branch December 8, 2020 07:06

antoninbas added a commit that referenced this pull request Dec 23, 2020

[IPv6] Support flow exporter (#1541)

64bf6df

Co-authored-by: Antonin Bas <antonin.bas@gmail.com> Co-authored-by: srikartati <stati@vmware.com>

		// Poll calls into conntrackDumper interface to dump conntrack flows. In a dual-stack setup, there will be two connsLens
		// get returned.

[IPv6] Support flow exporter #1541

[IPv6] Support flow exporter #1541

Conversation

lzhecheng commented Nov 12, 2020 • edited Loading

antrea-bot commented Nov 12, 2020

codecov-io commented Nov 12, 2020 • edited Loading

Codecov Report

lxiaopei commented Nov 12, 2020

srikartati left a comment

Choose a reason for hiding this comment

srikartati left a comment

Choose a reason for hiding this comment

lzhecheng commented Nov 13, 2020

lzhecheng commented Nov 13, 2020

srikartati left a comment

Choose a reason for hiding this comment

srikartati commented Nov 13, 2020

srikartati commented Nov 13, 2020

lzhecheng commented Nov 16, 2020

lzhecheng commented Dec 4, 2020

lzhecheng commented Dec 4, 2020

lzhecheng commented Dec 4, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lzhecheng commented Dec 7, 2020

lzhecheng commented Dec 7, 2020

srikartati Dec 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikartati left a comment

Choose a reason for hiding this comment

lzhecheng commented Dec 7, 2020

lzhecheng commented Dec 8, 2020

lzhecheng commented Dec 8, 2020

lzhecheng commented Dec 8, 2020

lzhecheng commented Dec 8, 2020

antoninbas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lzhecheng commented Dec 8, 2020

lzhecheng commented Dec 8, 2020

lzhecheng commented Dec 8, 2020

lzhecheng commented Nov 12, 2020 •

edited

Loading

codecov-io commented Nov 12, 2020 •

edited

Loading

srikartati Dec 7, 2020 •

edited

Loading