[IPv6] Support flow exporter #1444

lzhecheng · 2020-10-28T09:28:01Z

Support IPv4 or IPv6 flow exporter address.

antrea-bot · 2020-10-28T09:28:16Z

Thanks for your PR.
Unit tests and code linters are run automatically every time the PR is updated.
E2e, conformance and network policy tests can only be triggered by a member of the vmware-tanzu organization. Regular contributors to the project should join the org.

The following commands are available:

/test-e2e: to trigger e2e tests.
/skip-e2e: to skip e2e tests.
/test-conformance: to trigger conformance tests.
/skip-conformance: to skip conformance tests.
/test-whole-conformance: to trigger all conformance tests on linux.
/skip-whole-conformance: to skip all conformance tests on linux.
/test-networkpolicy: to trigger networkpolicy tests.
/skip-networkpolicy: to skip networkpolicy tests.
/test-windows-conformance: to trigger windows conformance tests.
/skip-windows-conformance: to skip windows conformance tests.
/test-windows-networkpolicy: to trigger windows networkpolicy tests.
/skip-windows-networkpolicy: to skip windows networkpolicy tests.
/test-hw-offload: to trigger ovs hardware offload test.
/skip-hw-offload: to skip ovs hardware offload test.
/test-all: to trigger all tests (except whole conformance).
/skip-all: to skip all tests (except whole conformance).

lzhecheng · 2020-10-28T09:31:12Z

@antoninbas could you please help push a newer version of ipfix-collector? --enable-ipv6 is needede for libipfix.
https://github.com/vmware-tanzu/antrea/pull/1444/files#diff-f6c3a6c603b9babc9ec5a69428cc251e937906c8e25e2ea753afbdb69795a9f8

codecov-io · 2020-10-28T10:20:22Z

Codecov Report

Merging #1444 into ipv6 will decrease coverage by 26.10%.
The diff coverage is 22.82%.

@@             Coverage Diff             @@
##             ipv6    #1444       +/-   ##
===========================================
- Coverage   67.71%   41.61%   -26.11%     
===========================================
  Files         165       85       -80     
  Lines       13060    10729     -2331     
===========================================
- Hits         8844     4465     -4379     
- Misses       3284     5864     +2580     
+ Partials      932      400      -532

Flag	Coverage Δ
integration-tests	`?`
kind-e2e-tests	`?`
unit-tests	`41.61% <22.82%> (-0.68%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/agent/agent.go	`12.20% <0.00%> (-37.65%)`	⬇️
pkg/agent/config/node_config.go	`0.00% <0.00%> (-100.00%)`	⬇️
pkg/agent/util/iptables/iptables.go	`0.00% <0.00%> (-62.50%)`	⬇️
pkg/agent/util/net.go	`31.81% <0.00%> (-32.47%)`	⬇️
pkg/agent/util/net_linux.go	`0.00% <0.00%> (-46.25%)`	⬇️
pkg/antctl/antctl.go	`100.00% <ø> (ø)`
pkg/ovs/openflow/ofctrl_builder.go	`21.34% <0.00%> (-56.95%)`	⬇️
pkg/agent/openflow/pipeline.go	`21.17% <12.44%> (-59.98%)`	⬇️
pkg/agent/openflow/client.go	`10.38% <37.03%> (-57.48%)`	⬇️
pkg/agent/flowexporter/connections/connections.go	`64.48% <50.00%> (-17.07%)`	⬇️
... and 146 more

antoninbas · 2020-10-28T20:01:42Z

@lzhecheng please review and merge #1449
I have already pushed the new tag (antrea/ipfix-collector:10282020.1) to Dockerhub

lzhecheng · 2020-10-29T01:47:56Z

/test-e2e
/test-ipv6-e2e
/test-ipv6-only-e2e

lzhecheng · 2020-10-29T06:31:59Z

/test-e2e
/test-ipv6-e2e
/test-ipv6-only-e2e

lzhecheng · 2020-10-29T11:05:31Z

/test-e2e
/test-ipv6-e2e
/test-ipv6-only-e2e

lzhecheng · 2020-10-30T01:59:31Z

/test-e2e
/test-ipv6-e2e
/test-ipv6-only-e2e

srikartati

Thanks for the PR, Zhecheng.

As Antrea Proxy plans to support IPv6, "destinationClusterIP" IPFIX field in Antrea registry need to support both fields IPv4 and IPv6. Currently, in Antrea registry, we have only IPv4 field for clusterIP. Will raise a PR in go-ipfix to support both fields.

srikartati · 2020-10-30T10:15:54Z

cmd/antrea-agent/options_test.go

+			[]string{"1.2.3.4", "80"},
+		},
+		{
+			"[fe80:1:2::3]:80:tcp",


Don't we need 16 bytes for IPv6 slice?

I think we are using string as input and output here?

My question was about number of bytes in considered address: fe80:1:2::3. I think rest of the bytes are 0s in 16 byte address--is that correct? Maybe a better example for IPv6 address is to consider whole 16 byte address?

Yes. Got it. Updated.

srikartati · 2020-10-30T10:23:19Z

pkg/agent/flowexporter/exporter/exporter_test.go

 }

 // TestFlowExporter_sendDataRecord tests essentially if element names in the switch-case matches globals
-// IANAInfoElements and AntreaInfoElements.
+// IANAInfoElementsIPv4 and AntreaInfoElements.


why no test with IANAInfoElementsIPv6? Are you waiting for the fix in go-ipfix?

Make sense. Updated test for IPv6.

srikartati · 2020-10-30T10:36:39Z

test/e2e/flowexporter_test.go

@@ -132,7 +131,7 @@ func checkRecordsWithPodIPs(t *testing.T, data *TestData, podAIP string, podBIP
 			templateRecords = templateRecords + 1
 		}

-		if strings.Contains(record, podAIP) && strings.Contains(record, podBIP) {
+		if strings.Contains(record, lengthenIPv6Addr(isIPv6, podAIP)) && strings.Contains(record, lengthenIPv6Addr(isIPv6, podBIP)) {


Are we seeing correct IPv6 address in flow records?
With the current go-ipfix code, ipv6 addresses do not get encoded on to flow records in the exporter code.
https://github.com/vmware/go-ipfix/blob/master/pkg/entities/ie.go#L390

I see that ipv6 encoding is being supported in release-0.2.0 branch, which is currently used in Antrea. However, it is missing in the master branch. This needs to be fixed if we move to future releases.

https://github.com/vmware/go-ipfix/blob/release-0.2.0/pkg/entities/record.go#L208

Ignore the above comment.

Just FYI, the master branch issue is taken care of in this PR: vmware/go-ipfix#64

srikartati

We support destination clusterIP IPv6 address in this patch release: https://github.com/vmware/go-ipfix/releases/tag/v0.2.3
Please use that to add IPv6 destination cluster IP for Pod-To-Service flows.

srikartati · 2020-11-02T11:47:30Z

cmd/antrea-agent/options_test.go

+			[]string{"1.2.3.4", "80"},
+		},
+		{
+			"[fe80:1:2::3]:80:tcp",


My question was about number of bytes in considered address: fe80:1:2::3. I think rest of the bytes are 0s in 16 byte address--is that correct? Maybe a better example for IPv6 address is to consider whole 16 byte address?

srikartati · 2020-11-02T11:48:55Z

test/e2e/flowexporter_test.go

@@ -132,7 +131,7 @@ func checkRecordsWithPodIPs(t *testing.T, data *TestData, podAIP string, podBIP
 			templateRecords = templateRecords + 1
 		}

-		if strings.Contains(record, podAIP) && strings.Contains(record, podBIP) {
+		if strings.Contains(record, lengthenIPv6Addr(isIPv6, podAIP)) && strings.Contains(record, lengthenIPv6Addr(isIPv6, podBIP)) {


Just FYI, the master branch issue is taken care of in this PR: vmware/go-ipfix#64

…ed (antrea-io#1460)

* ping failure when crossing nodes * incorrect ipset: without CIDR of current node * TestPodTrafficShaping for IPv6 * TestAntctlProxy: skip if ipv6 * DeleteRoutes parameter change

antrea-bot · 2020-11-04T09:01:06Z

Thanks for your PR.
Unit tests and code linters are run automatically every time the PR is updated.
E2e, conformance and network policy tests can only be triggered by a member of the vmware-tanzu organization. Regular contributors to the project should join the org.

The following commands are available:

/test-e2e: to trigger e2e tests.
/skip-e2e: to skip e2e tests.
/test-conformance: to trigger conformance tests.
/skip-conformance: to skip conformance tests.
/test-all-features-conformance: to trigger conformance tests with all alpha features enabled.
/skip-all-features-conformance: to skip conformance tests with all alpha features enabled.
/test-whole-conformance: to trigger all conformance tests on linux.
/skip-whole-conformance: to skip all conformance tests on linux.
/test-networkpolicy: to trigger networkpolicy tests.
/skip-networkpolicy: to skip networkpolicy tests.
/test-windows-conformance: to trigger windows conformance tests.
/skip-windows-conformance: to skip windows conformance tests.
/test-windows-networkpolicy: to trigger windows networkpolicy tests.
/skip-windows-networkpolicy: to skip windows networkpolicy tests.
/test-hw-offload: to trigger ovs hardware offload test.
/skip-hw-offload: to skip ovs hardware offload test.
/test-all: to trigger all tests (except whole conformance).
/skip-all: to skip all tests (except whole conformance).

Support IPv4 or IPv6 flow exporter address.

srikartati · 2020-11-05T10:22:17Z

I see there are a lot of commits are added to the PR--a total of 86 files. Is this PR still valid? Is it because of rebasing the ipv6 branch with the master?

lzhecheng · 2020-11-06T05:51:11Z

I see there are a lot of commits are added to the PR--a total of 86 files. Is this PR still valid? Is it because of rebasing the ipv6 branch with the master?

Yes, it is due to ipv6 branch updating to the master. This PR should rebase then.

jianjuns · 2020-11-10T00:39:52Z

cmd/antrea-agent/options.go

@@ -185,7 +192,10 @@ func (o *Options) validateFlowExporterConfig() error {
 			return fmt.Errorf("IPFIX flow collector address should be provided")
 		} else {
 			// Check if it is TCP or UDP
-			strSlice := strings.Split(o.config.FlowCollectorAddr, ":")
+			strSlice, err := parseFlowCollectorAddr(o.config.FlowCollectorAddr)


Maybe add a comment in antrea-agent.conf about IPv6 is supported for flowCollectorAddr.

jianjuns · 2020-11-10T00:45:24Z

pkg/agent/flowexporter/connections/connections.go

@@ -189,7 +192,11 @@ func (cs *ConnectionStore) Poll() (int, error) {
 	// We do not expect any error as resetConn is not returning any error
 	cs.ForAllConnectionsDo(resetConn)

-	filteredConnsList, totalConns, err := cs.connDumper.DumpFlows(openflow.CtZone)
+	var zone uint16 = openflow.CtZone
+	if cs.serviceCIDR != nil && cs.serviceCIDR.IP.To4() == nil {


If ConnectionStore.serviceCIDR is used only here, we just need to pass and save a v4 or v6 flag in the ConnectionStore struct?

Make sense. Updated.

lzhecheng · 2020-11-12T07:25:07Z

PR #1541 replaces this PR after ipv6 branch is merged to master.

lzhecheng requested review from antoninbas, wenyingd, srikartati and zyiou October 28, 2020 09:28

vmwclabot added the cla-not-required label Oct 28, 2020

lzhecheng force-pushed the ipv6-fx branch 2 times, most recently from 812b60d to b4c2fc3 Compare October 28, 2020 10:16

lzhecheng force-pushed the ipv6-fx branch 2 times, most recently from 91c42c5 to caab675 Compare October 29, 2020 01:47

lzhecheng force-pushed the ipv6-fx branch from caab675 to 2857f60 Compare October 29, 2020 04:01

lzhecheng force-pushed the ipv6-fx branch from 2857f60 to 29c73df Compare October 29, 2020 11:04

lzhecheng force-pushed the ipv6-fx branch from 29c73df to 39a8dbb Compare October 30, 2020 01:41

srikartati reviewed Oct 30, 2020

View reviewed changes

srikartati mentioned this pull request Oct 30, 2020

Support IPv6 addresses vmware/go-ipfix#15

Closed

lzhecheng force-pushed the ipv6-fx branch 3 times, most recently from 091d85e to 1a52ad3 Compare November 2, 2020 09:41

srikartati reviewed Nov 2, 2020

View reviewed changes

lzhecheng force-pushed the ipv6-fx branch 4 times, most recently from 30fe68c to 2145879 Compare November 3, 2020 02:17

Skip 2 Network Policy testcases before Network Policy IPv6 is support…

b3aa933

…ed (antrea-io#1460)

lzhecheng force-pushed the ipv6 branch from 08a0f84 to fc10cfc Compare November 4, 2020 06:46

[IPv6] Fix issues

3c3ec42

* ping failure when crossing nodes * incorrect ipset: without CIDR of current node * TestPodTrafficShaping for IPv6 * TestAntctlProxy: skip if ipv6 * DeleteRoutes parameter change

lzhecheng force-pushed the ipv6 branch from fc10cfc to 3c3ec42 Compare November 4, 2020 07:58

lzhecheng force-pushed the ipv6-fx branch from 2145879 to e609826 Compare November 4, 2020 09:00

lzhecheng force-pushed the ipv6-fx branch from e609826 to 79a8040 Compare November 4, 2020 09:19

[IPv6] Support flow exporter

c03a22f

Support IPv4 or IPv6 flow exporter address.

lzhecheng force-pushed the ipv6-fx branch from 79a8040 to c03a22f Compare November 4, 2020 09:36

lzhecheng force-pushed the ipv6 branch from 3c3ec42 to e7b753d Compare November 5, 2020 08:02

lzhecheng force-pushed the ipv6 branch from cb5d679 to 20fba03 Compare November 9, 2020 11:19

jianjuns reviewed Nov 10, 2020

View reviewed changes

lzhecheng force-pushed the ipv6 branch 13 times, most recently from dd85a9e to 267d1c0 Compare November 11, 2020 09:24

lzhecheng closed this Nov 11, 2020

lzhecheng deleted the branch antrea-io:ipv6 November 11, 2020 23:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IPv6] Support flow exporter #1444

[IPv6] Support flow exporter #1444

lzhecheng commented Oct 28, 2020

antrea-bot commented Oct 28, 2020

lzhecheng commented Oct 28, 2020

codecov-io commented Oct 28, 2020 •

edited

Loading

antoninbas commented Oct 28, 2020

lzhecheng commented Oct 29, 2020

lzhecheng commented Oct 29, 2020

lzhecheng commented Oct 29, 2020

lzhecheng commented Oct 30, 2020

srikartati left a comment

srikartati Oct 30, 2020

lzhecheng Nov 2, 2020 •

edited

Loading

srikartati Nov 2, 2020

lzhecheng Nov 3, 2020

srikartati Oct 30, 2020

lzhecheng Nov 2, 2020

srikartati Oct 30, 2020

srikartati Oct 30, 2020

srikartati Nov 2, 2020

srikartati left a comment

srikartati Nov 2, 2020

srikartati Nov 2, 2020

antrea-bot commented Nov 4, 2020

srikartati commented Nov 5, 2020

lzhecheng commented Nov 6, 2020

jianjuns Nov 10, 2020

lzhecheng Nov 12, 2020

jianjuns Nov 10, 2020

lzhecheng Nov 12, 2020

lzhecheng commented Nov 12, 2020

[IPv6] Support flow exporter #1444

[IPv6] Support flow exporter #1444

Conversation

lzhecheng commented Oct 28, 2020

antrea-bot commented Oct 28, 2020

lzhecheng commented Oct 28, 2020

codecov-io commented Oct 28, 2020 • edited Loading

Codecov Report

antoninbas commented Oct 28, 2020

lzhecheng commented Oct 29, 2020

lzhecheng commented Oct 29, 2020

lzhecheng commented Oct 29, 2020

lzhecheng commented Oct 30, 2020

srikartati left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lzhecheng Nov 2, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikartati left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antrea-bot commented Nov 4, 2020

srikartati commented Nov 5, 2020

lzhecheng commented Nov 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lzhecheng commented Nov 12, 2020

codecov-io commented Oct 28, 2020 •

edited

Loading

lzhecheng Nov 2, 2020 •

edited

Loading