Use symmetric return path for non-VPC traffic - alternate solution

[kishorj]

What type of PR is this?
bug

Which issue does this PR fix:
Fixes #1392, #1423

What does this PR do / Why do we need it:
When external SNAT is disabled, this PR does the following

• setup iptables rules to mark egress traffic from pods that are destined to external network
• external traffic will be sent through the primary interface

This is an alternate fix. It doesn't require permissive rp filter configuration on the secondary ENIs, or a new connmark.

If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:
N/A

Testing done on this change:

• When external SNAT is enabled, the iptables conmark rules added to the nat table get cleaned up

• With External SNAT disabled

  • connmark rules get added to the nat table
  • pods are able to initiate connections to hosts outside the VPC, traffic is through the primary interface
  • in case of NLB-IP, verify traffic flows as expected for client IP preservation enabled/disabled, pods on primary/secondary interfaces
  • UDP traffic works fine
  • instance mode NLB works as expected - verified externalTrafficPolicy Cluster, Local, IP preservation enabled/disabled, pods on primary/secondary interfaces
  • in-cluster traffic works as expected
  • enabled security group per pod, verified pods can communicate within the cluster, pods can receive client IP preserved traffic from the NLB

• With External SNAT enabled

  • connmark and SNAT rules get removed from the nat table
  • instance mode NLB works fine with IP preserve enabled/disabled, pods on primary/secondary interfaces
  • IP mode NLB works as expected with IP preserve enabled/disabled, pods on primary/secondary interfaces
  • pod can communicate to hosts within or outside VPC
  • enabled security group per pod, verified traffic flows

• New integration tests pass after some modification

Automation added to e2e:

Will this break upgrades or downgrades. Has updating a running cluster been tested?:
No impact on upgrade/downgrade. On, downgrade conn mark rules in the nat table remain until node is rebooted, but there is no impact on the functionality.

Does this change require updates to the CNI daemonset config files to work?:

Does this PR introduce any user-facing change?:

Fixes the issue with non-VPC traffic ingress from secondary ENIs when external SNAT feature is not used.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jayanthvn]

Lets wait for the integration tests to finish.