Fix Pod-to-external traffic on EKS in policyOnly mode #3975
Conversation
antoninbas
added
area/provider/aws
Codecov Report
@@ Coverage Diff @@
## main #3975 +/- ##
==========================================
+ Coverage 61.65% 64.88% +3.22%
==========================================
Files 295 295
Lines 43740 44115 +375
==========================================
+ Hits 26968 28623 +1655
+ Misses 14538 13192 -1346
- Partials 2234 2300 +66
|
antoninbas
force-pushed
the
eks-fix-pod-to-external-traffic
branch
from
688808f
to
ee5c0c7
Compare
When using Antrea in policyOnly mode on an EKS cluster, an additional iptables rule is needed in the PREROUTING chain of the nat table. The rule ensures that Pod-to-external traffic coming from Pods whose IP address comes from a secondary network interface (secondary ENI) is marked correctly, so that it hits the appropriate routing table. Without this, traffic is SNATed with the source IP address of the primary network interface, while being sent out of the secondary network interface, causing the VPC to drop the traffic. The fix is to add new PREROUTING rules, in the ANTREA-PREROUTING chain: ``` -A ANTREA-PREROUTING -i antrea-gw0 -m comment --comment "Antrea: AWS, outbound connections" -j AWS-CONNMARK-CHAIN-0 -A ANTREA-PREROUTING -m comment --comment "Antrea: AWS, CONNMARK (first packet)" -j CONNMARK --restore-mark --nfmask 0x80 --ctmask 0x80 ``` Fixes antrea-io#3946 Signed-off-by: Antonin Bas <abas@vmware.com>
antoninbas
force-pushed
the
eks-fix-pod-to-external-traffic
branch
from
ee5c0c7
to
200197a
Compare
| c.writeEKSMangleRule(iptablesData) | ||
| // When Antrea is used to enforce NetworkPolicies in EKS, additional iptables | ||
| // mangle rules are required. See https://github.com/antrea-io/antrea/issues/678. | ||
| // These rules are only needed for IPv4. |
There was a problem hiding this comment.
Does it have NodePort IPv6 case? If yes, is the rule still needed for IPv6, otherwise reverse path may be different?
There was a problem hiding this comment.
yes, I think you're right. This one should always be installed.
Note that we don't test Antrea on EKS with IPv6 enabled, so there may be other issues, but I'll fix that one.
There was a problem hiding this comment.
Actually scratch that. The AWS rules are only installed for IPv4: https://github.com/aws/amazon-vpc-cni-k8s/blob/d43309bdfdb5034df86907944e682d78608ba165/pkg/networkutils/network.go#L402-L418. Not entirely sure why, but I would think that routing is configured differently for IPv6, and these rules are not needed.
There was a problem hiding this comment.
There is this comment:
//Essentially a stub function for now in V6 mode. We will need it when we support v6 in secondary IP and
//custom networking modes. We don't need to install any SNAT rules in v6 mode and currently there is no need
//to mark packets entering via Primary ENI as all the pods in v6 mode will be behind primary ENI. Will have to
//start doing that once we start supporting custom networking mode in v6.
So that explains why there is no need for the rule: only the primary ENI is used, with no secondary route table.
| // While we do not have access to these environment variables, we could | ||
| // look for existing rules installed by the AWS VPC CNI, and determine | ||
| // whether we need to install this rule. |
There was a problem hiding this comment.
Is this still true? I didn't find this code.
There was a problem hiding this comment.
This is part of the TODO paragraph. We don't implement this as the moment, we assume that aws-node (the AWS DaemonSet) is running with default configuration, which is likely to be the case for 99% of users.
|
@tnqn feel free to merge this if you're ok with the change. The EKS CI test is passing for this PR. |
|
/test-all |
When using Antrea in policyOnly mode on an EKS cluster, an additional
iptables rule is needed in the PREROUTING chain of the nat table. The
rule ensures that Pod-to-external traffic coming from Pods whose IP
address comes from a secondary network interface (secondary ENI) is
marked correctly, so that it hits the appropriate routing table. Without
this, traffic is SNATed with the source IP address of the primary
network interface, while being sent out of the secondary network
interface, causing the VPC to drop the traffic.
Relevant rules (before the fix):
The fix is to add new PREROUTING rules, in the ANTREA-PREROUTING chain:
Fixes #3946
Signed-off-by: Antonin Bas abas@vmware.com