L7 Network Policy logs for allowed network policy rules

[Edward033]

Hi all,

I'm trying to inspect the http traffic of our pods with L7 Network Policy. I want to understand the http hosts (domains) being hit by our workloads.

Based on my tests, L7 network policy logs only show the http host when the policy is rejected on '/var/log/antrea/networkpolicy/l7engine/eve-YYYY-MM-DD.json'.

The allowed logs go to '/var/log/antrea/networkpolicy/np.log' with the rest of the network policy logs. Is there any setting I can change that would show me the http host for permitted traffic? Or is exposing these logs on the roadmap?

[antoninbas]

@Edward033 If your main / only goal is to inspect HTTP requests, defining a L7NetworkPolicy may not be the best way to go about it. That being said, my understanding is that if you enable logging correctly for your policy rule, then there should be logs for all requests, not just rejected ones. I will give it a try and get back to you.

/var/log/antrea/networkpolicy/np.log will not include any L7 information (HTTP data)

Other alternatives that may work for you:

1. in Antrea v1.15, the FlowExporter supports L7: https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#layer-7-network-flow-exporter. You can enable this feature and then enable centralized logging at the FlowAggregator. The nice part here is that all logs will be in the same place.
2. you can also deploy your own Suricata on each Node (as a DaemonSet) and configure traffic redirection from Antrea to Suricata. You can configure Suricata as you want this way, and define whichever rules you want. We have detailed documentation here: https://github.com/antrea-io/antrea/tree/main/docs/cookbooks/ids

[Edward033]

Thanks @antoninbas for the quick reply! Please keep me posted about your tests if you don't mind!

So far, I've been unable to obtain logs for permitted traffic running Antrea version 1.13.1.

Here is my Network Policy at a high level:

apiVersion: crd.antrea.io/v1alpha1
kind: ClusterNetworkPolicy
metadata:
  name: allow-http-traffic
spec:
  priority: 10
  tier: securityops
  appliedTo:
    - podSelector:
           matchLabels:
                run: debug
  egress:
      - name: test-http
         to: 
            - fqdn: proxy-test.internal.com
         ports:
             - protocol: TCP
               port: 80
          action: Allow
          l7Protocols:
             - http:
                   host: "*.com"
           enableLogging: true

Logs on /var/log/antrea/networkpolicy/l7engine/eve-YYYY-MM-DD.json only show rejected logs for me. I might be missing something.

Thanks for the additional options. Given we're running 1.13.1, those don't apply to us, but will keep them in mind as we progress with our Antrea releases.

[antoninbas]

That being said, my understanding is that if you enable logging correctly for your policy rule, then there should be logs for all requests, not just rejected ones.

Looks like my understanding was not correct. I tested with Antrea v1.15.0, and I could only see logs for rejected connections:

{"timestamp":"2024-02-12T23:02:53.103848+0000","flow_id":679088614117877,"in_iface":"antrea-l7-tap0","event_type":"alert","vlan":[1],"src_ip":"10.10.1.7","src_port":60434,"dest_ip":"10.10.1.8","dest_port":80,"proto":"TCP","alert":{"action":"blocked","gid":1,"signature_id":1,"rev":0,"signature":"Reject by AntreaNetworkPolicy:default/ingress-allow-http-request-to-api-v2","category":"","severity":3,"tenant_id":1},"http":{"hostname":"10.10.1.8","url":"/","http_user_agent":"curl/7.81.0","http_method":"PUT","protocol":"HTTP/1.1","length":0},"app_proto":"http","flow":{"pkts_toserver":3,"pkts_toclient":1,"bytes_toserver":279,"bytes_toclient":74,"start":"2024-02-12T23:02:53.102901+0000"}}
{"timestamp":"2024-02-12T23:02:53.111078+0000","flow_id":679088614117877,"in_iface":"antrea-l7-tap0","event_type":"packet","vlan":[1],"src_ip":"10.10.1.8","src_port":80,"dest_ip":"10.10.1.7","dest_port":60434,"proto":"TCP","packet":"mgwhHRhoTtwelujqCABFAAAorjYAAEAGtncKCgEICgoBBwBQ7BI2rPK1gcIRTFAUAfvu3wAA","packet_info":{"linktype":1}}

It seems that the only effect of setting enableLogging: true is that we get a second event with type "packet". The documentation doesn't say anything about this. We will have to check with @tnqn to see if this is the intended behavior. He is on vacation right now but will be back next week.

Thanks for the additional options. Given we're running 1.13.1, those don't apply to us, but will keep them in mind as we progress with our Antrea releases.

The second option of running your own Suricata does not require the most recent Antrea version.
There is an advantage in using the Mirror action instead of the Redirect action when sending traffic to Suricata. Mirror does not require disabling checksum offloading, while Redirect does, which could have an impact on throughput.
When you use L7NetworkPolicy, the Redirect action is used, and disabling checksum offloading is required. But when you only care about inspecting traffic (not potentially blocking it), the Mirror traffic can be used. Both option 1 & 2 described above can work with Mirror. Now I understand that option 1 is not available to you, but you may want to consider option 2.

[antoninbas]

Assigning to @tnqn to get his comments on logging pass actions in suricata.

[Edward033]

Hey @antoninbas, thanks! You're correct. I'll explore the suricata option!

Do you have any documentation or do you mind elaborating on the implications of disabling checksum offloading? We want to initially inspect the http traffic, but at some point, if the feature is feasible, we want to enforce it as well.

Thanks again!

[antoninbas]

Do you have any documentation or do you mind elaborating on the implications of disabling checksum offloading?

AFAIK the only implication would be an impact on throughput. I am not sure we have some data available. In the end, it's also likely to depend on your setup.

This requirement is documented in https://github.com/antrea-io/antrea/blob/main/docs/antrea-l7-network-policy.md#prerequisites, and there is a bit more context in #4231:

One potential issue is, TX checksum offloading is enabled by default in most environments, but when Suricata uses AF_PACKET to capture traffic, the physical network interface will not populate checksum for the packets after they are processed by Suricata userspace and sent back to datapath. There is a configuration parameter "disableTXChecksumOffload" in antrea-agent to disable TX checksum offloading, but it will only affect new Pods. Users would have to set the configuration parameter properly when deploying the cluster, otherwise there would be communication problems after L7NetworkPolicy is used. We need to check if there are other solutions to avoid the issue, or document the requirements clearly and evaluate the performance impact of disabling TX checksum offloading for all Pods.

[Edward033]

Hi @tnqn, @antoninbas

I've been trying to wrap my head around TX checksum offloading and how it relates to our L7 network policy deployment. We have TX checksum offloading enabled on our NIC:

ethtool -k eth0
...
Features for eth0:
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
		tx-checksum-ip-generic: on
		tx-checksum-ipv6: off [fixed]
		tx-checksym-fcoe-crc: off [fixed]
		tx-checksym-sctp: off [fixed]

One potential issue, is TX checksum offloading is enabled by default in most environments, but when Suricata uses AF_PACKET to capture traffic, the physical network interface will not populate checksum for the packets after they are process by Suricata userspace and sent back to datapath

What I understand from this sentence it's that L7 Network Policy (Suricata) prevents TX checksum offloading.

Is the Kernel or OVS unable to handle checksum calculations then? Or can the Kernel or OVS still handle the checksum calculations, just potentially slower?

"There is a configuration parameter "disableTXChecksumOffload" in antrea-agent to disable TX checksum offloading, but it will only affect new Pods. Users would have to set the configuration parameter properly when deploying the cluster otherwise there would be communication problems after L7NetworkPolicy is used."

What is the benefit of enabling this parameter in the antrea-agent config? I'm guessing OVS now has the ability to delegate checksum calculations to the kernel, instead of "assuming" the NIC would perform the offload, and causing the potential problems you stated above?

// Disable TX checksum offloading for container network interfaces. It's supposed to be set to true when the
// datapath doesn't support TX checksum offloading, which causes packets ot be dropped due to bad checksum.
// It affects Pods running on Linux Nodes only.

With regards to the description of the "disableTXChecksumOffload" parameter in the antrea-agent,I'm not clear about the last part of this sentence. What causes packets to be dropped due to bad checksum? Disabling TX checksum offloading?

Thanks!

[antoninbas]

@Edward033 Setting disableTXChecksumOffload to true will disable checksum offloading for the eth0 interface of each Pod (it doesn't affect configuration of your NIC). So when the Pod sends TCP traffic, instead of checksum computation being deferred "for as long as possible" to promote segmentation and hardware acceleration, the checksum will be computed by the kernel before sending packets out on the veth. This is likely to have an impact on overall throughput, but there is no impact on functionality.

What causes packets to be dropped due to bad checksum? Disabling TX checksum offloading?

No, disabling checksum offloading prevents packets from being dropped due to bad checksum.

Me providing more details may just create more confusion but here we go, assuming that checksum offloading is enabled: when packets are redirected to Suricata for IPS, a tap interface is used to send and receive packets to the userspace process (Suricata). When packets are sent from the Antrea OVS datapath to Suricata, the sk_buf data structure is aware that the checksum has not been computed yet (computation is deferred because checksum offloading is supported). However, after we send these packets to userspace and then get them back to reinject them into the Antrea OVS datapath, this information has been lost (obviously the sk_buf structure ceased to exist when we sent packets to userspace). So at that stage we have an invalid / missing checksum (it's never been calculated) and the kernel is also no longer aware that the checksum is missing. Because of the latter, the checksum never ends up being calculated and we send packets with an invalid checksum.
When we disable checksum offloading, we avoid this situation.

[Edward033]

Thanks for the detailed explanation!

it doesn't affect configuration of your NIC

Glad to hear this.

You did a great job explaining the offloading disabled scenario when forwarding traffic to Suricata. I'm now curious what happens if checksum offloading is indeed enabled in Antrea and Suricata is not in use, and why you think this is likely to impact throughput.

So when the Pod sends TCP traffic, instead of checksum computation being deferred "for as long as possible" to promote segmentation and hardware acceleration,

What could be performing this hardware acceleration? Isn't everything software based at this point?

I'm just not seeing the impact of computing the checksum sooner or the benefits of deferring checksum computation to a later stage unless you have something performing hardware acceleration. I understand that you need to perform checksum offloading when Suricata is used for the reasons you stated, but I feel I don't know enough about the other scenario (offloading enabled) to fully appreciate why you think this impacts throughput.

Thanks again for the help!

[antoninbas]

I'm now curious what happens if checksum offloading is indeed enabled in Antrea and Suricata is not in use, and why you think this is likely to impact throughput.

• computing checksums in hardware is faster than in software
• TCP segmentation offload (TSO) is unlikely to be possible when disabling TX checksum offload, and that can be a key acceleration feature

Below you will see the impact of setting disableTXChecksumOffload to true on offload configuration for the Pod veth interface (eth0). As you can see, it automatically disables TSO.

ethtool output - default

abasSMD6R:antrea abas$ kubectl exec -ti antrea-toolbox-fhhww -- ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: on
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: on
	tx-tcp-mangleid-segmentation: on
	tx-tcp6-segmentation: on

ethtool output - checksum offload disabled

abasSMD6R:antrea abas$ kubectl exec -ti antrea-toolbox-b5xmj -- ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: off
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: off
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: on
tcp-segmentation-offload: off
	tx-tcp-segmentation: off [requested on]
	tx-tcp-ecn-segmentation: off [requested on]
	tx-tcp-mangleid-segmentation: off [requested on]
	tx-tcp6-segmentation: off [requested on]
generic-segmentation-offload: on

If you want to confirm the impact on throughput, the easiest thing to would be to run iperf benchmarks for intra-Node and inter-Node traffic, in both configurations.

I did a quick experiment for inter-Node traffic, but it was a Kind cluster so it's not really representative of what would happen for a bare-metal cluster or a vSphere-based cluster:

• default configuration: 3300Mbps
• checksum offload disabled, as required for L7NetworkPolicy: 500Mbps

So about a 6-7x decrease in throughput (again, it would be better to try on your actual setup).

I understand that you need to perform checksum offloading when Suricata is used for the reasons you stated

Just wanted to emphasize that this is only needed when traffic is redirected, not when traffic is mirrored. In Suricata terms, this is only used for IPS (Intrusion prevention), not for IDS (Intrusion detection). This is why I suggested doing your own mirroring to Suricata (while L7NetworkPolicy requires redirection, as it is an "IDS feature").

[Edward033]

Appreciate the information!

On my current setup, iperf3 reaches ~ 3.7Gbits/s (3700Mbps) with default configuration. With checksum offload disabled I'm getting ~ 1.09Gbits/s so a 3x decrease approx. for me.

I've only tested external traffic. Traffic from a pod to an external server, but will perform intra-node and inter-node tests as well.

Thanks!

[tnqn]

It seems that the only effect of setting enableLogging: true is that we get a second event with type "packet". The documentation doesn't say anything about this. We will have to check with @tnqn to see if this is the intended behavior. He is on vacation right now but will be back next week.

It shouldn't be the intended behavior, otherwise there would be no point to set msg for the allow rule. @qiyueyao are you aware of the behavior? As it's possible to get all HTTP events for L7Exporter, I think there is no problem for L7NetworkPolicy too. Maybe we should add the following to the configuration?

types:
- http:
    extended: yes