[Filebeat] Dissect Parsing Error with Sophos Module

[rdrgporto]

For confirmed bugs, please report:

• Version: 7.9.2
• Operating System: Ubuntu 18.04

Hi, everyone

I have found a dissect parsing error by using Sophos module module of Filebeat 7.9.2.

The Sophos version is SFOS 18.0.4 MR-4 XG210

Here you are a log example:

<01>Feb 11 13:12:45 _gateway device="SFW" date=2021-02-11 time=13:12:45 timezone="CET" device_name="XG210" device_id=<device_id> log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=9 nat_rule_id=16 policy_type=1 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" vlan_id="" ether_type=Unknown (0x0000) bridge_name="" bridge_display_name="" in_interface="Port2.109" in_display_interface="CD21-IPs_WAN" out_interface="Port5.200" out_display_interface="<out_display_interface>" src_mac=<src_mac> dst_mac=<dst_mac> src_ip=<src_ip> src_country_code=ESP dst_ip=<dst-ip> dst_country_code=<dst_country_code> protocol="TCP" src_port=33370 dst_port=443 sent_pkts=0  recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=<trans_src_ip> tran_src_port=0 tran_dst_ip=<trans_dst_ip> tran_dst_port=0 srczonetype="WAN" srczone="WAN" dstzonetype="DMZ" dstzone="<dstzone>" dir_disp="" connevent="Start" connid="3933925696" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0

The dissect error is the following:

error.message: field [log.original] does not contain value_split [=]

KV in Ingest Node Pipeline does not have the same behaviour like Logstash. When it encounters =" does not match = as value separator.

If I add before a gsub, it works. However, I am not aware of this change may affect to Sophos Pipeline in another section of the pipeline.

{
  "gsub" : {
    "field" : "log.original",
    "pattern" : "=\"",
    "replacement" : " = \""
  }
}

Thanks in advance,

Rodrigo

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

I don't think that this is because of separator behaviour.

There are a collection of space-separated tokens in your log line that do not have a field split pattern; 'Feb', '11', '13:12:45', '_gateway' and '(0x0000)'. Looking on the Sophos Community forum the 'ether_type' field always appears to be quoted which solves the last token.

Removing the syslog date and the _gateway token, and quoting the hex value to the 'Unknown' prefix allows the parse to succeed, although the test subsequently fails due to unrecognised fields; 'sophos.xg.in_display_interface', 'sophos.xg.out_display_interface', 'sophos.xg.ether_type', 'sophos.xg.nat_rule_id'. These fields appear in Sophos Community forum posts so probably they should be added.

Altered log line (fictitious values added):

<01>device="SFW" date=2021-02-11 time=13:12:45 timezone="CET" device_name="XG210" device_id=<device_id> log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=9 nat_rule_id=16 policy_type=1 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" vlan_id="" ether_type="Unknown (0x0000)" bridge_name="" bridge_display_name="" in_interface="Port2.109" in_display_interface="CD21-IPs_WAN" out_interface="Port5.200" out_display_interface="<out_display_interface>" src_mac=11:22:33:44:55:66 dst_mac=66:55:44:33:22:11 src_ip=1.2.3.4 src_country_code=ESP dst_ip=4.3.2.1 dst_country_code=GB protocol="TCP" src_port=33370 dst_port=443 sent_pkts=0  recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=2.4.6.8 tran_src_port=0 tran_dst_ip=8.6.4.2 tran_dst_port=0 srczonetype="WAN" srczone="WAN" dstzonetype="DMZ" dstzone="<dstzone>" dir_disp="" connevent="Start" connid="3933925696" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0

Resulting document:

{
    "@timestamp": "2021-02-11T13:12:45.000-02:00",
    "client.bytes": 0,
    "client.ip": "1.2.3.4",
    "client.mac": "11:22:33:44:55:66",
    "client.nat.port": 0,
    "client.packets": 0,
    "client.port": 33370,
    "destination.bytes": 0,
    "destination.ip": "4.3.2.1",
    "destination.mac": "66:55:44:33:22:11",
    "destination.nat.ip": "8.6.4.2",
    "destination.nat.port": 0,
    "destination.packets": 0,
    "destination.port": 443,
    "event.action": "allowed",
    "event.category": [
        "network"
    ],
    "event.code": "010101600001",
    "event.dataset": "sophos.xg",
    "event.duration": 0,
    "event.end": "2021-02-11T13:12:45.000-02:00",
    "event.kind": "event",
    "event.module": "sophos",
    "event.original": "device=\"SFW\" date=2021-02-11 time=13:12:45 timezone=\"CET\" device_name=\"XG210\" device_id=<device_id> log_id=010101600001 log_type=\"Firewall\" log_component=\"Firewall Rule\" log_subtype=\"Allowed\" status=\"Allow\" priority=Information duration=0 fw_rule_id=9 nat_rule_id=16 policy_type=1 user_name=\"\" user_gp=\"\" iap=0 ips_policy_id=0 appfilter_policy_id=0 application=\"\" application_risk=0 application_technology=\"\" application_category=\"\" vlan_id=\"\" ether_type=\"Unknown (0x0000)\" bridge_name=\"\" bridge_display_name=\"\" in_interface=\"Port2.109\" in_display_interface=\"CD21-IPs_WAN\" out_interface=\"Port5.200\" out_display_interface=\"<out_display_interface>\" src_mac=11:22:33:44:55:66 dst_mac=66:55:44:33:22:11 src_ip=1.2.3.4 src_country_code=ESP dst_ip=4.3.2.1 dst_country_code=GB protocol=\"TCP\" src_port=33370 dst_port=443 sent_pkts=0  recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=2.4.6.8 tran_src_port=0 tran_dst_ip=8.6.4.2 tran_dst_port=0 srczonetype=\"WAN\" srczone=\"WAN\" dstzonetype=\"DMZ\" dstzone=\"<dstzone>\" dir_disp=\"\" connevent=\"Start\" connid=\"3933925696\" vconnid=\"\" hb_health=\"No Heartbeat\" message=\"\" appresolvedby=\"Signature\" app_is_cloud=0",
    "event.outcome": "success",
    "event.severity": "6",
    "event.start": "2021-02-11T13:12:45.000-02:00",
    "event.timezone": "-02:00",
    "event.type": [
        "allowed",
        "connection",
        "start"
    ],
    "fileset.name": "xg",
    "host.name": "firewall.localgroup.local",
    "input.type": "log",
    "log.level": "informational",
    "log.offset": 17878,
    "network.bytes": 0,
    "network.direction": "inbound",
    "network.packets": 0,
    "network.transport": "tcp",
    "observer.egress.interface.name": "Port5.200",
    "observer.egress.zone": "DMZ",
    "observer.ingress.interface.name": "Port2.109",
    "observer.ingress.zone": "WAN",
    "observer.product": "XG",
    "observer.serial_number": "<device_id>",
    "observer.type": "firewall",
    "observer.vendor": "Sophos",
    "related.hosts": [
        "firewall.localgroup.local"
    ],
    "related.ip": [
        "1.2.3.4",
        "2.4.6.8",
        "4.3.2.1",
        "8.6.4.2"
    ],
    "rule.id": "9",
    "rule.ruleset": "1",
    "server.bytes": 0,
    "server.ip": "4.3.2.1",
    "server.mac": "66:55:44:33:22:11",
    "server.nat.port": 0,
    "server.packets": 0,
    "server.port": 443,
    "service.type": "sophos",
    "sophos.xg.app_is_cloud": "0",
    "sophos.xg.appfilter_policy_id": "0",
    "sophos.xg.application_risk": "0",
    "sophos.xg.appresolvedby": "Signature",
    "sophos.xg.connevent": "Start",
    "sophos.xg.connid": "3933925696",
    "sophos.xg.device": "SFW",
    "sophos.xg.device_name": "XG210",
    "sophos.xg.dst_country_code": "GB",
    "sophos.xg.ether_type": "Unknown (0x0000)",
    "sophos.xg.hb_health": "No Heartbeat",
    "sophos.xg.iap": "0",
    "sophos.xg.in_display_interface": "CD21-IPs_WAN",
    "sophos.xg.ips_policy_id": "0",
    "sophos.xg.log_component": "Firewall Rule",
    "sophos.xg.log_subtype": "Allowed",
    "sophos.xg.log_type": "Firewall",
    "sophos.xg.message_id": "00001",
    "sophos.xg.nat_rule_id": "16",
    "sophos.xg.out_display_interface": "<out_display_interface>",
    "sophos.xg.priority": "Information",
    "sophos.xg.src_country_code": "ESP",
    "sophos.xg.status": "Allow",
    "source.bytes": 0,
    "source.ip": "1.2.3.4",
    "source.mac": "11:22:33:44:55:66",
    "source.nat.ip": "2.4.6.8",
    "source.nat.port": 0,
    "source.packets": 0,
    "source.port": 33370,
    "tags": [
        "forwarded",
        "sophos-xg"
    ]
}