diff --git a/CHANGELOG.next.md b/CHANGELOG.next.md index c9940f6059..455fc57bbc 100644 --- a/CHANGELOG.next.md +++ b/CHANGELOG.next.md @@ -45,6 +45,7 @@ Thanks, you're awesome :-) --> #### Improvements * Remove remaining Go deps after removing Go code generator. #1585 +* Add explicit `default_field: true` for Beats artifacts. #1633 #### Deprecated diff --git a/experimental/generated/beats/fields.ecs.yml b/experimental/generated/beats/fields.ecs.yml index 85053bd406..23bbe58dcd 100644 --- a/experimental/generated/beats/fields.ecs.yml +++ b/experimental/generated/beats/fields.ecs.yml @@ -63,6 +63,7 @@ not change if data is sent through queuing systems like Kafka, Redis, or processing systems such as Logstash or APM Server.' type: group + default_field: true fields: - name: build.original level: core @@ -127,6 +128,7 @@ behalf of a single administrative entity or domain that presents a common, clearly defined routing policy to the internet. type: group + default_field: true fields: - name: number level: extended @@ -163,6 +165,7 @@ in that category, you should still ensure that source and destination are filled appropriately.' type: group + default_field: true fields: - name: address level: extended @@ -438,6 +441,7 @@ in the cloud, the field contains cloud data from the machine the service is running on.' type: group + default_field: true fields: - name: account.id level: extended @@ -528,6 +532,7 @@ group: 2 description: These fields contain information about binary code signatures. type: group + default_field: true fields: - name: digest_algorithm level: extended @@ -615,6 +620,7 @@ These fields help correlate data based containers from any runtime.' type: group + default_field: true fields: - name: cpu.usage level: extended @@ -703,6 +709,7 @@ names cannot include `\`, `/`, `*`, `?`, `"`, `<`, `>`, `|`, ` ` (space character), `,`, or `#`. Please see the Elasticsearch reference for additional https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params[restrictions].' type: group + default_field: true fields: - name: dataset level: extended @@ -750,6 +757,7 @@ transaction. If the event also contains identification of the client and server roles, then the client and server fields should also be populated.' type: group + default_field: true fields: - name: address level: extended @@ -1029,6 +1037,7 @@ * Dynamic library (`.dylib`) commonly used on macOS' type: group + default_field: true fields: - name: code_signature.digest_algorithm level: extended @@ -1427,6 +1436,7 @@ query details as well as all of the answers that were provided for this query (`dns.type:answer`).' type: group + default_field: true fields: - name: answers level: extended @@ -1591,6 +1601,7 @@ group: 2 description: Meta-information specific to ECS. type: group + default_field: true fields: - name: version level: core @@ -1609,6 +1620,7 @@ group: 2 description: These fields contain Linux Executable Linkable Format (ELF) metadata. type: group + default_field: true fields: - name: architecture level: extended @@ -1798,6 +1810,7 @@ protocols that send and receive email messages such as SMTP are outside the scope of the `email.*` fields.' type: group + default_field: true fields: - name: attachments level: extended @@ -1967,6 +1980,7 @@ Use them for errors that happen while fetching events or in cases where the event itself contains an error.' type: group + default_field: true fields: - name: code level: core @@ -2012,6 +2026,7 @@ temperature. See the `event.kind` definition in this section for additional details about metric and state events.' type: group + default_field: true fields: - name: action level: core @@ -2334,6 +2349,7 @@ services). File fields provide details about the affected file associated with the event or metric.' type: group + default_field: true fields: - name: accessed level: extended @@ -3231,6 +3247,7 @@ This geolocation information can be derived from techniques such as Geo IP, or be user-supplied.' type: group + default_field: true fields: - name: city_name level: core @@ -3315,6 +3332,7 @@ description: The group fields are meant to represent groups that are relevant to the event. type: group + default_field: true fields: - name: domain level: extended @@ -3347,6 +3365,7 @@ a range of generic bytes. Entity-specific hashes such as ja3 or imphash are placed in the fieldsets to which they relate (tls and pe, respectively).' type: group + default_field: true fields: - name: md5 level: extended @@ -3383,6 +3402,7 @@ event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' type: group + default_field: true fields: - name: architecture level: core @@ -3645,6 +3665,7 @@ description: Fields related to HTTP activity. Use the `url` field set to store the url of the request. type: group + default_field: true fields: - name: request.body.bytes level: extended @@ -3759,6 +3780,7 @@ a single observer interface (e.g. network sensor on a span port) only the observer.ingress information should be populated. type: group + default_field: true fields: - name: alias level: extended @@ -3795,6 +3817,7 @@ The details specific to your event source are typically not logged under `log.*`, but rather in `event.*` or in other ECS fields.' type: group + default_field: true fields: - name: file.path level: extended @@ -3906,6 +3929,7 @@ The network.* fields should be populated with details about the network activity associated with an event.' type: group + default_field: true fields: - name: application level: extended @@ -4065,6 +4089,7 @@ or metric. Message queues and ETL components used in processing events or metrics are not considered observers in ECS.' type: group + default_field: true fields: - name: egress level: extended @@ -4378,6 +4403,7 @@ description: Fields that describe the resources which container orchestrators manage or act upon. type: group + default_field: true fields: - name: api_version level: extended @@ -4449,6 +4475,7 @@ These fields help you arrange or filter data stored in an index by one or multiple organizations.' type: group + default_field: true fields: - name: id level: extended @@ -4469,6 +4496,7 @@ group: 2 description: The OS fields contain information about the operating system. type: group + default_field: true fields: - name: family level: extended @@ -4536,6 +4564,7 @@ It contains general information about a package, such as name, version or size. It also contains installation details, such as time or location. type: group + default_field: true fields: - name: architecture level: extended @@ -4632,6 +4661,7 @@ group: 2 description: These fields contain Windows Portable Executable (PE) metadata. type: group + default_field: true fields: - name: architecture level: extended @@ -4905,6 +4935,7 @@ from a log message. The `process.pid` often stays in the metric itself and is copied to the global field for correlation.' type: group + default_field: true fields: - name: args level: extended @@ -7680,6 +7711,7 @@ group: 2 description: Fields related to Windows Registry operations. type: group + default_field: true fields: - name: data.bytes level: extended @@ -7755,6 +7787,7 @@ to `related.ip`, you can then search for a given IP trivially, no matter where it appeared, by querying `related.ip:192.0.2.15`.' type: group + default_field: true fields: - name: hash level: extended @@ -7792,6 +7825,7 @@ application firewalls, url filters, endpoint detection and response (EDR) systems, etc.' type: group + default_field: true fields: - name: author level: extended @@ -7894,6 +7928,7 @@ in that category, you should still ensure that source and destination are filled appropriately.' type: group + default_field: true fields: - name: address level: extended @@ -8166,6 +8201,7 @@ These fields help you find and correlate logs for a specific service and version.' type: group + default_field: true fields: - name: address level: extended @@ -8279,6 +8315,7 @@ transaction. If the event also contains identification of the client and server roles, then the client and server fields should also be populated.' type: group + default_field: true fields: - name: address level: extended @@ -8554,6 +8591,7 @@ \ which kind of approach is used by this detected threat, to accomplish the\ \ goal (e.g. \"endpoint denial of service\")." type: group + default_field: true fields: - name: enrichments level: extended @@ -11774,6 +11812,7 @@ protocol itself and intentionally avoids in-depth analysis of the related x.509 certificate files. type: group + default_field: true fields: - name: cipher level: extended @@ -12384,6 +12423,7 @@ description: URL fields provide support for complete or partial URLs, and supports the breaking down into scheme, domain, path, and so on. type: group + default_field: true fields: - name: domain level: extended @@ -12531,6 +12571,7 @@ Fields can have one entry or multiple entries. If a user has more than one id, provide an array that includes all of them.' type: group + default_field: true fields: - name: changes.domain level: extended @@ -12844,6 +12885,7 @@ They often show up in web service logs coming from the parsed user agent string.' type: group + default_field: true fields: - name: device.name level: extended @@ -12952,6 +12994,7 @@ specific information when observer events contain discrete ingress and egress VLAN information, typically provided by firewalls, routers, or load balancers.' type: group + default_field: true fields: - name: id level: extended @@ -12973,6 +13016,7 @@ description: The vulnerability fields describe information about a vulnerability that is relevant to an event. type: group + default_field: true fields: - name: category level: extended @@ -13111,6 +13155,7 @@ use the x509 fields under the relevant TLS fields: `tls.server.x509` and/or `tls.client.x509`.' type: group + default_field: true fields: - name: alternative_names level: extended diff --git a/generated/beats/fields.ecs.yml b/generated/beats/fields.ecs.yml index c9fc9c012a..e2bdfbe91f 100644 --- a/generated/beats/fields.ecs.yml +++ b/generated/beats/fields.ecs.yml @@ -63,6 +63,7 @@ not change if data is sent through queuing systems like Kafka, Redis, or processing systems such as Logstash or APM Server.' type: group + default_field: true fields: - name: build.original level: core @@ -127,6 +128,7 @@ behalf of a single administrative entity or domain that presents a common, clearly defined routing policy to the internet. type: group + default_field: true fields: - name: number level: extended @@ -163,6 +165,7 @@ in that category, you should still ensure that source and destination are filled appropriately.' type: group + default_field: true fields: - name: address level: extended @@ -438,6 +441,7 @@ in the cloud, the field contains cloud data from the machine the service is running on.' type: group + default_field: true fields: - name: account.id level: extended @@ -528,6 +532,7 @@ group: 2 description: These fields contain information about binary code signatures. type: group + default_field: true fields: - name: digest_algorithm level: extended @@ -615,6 +620,7 @@ These fields help correlate data based containers from any runtime.' type: group + default_field: true fields: - name: id level: core @@ -665,6 +671,7 @@ names cannot include `\`, `/`, `*`, `?`, `"`, `<`, `>`, `|`, ` ` (space character), `,`, or `#`. Please see the Elasticsearch reference for additional https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params[restrictions].' type: group + default_field: true fields: - name: dataset level: extended @@ -712,6 +719,7 @@ transaction. If the event also contains identification of the client and server roles, then the client and server fields should also be populated.' type: group + default_field: true fields: - name: address level: extended @@ -991,6 +999,7 @@ * Dynamic library (`.dylib`) commonly used on macOS' type: group + default_field: true fields: - name: code_signature.digest_algorithm level: extended @@ -1179,6 +1188,7 @@ query details as well as all of the answers that were provided for this query (`dns.type:answer`).' type: group + default_field: true fields: - name: answers level: extended @@ -1343,6 +1353,7 @@ group: 2 description: Meta-information specific to ECS. type: group + default_field: true fields: - name: version level: core @@ -1361,6 +1372,7 @@ group: 2 description: These fields contain Linux Executable Linkable Format (ELF) metadata. type: group + default_field: true fields: - name: architecture level: extended @@ -1549,6 +1561,7 @@ Use them for errors that happen while fetching events or in cases where the event itself contains an error.' type: group + default_field: true fields: - name: code level: core @@ -1594,6 +1607,7 @@ temperature. See the `event.kind` definition in this section for additional details about metric and state events.' type: group + default_field: true fields: - name: action level: core @@ -1916,6 +1930,7 @@ services). File fields provide details about the affected file associated with the event or metric.' type: group + default_field: true fields: - name: accessed level: extended @@ -2603,6 +2618,7 @@ This geolocation information can be derived from techniques such as Geo IP, or be user-supplied.' type: group + default_field: true fields: - name: city_name level: core @@ -2687,6 +2703,7 @@ description: The group fields are meant to represent groups that are relevant to the event. type: group + default_field: true fields: - name: domain level: extended @@ -2719,6 +2736,7 @@ a range of generic bytes. Entity-specific hashes such as ja3 or imphash are placed in the fieldsets to which they relate (tls and pe, respectively).' type: group + default_field: true fields: - name: md5 level: extended @@ -2755,6 +2773,7 @@ event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' type: group + default_field: true fields: - name: architecture level: core @@ -3017,6 +3036,7 @@ description: Fields related to HTTP activity. Use the `url` field set to store the url of the request. type: group + default_field: true fields: - name: request.body.bytes level: extended @@ -3131,6 +3151,7 @@ a single observer interface (e.g. network sensor on a span port) only the observer.ingress information should be populated. type: group + default_field: true fields: - name: alias level: extended @@ -3167,6 +3188,7 @@ The details specific to your event source are typically not logged under `log.*`, but rather in `event.*` or in other ECS fields.' type: group + default_field: true fields: - name: file.path level: extended @@ -3278,6 +3300,7 @@ The network.* fields should be populated with details about the network activity associated with an event.' type: group + default_field: true fields: - name: application level: extended @@ -3437,6 +3460,7 @@ or metric. Message queues and ETL components used in processing events or metrics are not considered observers in ECS.' type: group + default_field: true fields: - name: egress level: extended @@ -3750,6 +3774,7 @@ description: Fields that describe the resources which container orchestrators manage or act upon. type: group + default_field: true fields: - name: api_version level: extended @@ -3821,6 +3846,7 @@ These fields help you arrange or filter data stored in an index by one or multiple organizations.' type: group + default_field: true fields: - name: id level: extended @@ -3841,6 +3867,7 @@ group: 2 description: The OS fields contain information about the operating system. type: group + default_field: true fields: - name: family level: extended @@ -3908,6 +3935,7 @@ It contains general information about a package, such as name, version or size. It also contains installation details, such as time or location. type: group + default_field: true fields: - name: architecture level: extended @@ -4004,6 +4032,7 @@ group: 2 description: These fields contain Windows Portable Executable (PE) metadata. type: group + default_field: true fields: - name: architecture level: extended @@ -4067,6 +4096,7 @@ from a log message. The `process.pid` often stays in the metric itself and is copied to the global field for correlation.' type: group + default_field: true fields: - name: args level: extended @@ -5032,6 +5062,7 @@ group: 2 description: Fields related to Windows Registry operations. type: group + default_field: true fields: - name: data.bytes level: extended @@ -5107,6 +5138,7 @@ to `related.ip`, you can then search for a given IP trivially, no matter where it appeared, by querying `related.ip:192.0.2.15`.' type: group + default_field: true fields: - name: hash level: extended @@ -5144,6 +5176,7 @@ application firewalls, url filters, endpoint detection and response (EDR) systems, etc.' type: group + default_field: true fields: - name: author level: extended @@ -5246,6 +5279,7 @@ in that category, you should still ensure that source and destination are filled appropriately.' type: group + default_field: true fields: - name: address level: extended @@ -5518,6 +5552,7 @@ These fields help you find and correlate logs for a specific service and version.' type: group + default_field: true fields: - name: address level: extended @@ -5631,6 +5666,7 @@ transaction. If the event also contains identification of the client and server roles, then the client and server fields should also be populated.' type: group + default_field: true fields: - name: address level: extended @@ -5906,6 +5942,7 @@ \ which kind of approach is used by this detected threat, to accomplish the\ \ goal (e.g. \"endpoint denial of service\")." type: group + default_field: true fields: - name: enrichments level: extended @@ -8706,6 +8743,7 @@ protocol itself and intentionally avoids in-depth analysis of the related x.509 certificate files. type: group + default_field: true fields: - name: cipher level: extended @@ -9316,6 +9354,7 @@ description: URL fields provide support for complete or partial URLs, and supports the breaking down into scheme, domain, path, and so on. type: group + default_field: true fields: - name: domain level: extended @@ -9463,6 +9502,7 @@ Fields can have one entry or multiple entries. If a user has more than one id, provide an array that includes all of them.' type: group + default_field: true fields: - name: changes.domain level: extended @@ -9776,6 +9816,7 @@ They often show up in web service logs coming from the parsed user agent string.' type: group + default_field: true fields: - name: device.name level: extended @@ -9884,6 +9925,7 @@ specific information when observer events contain discrete ingress and egress VLAN information, typically provided by firewalls, routers, or load balancers.' type: group + default_field: true fields: - name: id level: extended @@ -9905,6 +9947,7 @@ description: The vulnerability fields describe information about a vulnerability that is relevant to an event. type: group + default_field: true fields: - name: category level: extended @@ -10043,6 +10086,7 @@ use the x509 fields under the relevant TLS fields: `tls.server.x509` and/or `tls.client.x509`.' type: group + default_field: true fields: - name: alternative_names level: extended diff --git a/scripts/generators/beats.py b/scripts/generators/beats.py index f36eebad55..adb0bf40b2 100644 --- a/scripts/generators/beats.py +++ b/scripts/generators/beats.py @@ -23,6 +23,8 @@ def generate(ecs_nested, ecs_version, out_dir): continue beats_field = ecs_helpers.dict_copy_keys_ordered(fieldset, allowed_fieldset_keys) + if 'default_field' not in beats_field: + beats_field['default_field'] = True beats_field['fields'] = fieldset_field_array(fieldset['fields'], df_allowlist, fieldset['prefix']) beats_fields.append(beats_field)