Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[META] Implement ECS-Compatibility Mode in Bundled Plugins #11635

Closed
yaauie opened this issue Feb 26, 2020 · 3 comments
Closed

[META] Implement ECS-Compatibility Mode in Bundled Plugins #11635

yaauie opened this issue Feb 26, 2020 · 3 comments
Labels
ecs elastic-common-schema meta

Comments

@yaauie
Copy link
Member

yaauie commented Feb 26, 2020 •
edited by roaksoax
Loading

Plugins that are bundled with Logstash need to be able to run in an ECS-Compatible mode by default in an upcoming major release of Logstash, unless explicitly configured to do otherwise.

To determine the scope of changes needed to implement an ECS-Compatibility mode, a
Logstash plugin will need to be categorized based on its implicit behavour to determine
the effort to bring their defaults into ECS compliance:

  • Exclusively uses fields that align with ECS (no ECS-Compatibility mode needed)
  • Uses one or more fields that conflict with ECS (ECS-Compatibility mode required)
  • Uses one or more fields that are undefined in the latest ECS and therefore at risk of future conflict (ECS-Compatibility mode recommended, aiming to minimize this risk)

-- #11623

Below is our list of bundled plugins. The list will be kept up-to-date with links issues on the individual projects, along with one of the three categories (align, conflict, undefined).

Input Plugins

Input Plugins are generally reliant on Codec Plugins to produce their event
structure, but many add metadata related to the source of the input (e.g.,
sender host info for inbound connections, file paths for discovered files, etc.),
and some include embedded Filter Plugins that must separately implement
ECS-Compatibility.

Codec Plugins

Codec Plugins produce Events from a sequence of Bytes, and are responsible for
the basic structure of those Events, and while some implicitly take the produced
structure verbatim from the deserialized input (e.g., JSON*), others decode the
bytes they are given into their own structure (e.g., CEF).

Filter Plugins

Filter plugins manipulate Events by reading and writing from their fields. To be
ECS-Compatible, a Filter must not read from or write to ECS-conflicting fields
unless the field name is explicitly given in the plugin's configuration.

Output Plugins

Output Plugins do not generally manipulate event structure, but are included in
this list for completeness as some may benefit from an ECS-Compatibility
mode (e.g. Elasticsearch's template management).

Output Plugins

Follow-up Work
@yaauie yaauie mentioned this issue Feb 26, 2020
Closed
This was referenced Mar 12, 2020
Closed
Closed
@roaksoax roaksoax added the meta label Apr 7, 2020
@inqueue
Copy link
Member

inqueue commented Apr 13, 2020

Are there any plans to add Zeek/Suricata ECS support in Logstash?

@roaksoax
Copy link
Contributor

Related issue: #11306

@roaksoax roaksoax pinned this issue Apr 17, 2020
@yaauie
Copy link
Member Author

yaauie commented Jul 7, 2020

Are there any plans to add Zeek/Suricata ECS support in Logstash?

No specific effort that I'm aware of, as both the Zeek and Suricata modules referenced are implemented in Beats (not Logstash).

yaauie added a commit to logstash-plugins/logstash-output-elasticsearch that referenced this issue Jul 14, 2020
@yaauie
Implement ECS Compatibility Mode (#952)
e7116f4 
Adds support for ECS-compatibility mode to allow users to opt into the use of ECS-compatible templates.

Part of the effort to make implicit behaviour more ECS-friendly in future releases of Logstash 

Related: elastic/logstash#11635
Resolves: #924
@merlixelastic merlixelastic mentioned this issue Jul 22, 2020
Open
@acchen97 acchen97 mentioned this issue Aug 14, 2020
Closed
@yaauie yaauie mentioned this issue Oct 6, 2020
Merged
5 tasks
@yaauie yaauie mentioned this issue Dec 2, 2020
Closed
@kares kares mentioned this issue Jun 22, 2021
Closed
11 tasks
@kares kares mentioned this issue Jul 29, 2021
Open
8 tasks
@roaksoax roaksoax added the ecs label Aug 18, 2021
@roaksoax roaksoax added meta and removed meta labels Sep 1, 2021
@yaauie yaauie unpinned this issue Sep 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ecs elastic-common-schema meta
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kares @yaauie @roaksoax @inqueue