Adopt parsers in all Filebeat inputs

[kvch]

Goal

Parsers are the successors of readers of the log input. They provide a more flexible message parsing. At the moment they are only available in the filestream input. But the functionality could be useful in other inputs.

We have to inspect all available inputs in Filebeat to see if it makes sense to add parsers. If the answer is yes, then the feature should be added.

Parsers

• ndjson
• multiline
• container Add new parser to filestream input: container #26115
• syslog [libbeat] Add Syslog parser and processor #30541

How to add parsers to an input

1. Add parsers code to the input

The parser.Config object has its own Unpack function, so it is enough for you to add it as an attribute to your configuration.

type myInputConfiguration struct {
   // other attributes
    parsers parser.Config `config:",inline"`
}

Then create the parser based on the configuration. The Create function expects a reader.Reader as a parameter it can read from.

var reader reader.Reader
p = inp.config.parsers.Create(reader)

Example configuration accepted by the code above

parsers:
  - multiline:
      type: count
      lines_count: 3

2. Add parsers to the reference configuration

Add the reference configuration to either

• filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl
• x-pack/filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl

3. Add parsers to the documentation of the input

There is no common parsers section at the moment because it might not make sense to document all parsers for every input.

Inputs

• AWS CloudWatch Add json.* config support in the aws-cloudwatch input #26429
• AWS S3 [Filebeat] change multiline configuration in awss3 input to parsers #25873
• Azure Event Hub
• Cloud Foundry
  - [ ] Container must use filestream input in the future
  - [ ] Docker must use filestream input in the future
• filestream
• GCP Pub/Sub
• HTTP Endpoint
• HTTP JSON
• journald
• Kafka [Filebeat] kafka v2 using parsers #27335
  - [ ] Log must use filestream input in the future
• MQTT
• NetFlow
• Office 365 Management Activity API
• Redis
  - [ ] Stdin will be deprecated
  - [ ] Syslog syslog will be a parser
• TCP input with parsers #31023
• UDP input with parsers #31024
• Unix Add parsers to UNIX input #27858

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[ascendant512]

May I vote for multiline in the unix input? It is the closest thing Filebeat has to a fifo. Immensely useful for injecting arbitrary logs with netcat without messing with log directories or restarting Filebeat.

[jlind23]

@kvch @nimarezainia what are the missing ones on which we should work?
I'll plan them as soon as possible like in 8.2 to clean it up.

[kvch]

Ideally, all of them. But TCP, UDP, UNIX should be done by our team. For the other inputs, I have to track down others to see if it makes sense to add parsers, and bug them to add if necessary.

[jlind23]

@kvch @cmacknz will it be reworked with the input manager changes ongoing? If yes, I would rather postponing it for now. Thoughts?

[jlind23]

Those are the missing inputs where there is no parsers so far.

• AWS CloudWatch Add json.* config support in the aws-cloudwatch input #26429
• Azure Event Hub
• Cloud Foundry
• GCP Pub/Sub

@masci those ones seems to belong to the @elastic/obs-cloud-monitoring team. Could you please create the appropriate issues if it's relevant?

• HTTP Endpoint
• HTTP JSON
• Office 365 Management Activity API

@elastic/security-external-integrations those ones seems to belong to your area, could you please create the appropriate issues and check them if it is not needed.

• MQTT
• NetFlow
• Redis

@rameshelastic those ones belongs to the @elastic/obs-service-integrations team, could you take them on your behalf.

• TCP TCP input with parsers #31023
• UDP UDP input with parsers #31024
• Unix Add parsers to UNIX input #27858

@cmacknz those ones belong to us, let's try to make it happened.

[nimarezainia]

@kvch @jlind23 sorry catching this thread a bit late. Isn't filestream and enhancement and ultimately replacing log input? should we be making these changes if they are already available in filestream?

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!