pfElk-> pfEfk? (or adding support for fluentd)

[justchris1]

Hey all - first this is a really great project. I am thinking about using this on and this project is an amazing start. However, with all the fuss with Elastic and the apparent wars brewing between Elasticsearch and Opensearch (the Amazon fork), there appears to be churn in the broader elk environment. I guess the next shot across the bow from Elastic was to intentionally break logstash so it can't work with the OpenSearch fork (or that is what it appears to me, at least). I was looking to use OpenSearch since it has built-in security and other items that are paywalled away from ES. So with that, I was looking at logstash alternatives, and of course Fluentd pops up. However, it appears that although logstash and fluentd do the same general thing, they don't have compatible parsers (configuration files for the inputs) at all!

If you want to stay focused on logstash, that is totally fine (it is called pfElfk!), but I figured it was worth having the discussion to see if you/the project had considered migrating to fluentd instead of logstash in the future. I mean to disrespect and am not trying to add to your workload if this isn't a direction you want to head in. I just wanted to pulse the thought process of the project and see if it had crossed your collective minds at all.

Thanks again for the contributions in pfElk!

[a3ilson]

pfFO?
I haven't had much time to dedicate to this project this year due to other commitments. However, I'll be freed up in July and will take a look after.

[justchris1]

pfFO?
I haven't had much time to dedicate to this project this year due to other commitments. However, I'll be freed up in July and will take a look after.

This may not be as difficult as I was imagining. There turns out to be a grok plugin for fluentd. I am still sure that there is work, but the grok parsing seems to be one the more significant investments that pfElk brings.

[adn77]

I spent the past week converting the Logstash pipeline to Fluentd.
It actually turned out to be quite a hassle as conditional behavior can only be accomplished by modifying the tag and therefore the routing of a log message.
It's much easier to do lookups in a JSON table than checking different interface names for instance.

I am almost satisfied with the outcome and will create a PR soon.
Besides the excellent Grok patterns with beautifully adhering to ECS naming, I like the dashboards. I ran into some problems there as well since I use the OpenSearch stack which lacks the Lens visualizations. I had to remove those visuals for now.

Is there any place where I get to see what I am missing?

[a3ilson]

@adn77 - can you elaborate on your last question (...any place where I get to see what I am missing?")

[adn77]

Sorry, what I meant was is there any visuals of explaining the dashboards? My Firewall dashboard looks complete compared to the animated GIFs... but I was wondering what those LENS visualizations would look like.

[a3ilson]

@adn77 - can you provide a screenshot of the visualizations (lens) that are not rendering?