pfsense - pfblockerng logs to pfelk - working solution #486
Replies: 7 comments 2 replies
-
I forgot to mention: And please, before you change any pfelk settings and files, make a backup copy of the original working state so you can revert back to it in case you miss a { or ( or any other typo. Believe me, I have learned that the hard way. |
Beta Was this translation helpful? Give feedback.
-
To Implement: One thing that https://github.com/VictorRobellini/pfSense-Dashboard has, that I absolutely also want in my kibana dashboard: |
Beta Was this translation helpful? Give feedback.
-
updated the GROK pattern to accommodate the \n at the end of every message and changed the second timestamp, which does not match any other available pattern. This because the first item after the timestamp appears to be microseconds but is in fact the first value of the log, the event.id. |
Beta Was this translation helpful? Give feedback.
-
While tinkering with the php script originally designed by VictorRobellini, I saw that the GROK patterns above needed tweaking. I have updated the post to be the patterns as they run now. |
Beta Was this translation helpful? Give feedback.
-
I have created a dashboard for the pfBlockerNG streams ipblock and dnsbl. |
Beta Was this translation helpful? Give feedback.
-
@robeweber
Thanks again for enhancing this project!!! |
Beta Was this translation helpful? Give feedback.
-
Hi,
First let me tell you how happy I am to have found this repo and product.
Nice work and so detailed. I am impressed with your tutorials and guides. Realy great work and I know how much work that is.
I have for a long time wanted to get logs from pfblockerng into a dashboard. I have used pfsense dashboard project from Victor Robellini for a while now. He is using telegraf to write log entries to influx and then showing them in a rather huge grafana dashboard.
His solution does not work for the IP Block logs for a few years now. So I had to look further.
I found a discussion thread that is now closed which talks about integrating pfblockerng. The discussion stops with "started a new project". However I can not find this project. So I thought, I would dive into it and share back here. Not sure if this is the right discussion category. Let me know and I move it to wherever it is needed.
I had success with my dev skills and wanted to share this for anyone still looking for a solution.
Yesterday, I have started from scratch. Installing an Ubuntu 22.04.2 LTS and running the install script from your git.
My pfSense version is 2.7.0-RELEASE (amd64)
pfSense running on Protecli.com vp2410
I have the syslog-ng plugin and made two new Source/Log/Destination entries for the two log files I was interested in and sending them to two different ports: 5055 and 5056
for the ip_block.log:
![image](https://private-user-images.githubusercontent.com/23274427/250343580-3b5990d0-85ff-47fa-b4d0-be2c970de1f3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNTE0OTEsIm5iZiI6MTcyNjE1MTE5MSwicGF0aCI6Ii8yMzI3NDQyNy8yNTAzNDM1ODAtM2I1OTkwZDAtODVmZi00N2ZhLWI0ZDAtYmUyYzk3MGRlMWYzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDE0MjYzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTM0YTQwZmU4NDI4NzI3OGUyMGFiM2U1NGMyMjA4ZDQ3ZGExNjFjODVhYTA1OWJiNGQ1YWE2MzcxNDM4ZjI2MDgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.T467el-u8-SE62zaCyhI7Lznu7-Ru4PPAfvMNO0NTcQ)
and for the dnsbl.log:
![image](https://private-user-images.githubusercontent.com/23274427/250343659-8461d54f-b265-4038-9596-8eefe1580ca4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNTE0OTEsIm5iZiI6MTcyNjE1MTE5MSwicGF0aCI6Ii8yMzI3NDQyNy8yNTAzNDM2NTktODQ2MWQ1NGYtYjI2NS00MDM4LTk1OTYtOGVlZmUxNTgwY2E0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDE0MjYzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRhMzIzYjg4ZDI0YmM5M2U0OWRlM2RiYjBkYjlmNGZlYzMxZTFlMGE4MzllZTQ1MzE1MDYzODAyMDRkMjc0YmQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.IZq3VlYIXzi7d_6WI3fCXvqMOj88UW9hsoYvEAlWhis)
The rest happens in the PFELK setup:
On Pfelk I have added a new input pfelk file: 02-input-pfblockerng.pfelk
And in the pattern pfelk.grok I have added the following lines:
in the 05-apps.pfelk configuration, I have added the following two new sections:
I am using the 45-enhanced_private.pfelk config and had to amend line 10 from:
if "dhcp" in [tags] or "unbound" in [tags] or "squid" in [tags] {
to:
if "dhcp" in [tags] or "unbound" in [tags] or "squid" in [tags] or "pfblockerng" in [tags] {
Finally, in 50-outputs.pfelk I have added two new datastreams just before the "catch all else" else -> blabla "unknown" bucket clause:
This now creates two new data streams in elastic search that you can create data views on and eventually dashboards for visualization.
I thought I would share this here and start a support or thinktank discussion for changes or improvements.
What do you think guys?
I have now uploaded all files as I am using them at this moment.
https://github.com/robeweber/pfelk/tree/main
Only download and overwrite them in your setup if you have not changed anything else yourselves in them and only if you follow all the steps of (#486) and #487
Beta Was this translation helpful? Give feedback.
All reactions