[Filebeat] Ensure Kibana audit `event.category` and `event.type` are still processed as strings. #25101

lukeelmers · 2021-04-15T04:34:14Z

This is my first PR to the Beats repo, please don't hesitate to let me know if there's anything important I'm missing ❤️

What does this PR do?

These changes affect the filebeat kibana module. In elastic/kibana#96350, we adjusted the event.type and event.category fields in kibana's audit logs to be arrays instead of strings. To preserve backwards compatibility, this updates the audit ingest pipeline to pull the first value out of the array and send it as a string (there will only be one value in these arrays).

Note: This should be merged in the same release as elastic/kibana#96350 (targeting 7.13)

Why is it important?

Kibana's core team is working to ensure that our new logging system adheres to ECS, and these were two existing fields we identified that should be changed.

Checklist

My code follows the style guidelines of this project
~~[ ] I have commented my code, particularly in hard-to-understand areas~~
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

cc @legrego @thomheymann

elasticmachine · 2021-04-15T05:27:51Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25101 updated
Start Time: 2021-04-15T17:06:51.818+0000
Duration: 54 min 33 sec
Commit: 84e3f9b

Test stats 🧪

Test	Results
Failed	0
Passed	13532
Skipped	2285
Total	15817

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	13532
Skipped	2285
Total	15817

filebeat/module/kibana/audit/ingest/pipeline-json.yml

filebeat/module/kibana/audit/test/test-audit-711.log

elasticmachine · 2021-04-15T17:07:29Z

Pinging @elastic/integrations (Team:Integrations)

legrego

Thanks for addressing my earlier comments - I think this does exactly what we need it to do. Code changes LGTM, but I'm on PTO from now until after feature freeze, so I won't have a chance to pull this down to verify. It'd be great if @thomheymann or someone else from @elastic/kibana-security can do the official review/approval in my absence.

sayden

Looks good from the Beats side. I'm almost sure that it won't break anything in Logs UI. I'm asking just in case but I think we are good to go

ycombinator

LGTM.

thomheymann

Tested locally and verified that audit logs with new format get ingested as expected.

LGTM.

lukeelmers · 2021-04-20T21:09:19Z

Thanks all!

I'm almost sure that it won't break anything in Logs UI. I'm asking just in case but I think we are good to go

Can confirm that this preserves the same formatting in both Discover and Logs UI. Here's an example log entry from both where the array ['web'] is correctly converted to a string:

logs ui

discover

…still processed as strings. (#25101) (cherry picked from commit 5203596)

…still processed as strings. (elastic#25101)

…still processed as strings. (#25101) (#25185)

…-github-pr-comment-template * upstream/master: Check native environment before starting (elastic#25186) Change event.code and winlog.event_id type (elastic#25176) [Ingest Manager] Proxy processes/elastic-agent to stats (elastic#25193) Update mergify backporting to 7.x and 7.13 (elastic#25196) [Heartbeat]: ensure synthetics version co* [Heartbeat]: ensure synthetics version compatability for suites * address review and fix notice * fix lowercase struct * fix version conflict and rebase * update go.* stuff to master * fix notice.txt * move validate inside sourcempatability for suites (elastic#24777) [Filebeat] Ensure Kibana audit `event.category` and `event.type` are still processed as strings. (elastic#25101) Update replace.asciidoc (elastic#25055) Fix nil panic when overwriting metadata (elastic#24741) [Filebeat] Add Malware Bazaar to Threat Intel Module (elastic#24570) Fix k8s svc selectors mapping (elastic#25169) [Ingest Manager] Make agent retry values for bootstraping configurable (elastic#25163) [Metricbeat] Remove elasticsearc.index.created from the SM code (elastic#25113)

…ng-versions-stack * upstream/master: (28 commits) Add support for parsers in filestream input (elastic#24763) Skip flaky test TestFilestreamTruncate (elastic#25218) backport: Add 7.13 branch (elastic#25189) Update decode_json_fields.asciidoc (elastic#25056) [Elastic Agent] Fix status and inspect command to work inside running container (elastic#25204) Check native environment before starting (elastic#25186) Change event.code and winlog.event_id type (elastic#25176) [Ingest Manager] Proxy processes/elastic-agent to stats (elastic#25193) Update mergify backporting to 7.x and 7.13 (elastic#25196) [Heartbeat]: ensure synthetics version co* [Heartbeat]: ensure synthetics version compatability for suites * address review and fix notice * fix lowercase struct * fix version conflict and rebase * update go.* stuff to master * fix notice.txt * move validate inside sourcempatability for suites (elastic#24777) [Filebeat] Ensure Kibana audit `event.category` and `event.type` are still processed as strings. (elastic#25101) Update replace.asciidoc (elastic#25055) Fix nil panic when overwriting metadata (elastic#24741) [Filebeat] Add Malware Bazaar to Threat Intel Module (elastic#24570) Fix k8s svc selectors mapping (elastic#25169) [Ingest Manager] Make agent retry values for bootstraping configurable (elastic#25163) [Metricbeat] Remove elasticsearc.index.created from the SM code (elastic#25113) [Ingest Manager] Keep http and logging config during enroll (elastic#25132) Refactor kubernetes autodiscover to avoid skipping short-living pods (elastic#24742) [libbeat] New decode xml wineventlog processor (elastic#25115) ...

Ensure event.category and event.type are still processed as strings.

8e8a59d

lukeelmers added enhancement backport-v7.13.0 Automated backport with mergify labels Apr 15, 2021

lukeelmers self-assigned this Apr 15, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 15, 2021

lukeelmers added Filebeat Filebeat Team:Integrations Label for the Integrations team labels Apr 15, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 15, 2021

lukeelmers mentioned this pull request Apr 15, 2021

[core.logging] Ensure LogMeta is ECS-compliant. elastic/kibana#96350

Merged

thomheymann reviewed Apr 15, 2021

View reviewed changes

filebeat/module/kibana/audit/ingest/pipeline-json.yml Show resolved Hide resolved

legrego reviewed Apr 15, 2021

View reviewed changes

filebeat/module/kibana/audit/test/test-audit-711.log Outdated Show resolved Hide resolved

Check that category and type are a list & update tests.

84e3f9b

lukeelmers marked this pull request as ready for review April 15, 2021 17:07

lukeelmers requested review from thomheymann, legrego, sayden and ycombinator April 15, 2021 19:32

legrego reviewed Apr 15, 2021

View reviewed changes

sayden approved these changes Apr 19, 2021

View reviewed changes

ycombinator approved these changes Apr 19, 2021

View reviewed changes

thomheymann approved these changes Apr 20, 2021

View reviewed changes

lukeelmers merged commit 5203596 into elastic:master Apr 20, 2021

lukeelmers deleted the filebeat-kibana-audit-log branch April 20, 2021 21:09

lukeelmers added v7.13.0 v8.0.0 and removed backport-v7.13.0 Automated backport with mergify labels Apr 20, 2021

mergify bot pushed a commit that referenced this pull request Apr 20, 2021

[Filebeat] Ensure Kibana audit event.category and event.type are …

f359e94

…still processed as strings. (#25101) (cherry picked from commit 5203596)

mergify bot mentioned this pull request Apr 20, 2021

[Filebeat] Ensure Kibana audit event.category and event.type are still processed as strings. (backport #25101) #25184

Closed

lukeelmers added a commit to lukeelmers/beats that referenced this pull request Apr 20, 2021

[Filebeat] Ensure Kibana audit event.category and event.type are …

1e5e202

…still processed as strings. (elastic#25101)

lukeelmers mentioned this pull request Apr 20, 2021

Cherry-pick to 7.x: [Filebeat] Ensure Kibana audit event.category and event.type are still processed as strings. (#25101) #25185

Merged

lukeelmers added a commit that referenced this pull request Apr 20, 2021

[Filebeat] Ensure Kibana audit event.category and event.type are …

c52c439

…still processed as strings. (#25101) (#25185)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Ensure Kibana audit `event.category` and `event.type` are still processed as strings. #25101

[Filebeat] Ensure Kibana audit `event.category` and `event.type` are still processed as strings. #25101

lukeelmers commented Apr 15, 2021 •

edited

Loading

elasticmachine commented Apr 15, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

elasticmachine commented Apr 15, 2021

legrego left a comment

sayden left a comment •

edited

Loading

ycombinator left a comment

thomheymann left a comment

lukeelmers commented Apr 20, 2021

[Filebeat] Ensure Kibana audit event.category and event.type are still processed as strings. #25101

[Filebeat] Ensure Kibana audit event.category and event.type are still processed as strings. #25101

Conversation

lukeelmers commented Apr 15, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Apr 15, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Apr 15, 2021

legrego left a comment

Choose a reason for hiding this comment

sayden left a comment • edited Loading

Choose a reason for hiding this comment

ycombinator left a comment

Choose a reason for hiding this comment

thomheymann left a comment

Choose a reason for hiding this comment

lukeelmers commented Apr 20, 2021

[Filebeat] Ensure Kibana audit `event.category` and `event.type` are still processed as strings. #25101

[Filebeat] Ensure Kibana audit `event.category` and `event.type` are still processed as strings. #25101

lukeelmers commented Apr 15, 2021 •

edited

Loading

elasticmachine commented Apr 15, 2021 •

edited by jenkins-beats-ci bot

Loading

sayden left a comment •

edited

Loading