[Winlogbeat] Switch to ingest node processing #29435

leehinman · 2021-12-15T04:23:04Z

What does this PR do?

Switches to ingest node processing instead of local javascript
processing for modules.

Why is it important?

For Agent we are using ingest node processing. This keeps the
Winlogbeat and Agent processing in sync and allows bug fixes to be
shared between them.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Should we move the loading of pipelines from filebeat/fileset to libbeat?
Should we move adapting pipelines for compatibility to libbeat?
need to add tests

How to test this PR locally

Test export of pipelines

.\winlogbeat export pipelines --es.version 7.13.1

Test uploading of pipelines with setup

.\winlogbeat setup

Test uploading of pipelines when pipelines don't exist in ES

.\winlogbeat

Related issues

Closes [Winlogbeat] move module processing logic from edge to ingest nodes #29184

mergify · 2021-12-15T04:23:27Z

This pull request does not have a backport label. Could you fix it @leehinman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2021-12-15T05:20:10Z

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2021-12-17T19:09:34Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

taylor-swanson · 2021-12-17T19:53:40Z

winlogbeat/module/pipeline.go

+		}
+		f, err := os.Create(filepath.Join(directory, pipeline.id+".json"))
+		if err != nil {
+			return fmt.Errorf("unable to create file to export pipeline to: %+v", err)


Do we want to use %w here for the error so it gets wrapped? Same applies for other instances below.

good idea, I switched them.

andrewkroh

I like how simple the winlogbeat.yml became. We'll need to do doc updates in a separate PR (that is less time urgent).

libbeat/docs/command-reference.asciidoc

andrewkroh · 2021-12-20T16:53:16Z

winlogbeat/beater/winlogbeat.go


 	"github.com/elastic/beats/v7/winlogbeat/checkpoint"
 	"github.com/elastic/beats/v7/winlogbeat/config"
 	"github.com/elastic/beats/v7/winlogbeat/eventlog"
 )

+const pipelinesWarning = "Winlogbeat is unable to load the ingest pipelines" +
+	" because the Elasticsearch output is not configured/enabled. If you have" +
+	" already loaded the ingest pipelines or are using Logstash pipelines, you" +


or are using Logstash pipelines

This was probably copied from Filebeat. I don't think it should mention Logstash pipelines.

winlogbeat/beater/winlogbeat.go

winlogbeat/module/pipeline.go

x-pack/winlogbeat/_meta/config/output-elasticsearch.yml.tmpl

- remove JS processing - remove testing of JS processing - add ingest node pipelines from agent - use routing pipeline to direct to correct module pipeline - embed pipelines in winlogbeat executable - add `export pipeline` command - add loading of pipelines to setup command - add loading of pipelines to every ES connection if the pipelines don't exist - exports several functions from filebeat/fileset to avoid code duplication - update docs - update default config - update reference config Closes elastic#29184

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

- remove reference to Logstash pipelines - cleanup grammar in godoc - move logger creation

- always honor winlogbeat.overwrite_pipelines - switch to prefix-version-short_id for pipeline ids

winlogbeat/module/pipeline.go

x-pack/winlogbeat/cmd/root.go

andrewkroh

LGTM 🚀

andrewkroh · 2022-01-04T16:32:31Z

I'll merge this when it goes green.

What does this PR do? Switches to ingest node processing instead of local javascript processing for modules. Why is it important? For Agent we are using ingest node processing. This keeps the Winlogbeat and Agent processing in sync and allows bug fixes to be shared between them. Changes - remove JS processing - remove testing of JS processing - add ingest node pipelines from agent - use routing pipeline to direct to correct module pipeline - embed pipelines in winlogbeat executable - add `export pipeline` command - add loading of pipelines to setup command - add loading of pipelines to every ES connection if the pipelines don't exist - exports several functions from filebeat/fileset to avoid code duplication - update docs - update default config - update reference config Closes #29184 (cherry picked from commit a3c2208)

nicpenning · 2022-01-04T20:02:02Z

This looks like a big blow to the Logstash users out there.

Is the local processing being removed entirely?

This would put in a very challenging barrier to upgrade for the users that rely on Logstash for all enrichment before sending to Elasticsearch.

The move makes sense but it will be great to have a plan for those who need to migrate away from heavy Logstash use.

One simple use case is that specific logs are tagged and indexed based on preparsed data from Winlogbeat. Maybe that will go away with data streams but it is without question users will be forced to use ingest pipelines to match current enrichment capabilities and they hope the ingest pipelines have the functionality needed to reproduce what Logstash can without adding complexity to the stack.

Also, I am wondering what this processing performance will look like as it goes from the individual systems to centralizing it on ingest nodes. When talking 10k+ or 100k+ winlogbeat agents, how much ingest processing will be needed since the decentralized processing goes away.

This is just my current reaction to what I can understand from this issue. Great capabilities regardless.

What does this PR do? Switches to ingest node processing instead of local javascript processing for modules. Why is it important? For Agent we are using ingest node processing. This keeps the Winlogbeat and Agent processing in sync and allows bug fixes to be shared between them. Changes - remove JS processing - remove testing of JS processing - add ingest node pipelines from agent - use routing pipeline to direct to correct module pipeline - embed pipelines in winlogbeat executable - add `export pipeline` command - add loading of pipelines to setup command - add loading of pipelines to every ES connection if the pipelines don't exist - exports several functions from filebeat/fileset to avoid code duplication - update docs - update default config - update reference config Closes #29184 (cherry picked from commit a3c2208) Co-authored-by: Lee E Hinman <57081003+leehinman@users.noreply.github.com>

herrBez · 2022-02-11T18:52:21Z

Hi there. Just a question. Would it not make sense to add the ECS field event.ingested directly in the routing pipeline, so that also logs from pplication and system would also have this field populated?

andrewkroh · 2022-02-14T21:22:22Z

@herrBez Please open a new issue. 👍

This brings partially processed event data from a modified version of the testing code at 8896fd3 (the commit immediately prior to the removal of the javascript processing pipeline in elastic#29435: commit at 2f3b0c5cbe9cfdd10e11fd52e2a259e564001100). The evtx.golden.json files were generated by removing the js processing call at https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135 to match the code in that file as it appears here and then run go test -update in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019 host. The test package for each of the modules is also resurrected with modifications reflecting the loss of the javascript processor. Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this commit.

This is a moderately involved change. The process is described in the steps below, but the reader should refer to the commits in the PR to see exactly what was done and when. * port forward semi-processed events for tests This brings partially processed event data from a modified version of the testing code at 8896fd3 (the commit immediately prior to the removal of the javascript processing pipeline in #29435: commit at 2f3b0c5cbe9cfdd10e11fd52e2a259e564001100). The evtx.golden.json files were generated by removing the js processing call at https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135 to match the code in that file as it appears here and then run go test -update in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019 host. The test package for each of the modules is also resurrected with modifications reflecting the loss of the javascript processor. Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this commit. * make sure metadata is available for enrichment of raw values This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in 33acb3c (2022-01-25) after the origin of the forward port origin (2021-12-02). * update golden file for sysmon-11-filedeletedetected.evtx * ignore opcode field on Windows 2022 Also prohibit generating golden files for PowerShell on Windows 2022 to prevent unnecessary work in discovering that this will fail on other versions. * defer event field filtering until value comparison This will result in additional diff noise if golden values are generated on multiple versions of windows so it may be worth keeping the version used reasonably constant. The version used here was 2019.

This is a moderately involved change. The process is described in the steps below, but the reader should refer to the commits in the PR to see exactly what was done and when. * port forward semi-processed events for tests This brings partially processed event data from a modified version of the testing code at 8896fd3 (the commit immediately prior to the removal of the javascript processing pipeline in #29435: commit at 2f3b0c5cbe9cfdd10e11fd52e2a259e564001100). The evtx.golden.json files were generated by removing the js processing call at https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135 to match the code in that file as it appears here and then run go test -update in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019 host. The test package for each of the modules is also resurrected with modifications reflecting the loss of the javascript processor. Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this commit. * make sure metadata is available for enrichment of raw values This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in 33acb3c (2022-01-25) after the origin of the forward port origin (2021-12-02). * update golden file for sysmon-11-filedeletedetected.evtx * ignore opcode field on Windows 2022 Also prohibit generating golden files for PowerShell on Windows 2022 to prevent unnecessary work in discovering that this will fail on other versions. * defer event field filtering until value comparison This will result in additional diff noise if golden values are generated on multiple versions of windows so it may be worth keeping the version used reasonably constant. The version used here was 2019. (cherry picked from commit f22abe3) # Conflicts: # x-pack/winlogbeat/module/security/test/testdata/security-windows2019_4688_Process_Created.evtx.golden.json # x-pack/winlogbeat/module/testing_windows.go

…t processing (#31491) * {,x-pack/}winlogbeat: resurrect tests for event processing (#31464) This is a moderately involved change. The process is described in the steps below, but the reader should refer to the commits in the PR to see exactly what was done and when. * port forward semi-processed events for tests This brings partially processed event data from a modified version of the testing code at 8896fd3 (the commit immediately prior to the removal of the javascript processing pipeline in #29435: commit at 2f3b0c5cbe9cfdd10e11fd52e2a259e564001100). The evtx.golden.json files were generated by removing the js processing call at https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135 to match the code in that file as it appears here and then run go test -update in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019 host. The test package for each of the modules is also resurrected with modifications reflecting the loss of the javascript processor. Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this commit. * make sure metadata is available for enrichment of raw values This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in 33acb3c (2022-01-25) after the origin of the forward port origin (2021-12-02). * update golden file for sysmon-11-filedeletedetected.evtx * ignore opcode field on Windows 2022 Also prohibit generating golden files for PowerShell on Windows 2022 to prevent unnecessary work in discovering that this will fail on other versions. * defer event field filtering until value comparison This will result in additional diff noise if golden values are generated on multiple versions of windows so it may be worth keeping the version used reasonably constant. The version used here was 2019. (cherry picked from commit f22abe3) # Conflicts: # x-pack/winlogbeat/module/security/test/testdata/security-windows2019_4688_Process_Created.evtx.golden.json # x-pack/winlogbeat/module/testing_windows.go * fix conflict scars * silence bad linter Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

ag-michael · 2022-06-01T17:40:32Z

@leehinman Does this mean users that use agent module javascript processors before forwarding to logstash are no longer supported?

nicpenning · 2022-06-01T18:12:03Z

@leehinman Does this mean users that use agent module javascript processors before forwarding to logstash are no longer supported?

That is what I believe since in the latest version (8.2.2) of the Winlogbeat download (zip) they no longer include the javascript modules to parse the data on ingest pipelines for Elastic ingest nodes. :(

efd6 · 2022-06-01T23:50:54Z

@ag-michael Yes, that is correct.

flemerle · 2022-06-23T14:39:15Z

Yes the javascript modules are no longer in the zip, I just happened to figure it as I upgraded an agent today. I imported the module folder from an other agent and it works, this way I can still use logstas as my front-end for my agents.
If found it kind of sad to go away from Logstash like this.

This is a moderately involved change. The process is described in the steps below, but the reader should refer to the commits in the PR to see exactly what was done and when. * port forward semi-processed events for tests This brings partially processed event data from a modified version of the testing code at 8896fd3 (the commit immediately prior to the removal of the javascript processing pipeline in #29435: commit at 2f3b0c5cbe9cfdd10e11fd52e2a259e564001100). The evtx.golden.json files were generated by removing the js processing call at https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135 to match the code in that file as it appears here and then run go test -update in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019 host. The test package for each of the modules is also resurrected with modifications reflecting the loss of the javascript processor. Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this commit. * make sure metadata is available for enrichment of raw values This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in 33acb3c (2022-01-25) after the origin of the forward port origin (2021-12-02). * update golden file for sysmon-11-filedeletedetected.evtx * ignore opcode field on Windows 2022 Also prohibit generating golden files for PowerShell on Windows 2022 to prevent unnecessary work in discovering that this will fail on other versions. * defer event field filtering until value comparison This will result in additional diff noise if golden values are generated on multiple versions of windows so it may be worth keeping the version used reasonably constant. The version used here was 2019.

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 15, 2021

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Dec 15, 2021

leehinman added backport-v8.0.0 Automated backport with mergify Team:Security-External Integrations labels Dec 15, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 15, 2021

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Dec 15, 2021

leehinman force-pushed the winlogbeat_pipelines branch from 98c4e77 to 2f3b0c5 Compare December 17, 2021 19:07

leehinman marked this pull request as ready for review December 17, 2021 19:09

leehinman requested a review from a team as a code owner December 17, 2021 19:09

taylor-swanson reviewed Dec 17, 2021

View reviewed changes

leehinman force-pushed the winlogbeat_pipelines branch from 2f3b0c5 to 646b91d Compare December 17, 2021 22:05

andrewkroh reviewed Dec 20, 2021

View reviewed changes

leehinman and others added 9 commits December 24, 2021 15:12

wrap newly created errors

86d05c3

Update libbeat/docs/command-reference.asciidoc

5f63913

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update libbeat/docs/command-reference.asciidoc

41b68a9

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update winlogbeat/module/pipeline.go

a6220c4

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update winlogbeat/module/pipeline.go

0a1486f

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update x-pack/winlogbeat/_meta/config/output-elasticsearch.yml.tmpl

1f177e1

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

remove license check

be0b240

- remove reference to Logstash pipelines - cleanup grammar in godoc - move logger creation

incorporate feedback

383c80d

- always honor winlogbeat.overwrite_pipelines - switch to prefix-version-short_id for pipeline ids

leehinman force-pushed the winlogbeat_pipelines branch from 61dd69b to 383c80d Compare December 24, 2021 23:13

efd6 reviewed Jan 3, 2022

View reviewed changes

winlogbeat/module/pipeline.go Show resolved Hide resolved

x-pack/winlogbeat/cmd/root.go Outdated Show resolved Hide resolved

efd6 mentioned this pull request Jan 3, 2022

Add source.address to EVENT ID 3 #29658

Closed

6 tasks

Address Dan's comments

bb908aa

andrewkroh approved these changes Jan 4, 2022

View reviewed changes

andrewkroh added breaking change release-note:breaking The content should be included as a breaking change labels Jan 4, 2022

andrewkroh merged commit a3c2208 into elastic:master Jan 4, 2022

mergify bot mentioned this pull request Jan 4, 2022

[8.0](backport #29435) [Winlogbeat] Switch to ingest node processing #29685

Merged

leehinman mentioned this pull request Feb 15, 2022

[Winlogbeat] add tests for ingest node pipelines #30406

Closed

herrBez mentioned this pull request Feb 22, 2022

[Winlogbeat] Move addition of event.ingested field from the modules' pipelines to the winlogbeat-routing pipeline #30516

Closed

efd6 mentioned this pull request Jul 28, 2022

winlogbeat: provide clearer documentation for migration from endpoint to ingest node processing #32531

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Winlogbeat] Switch to ingest node processing #29435

[Winlogbeat] Switch to ingest node processing #29435

leehinman commented Dec 15, 2021 •

edited

Loading

mergify bot commented Dec 15, 2021

elasticmachine commented Dec 15, 2021 •

edited by jenkins-beats-ci bot

Loading

elasticmachine commented Dec 17, 2021

taylor-swanson Dec 17, 2021

leehinman Dec 18, 2021

andrewkroh left a comment

andrewkroh Dec 20, 2021

andrewkroh left a comment

andrewkroh commented Jan 4, 2022

nicpenning commented Jan 4, 2022

herrBez commented Feb 11, 2022

andrewkroh commented Feb 14, 2022

ag-michael commented Jun 1, 2022

nicpenning commented Jun 1, 2022

efd6 commented Jun 1, 2022

flemerle commented Jun 23, 2022

[Winlogbeat] Switch to ingest node processing #29435

[Winlogbeat] Switch to ingest node processing #29435

Conversation

leehinman commented Dec 15, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

mergify bot commented Dec 15, 2021

elasticmachine commented Dec 15, 2021 • edited by jenkins-beats-ci bot Loading

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Dec 17, 2021

taylor-swanson Dec 17, 2021

Choose a reason for hiding this comment

leehinman Dec 18, 2021

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh Dec 20, 2021

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh commented Jan 4, 2022

nicpenning commented Jan 4, 2022

herrBez commented Feb 11, 2022

andrewkroh commented Feb 14, 2022

ag-michael commented Jun 1, 2022

nicpenning commented Jun 1, 2022

efd6 commented Jun 1, 2022

flemerle commented Jun 23, 2022

leehinman commented Dec 15, 2021 •

edited

Loading

elasticmachine commented Dec 15, 2021 •

edited by jenkins-beats-ci bot

Loading