{,x-pack/}winlogbeat: resurrect tests for event processing #31464

efd6 · 2022-04-29T02:08:11Z

What does this PR do?

This resurrects a partial testing of the winlogbeat event processing. It does not examine the ingest pipeline (that will be done as #30406) but does ensure we do not have regressions in the parts of the input that deal directly with the Windows event logging API.

Why is it important?

We had inadequate testing.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

Author's Checklist

The change here clobbers all the existing tests. At some stage the original state of those should be restored, or alternatively a new set should be added as system tests with the full ingest enrichment which would be the state that these were prior to this change.

How to test this PR locally

Tests can be run locally using the go tool on a windows host.

Related issues

Use cases

Screenshots

Logs

elasticmachine · 2022-04-29T02:08:14Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2022-04-29T02:43:57Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-05-02T23:46:21.257+0000
Duration: 50 min 22 sec

Test stats 🧪

Test	Results
Failed	0
Passed	930
Skipped	9
Total	939

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

efd6 · 2022-04-29T03:29:24Z

winlogbeat/eventlog/wineventlog.go

+	md, err := win.NewPublisherMetadataStore(win.NilHandle, e.Provider.Name, logp.L())
+	if err != nil {
+		// Return an empty store on error (can happen in cases where the
+		// log was forwarded and the provider doesn't exist on collector).
+		md = win.NewEmptyPublisherMetadataStore(e.Provider.Name, logp.L())
+		logp.Warn("failed to load publisher metadata for %v "+
+			"(returning an empty metadata store): %v", e.Provider.Name, err)
+	}


It may be worth caching this in the *winEventLog, but I'd like to do that after getting the testing done.

mergify · 2022-04-29T07:23:10Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b winlogbeatforwardporttests upstream/winlogbeatforwardporttests
git merge upstream/main
git push upstream winlogbeatforwardporttests

This brings partially processed event data from a modified version of the testing code at 8896fd3 (the commit immediately prior to the removal of the javascript processing pipeline in elastic#29435: commit at 2f3b0c5cbe9cfdd10e11fd52e2a259e564001100). The evtx.golden.json files were generated by removing the js processing call at https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135 to match the code in that file as it appears here and then run go test -update in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019 host. The test package for each of the modules is also resurrected with modifications reflecting the loss of the javascript processor. Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this commit.

…f raw values This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in 33acb3c (2022-01-25) after the origin of the forward port origin (2021-12-02).

…eletedetected.evtx

x-pack/winlogbeat/module/powershell/test/powershell_windows_test.go

Also prohibit generating golden files for PowerShell on Windows 2022 to prevent unnecessary work in discovering that this will fail on other versions.

andrewkroh

It's really good to see these tests re-enabled.

x-pack/winlogbeat/module/testing_windows.go

This will result in additional diff noise if golden values are generated on multiple versions of windows so it may be worth keeping the version used reasonably constant. The version used here was 2019.

This is a moderately involved change. The process is described in the steps below, but the reader should refer to the commits in the PR to see exactly what was done and when. * port forward semi-processed events for tests This brings partially processed event data from a modified version of the testing code at 8896fd3 (the commit immediately prior to the removal of the javascript processing pipeline in #29435: commit at 2f3b0c5cbe9cfdd10e11fd52e2a259e564001100). The evtx.golden.json files were generated by removing the js processing call at https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135 to match the code in that file as it appears here and then run go test -update in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019 host. The test package for each of the modules is also resurrected with modifications reflecting the loss of the javascript processor. Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this commit. * make sure metadata is available for enrichment of raw values This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in 33acb3c (2022-01-25) after the origin of the forward port origin (2021-12-02). * update golden file for sysmon-11-filedeletedetected.evtx * ignore opcode field on Windows 2022 Also prohibit generating golden files for PowerShell on Windows 2022 to prevent unnecessary work in discovering that this will fail on other versions. * defer event field filtering until value comparison This will result in additional diff noise if golden values are generated on multiple versions of windows so it may be worth keeping the version used reasonably constant. The version used here was 2019. (cherry picked from commit f22abe3) # Conflicts: # x-pack/winlogbeat/module/security/test/testdata/security-windows2019_4688_Process_Created.evtx.golden.json # x-pack/winlogbeat/module/testing_windows.go

…t processing (#31491) * {,x-pack/}winlogbeat: resurrect tests for event processing (#31464) This is a moderately involved change. The process is described in the steps below, but the reader should refer to the commits in the PR to see exactly what was done and when. * port forward semi-processed events for tests This brings partially processed event data from a modified version of the testing code at 8896fd3 (the commit immediately prior to the removal of the javascript processing pipeline in #29435: commit at 2f3b0c5cbe9cfdd10e11fd52e2a259e564001100). The evtx.golden.json files were generated by removing the js processing call at https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135 to match the code in that file as it appears here and then run go test -update in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019 host. The test package for each of the modules is also resurrected with modifications reflecting the loss of the javascript processor. Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this commit. * make sure metadata is available for enrichment of raw values This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in 33acb3c (2022-01-25) after the origin of the forward port origin (2021-12-02). * update golden file for sysmon-11-filedeletedetected.evtx * ignore opcode field on Windows 2022 Also prohibit generating golden files for PowerShell on Windows 2022 to prevent unnecessary work in discovering that this will fail on other versions. * defer event field filtering until value comparison This will result in additional diff noise if golden values are generated on multiple versions of windows so it may be worth keeping the version used reasonably constant. The version used here was 2019. (cherry picked from commit f22abe3) # Conflicts: # x-pack/winlogbeat/module/security/test/testdata/security-windows2019_4688_Process_Created.evtx.golden.json # x-pack/winlogbeat/module/testing_windows.go * fix conflict scars * silence bad linter Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

…d fix bugs (#31096) * winlogbeat: fix event handling for Windows 2022 (#30942) This fixes failures in event handling on Windows 2022[1] where parts of events available from the Windows API are not reflected in the events recovered by winlogbeat[2]. There is unfortunately quite a lot of movement in this change due to the need to satisfy linter requirements. Beyond those changes, the substantive changes here are: 1. Addition of new testing infrastructure to allow addition of evtx files and comparison with there expected XML renderings, and adding some test cases (shown below). 2. Fixing a buffer length parameter in the call to _EvtFormatMessage in evtFormatMessage that was the result of a lack of clarity in the API documentation for that syscall. 3. Fixing a var shadowing decl of the publisher handle EvtHandle in FormatEventString. 4. Providing a call back for the legacy (non-experimental) API through wineventlog.Message to allow it to obtain the event message in the case that the RenderingInfo element is not available via the Windows API. 5. Ensure that keyword, opcode and level are obtained by the non-experimental API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML. This change also required making winevent.Event.OpcodeRaw a pointer to allow an absent System>Opcode element to be distinquished from the zero, but present element. The change also enables testing on Windows 2022. [1]#30621 [2]#30622 (comment) New events in testing as rendered by the Event Viewer: ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="EventCreate" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" /> <EventRecordID>316</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>My custom error event for the application log</Data> </EventData> </Event> ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="WinWord" /> <EventID Qualifiers="0">999</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" /> <EventRecordID>317</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Winword event 999 happened due to low diskspace</Data> </EventData> </Event> ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="system" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" /> <EventRecordID>1413</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Catastrophe!</Data> </EventData> </Event> ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Backup" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" /> <EventRecordID>1414</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Backup failure</Data> </EventData> </Event> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 34bdc3d) # Conflicts: # winlogbeat/Jenkinsfile.yml # winlogbeat/beater/winlogbeat.go # winlogbeat/eventlog/wineventlog.go # winlogbeat/eventlog/wineventlog_experimental.go # winlogbeat/sys/wineventlog/wineventlog_windows.go # x-pack/winlogbeat/Jenkinsfile.yml * clean up conflict scars * cherrypick winlogbeat test fixes relevant to 7.17 from #31464 Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

This is a moderately involved change. The process is described in the steps below, but the reader should refer to the commits in the PR to see exactly what was done and when. * port forward semi-processed events for tests This brings partially processed event data from a modified version of the testing code at 8896fd3 (the commit immediately prior to the removal of the javascript processing pipeline in #29435: commit at 2f3b0c5cbe9cfdd10e11fd52e2a259e564001100). The evtx.golden.json files were generated by removing the js processing call at https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135 to match the code in that file as it appears here and then run go test -update in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019 host. The test package for each of the modules is also resurrected with modifications reflecting the loss of the javascript processor. Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this commit. * make sure metadata is available for enrichment of raw values This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in 33acb3c (2022-01-25) after the origin of the forward port origin (2021-12-02). * update golden file for sysmon-11-filedeletedetected.evtx * ignore opcode field on Windows 2022 Also prohibit generating golden files for PowerShell on Windows 2022 to prevent unnecessary work in discovering that this will fail on other versions. * defer event field filtering until value comparison This will result in additional diff noise if golden values are generated on multiple versions of windows so it may be worth keeping the version used reasonably constant. The version used here was 2019.

efd6 added enhancement Team:Security-External Integrations backport-skip Skip notification from the automated backport with mergify 8.3-candidate labels Apr 29, 2022

efd6 requested a review from a team as a code owner April 29, 2022 02:08

efd6 self-assigned this Apr 29, 2022

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 29, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 29, 2022

efd6 requested a review from andrewkroh April 29, 2022 02:09

efd6 force-pushed the winlogbeatforwardporttests branch 2 times, most recently from 11b593f to b028010 Compare April 29, 2022 03:26

efd6 commented Apr 29, 2022

View reviewed changes

efd6 added backport-v8.2.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Apr 29, 2022

efd6 added 3 commits May 2, 2022 07:32

x-pack/winlogbeat/module/sysmon/test: golden file for sysmon-11-filed…

8039eaa

…eletedetected.evtx

efd6 force-pushed the winlogbeatforwardporttests branch from b028010 to 6cf2660 Compare May 1, 2022 22:38

efd6 commented May 1, 2022

View reviewed changes

x-pack/winlogbeat/module/powershell/test/powershell_windows_test.go Outdated Show resolved Hide resolved

efd6 force-pushed the winlogbeatforwardporttests branch from 6cf2660 to 4a3d585 Compare May 1, 2022 23:43

efd6 mentioned this pull request May 2, 2022

[7.17](backport #30942) winlogbeat/sys/wineventlog: extend testing and fix bugs #31096

Merged

X-pack/winlogbeat/module: ignore opcode field on Windows 2022

24da692

Also prohibit generating golden files for PowerShell on Windows 2022 to prevent unnecessary work in discovering that this will fail on other versions.

efd6 force-pushed the winlogbeatforwardporttests branch from 4a3d585 to 24da692 Compare May 2, 2022 01:48

andrewkroh approved these changes May 2, 2022

View reviewed changes

x-pack/winlogbeat/module/testing_windows.go Outdated Show resolved Hide resolved

defer event field filtering until value comparison

7e5ec1b

This will result in additional diff noise if golden values are generated on multiple versions of windows so it may be worth keeping the version used reasonably constant. The version used here was 2019.

efd6 mentioned this pull request May 2, 2022

winlogbeat: fix handling of opcode string recovery on Windows 2022 #31490

Closed

add tracking issue for Windows 2022 opcode handling

4089f31

andrewkroh approved these changes May 3, 2022

View reviewed changes

efd6 merged commit f22abe3 into elastic:main May 3, 2022

mergify bot mentioned this pull request May 3, 2022

[8.2](backport #31464) {,x-pack/}winlogbeat: resurrect tests for event processing #31491

Merged

efd6 added a commit that referenced this pull request May 3, 2022

cherrypick winlogbeat test fixes relevant to 7.17 from #31464

32daffe

efd6 mentioned this pull request May 4, 2022

winlogbeat/eventlog: cache winevent.WinMeta values #31504

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{,x-pack/}winlogbeat: resurrect tests for event processing #31464

{,x-pack/}winlogbeat: resurrect tests for event processing #31464

efd6 commented Apr 29, 2022 •

edited

Loading

elasticmachine commented Apr 29, 2022

elasticmachine commented Apr 29, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

efd6 Apr 29, 2022

mergify bot commented Apr 29, 2022

andrewkroh left a comment

{,x-pack/}winlogbeat: resurrect tests for event processing #31464

{,x-pack/}winlogbeat: resurrect tests for event processing #31464

Conversation

efd6 commented Apr 29, 2022 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Apr 29, 2022

elasticmachine commented Apr 29, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

efd6 Apr 29, 2022

Choose a reason for hiding this comment

mergify bot commented Apr 29, 2022

andrewkroh left a comment

Choose a reason for hiding this comment

efd6 commented Apr 29, 2022 •

edited

Loading

elasticmachine commented Apr 29, 2022 •

edited by jenkins-beats-ci bot

Loading