[Filebeat] change multiline configuration in awss3 input to parsers #25873

leehinman · 2021-05-26T15:39:47Z

What does this PR do?

Changes configuration of multiline in awss3 input to parsers to match multiline configuration in filestream input.

Why is it important?

This is the new paradigm for configuring multiline and we want consistency between inputs.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Setup S3 & SQS in AWS
Set QUEUE_URL shell variable
Set AWS_PROFILE_NAME shell variable
Set S3_BUCKET_NAME shell variable
Set S3_BUCKET_REGION shell variable
Run go test --tags=integration,aws

Related issues

Closes [Filebeat] Support multiline options in aws s3 input #25249

elasticmachine · 2021-05-26T15:49:46Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25873 updated
Start Time: 2021-06-29T17:58:27.277+0000
Duration: 111 min 31 sec
Commit: 8c88824

Test stats 🧪

Test	Results
Failed	0
Passed	7402
Skipped	1201
Total	8603

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	7402
Skipped	1201
Total	8603

elasticmachine · 2021-05-28T18:26:23Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

urso · 2021-05-28T19:02:34Z

x-pack/filebeat/input/awss3/parser.go

+	lineTerminator readfile.LineTerminator
+}
+
+func newParsers(in reader.Reader, pCfg parserConfig, c []common.ConfigNamespace) (parser, error) {


@kvch this looks like we have to duplicate parser setup from the filestream input here. This is somewhat unfortunate, especially if we want to add similar functionality to other inputs. How about providing a common package for parser setup?

Inputs that want to make use of parsers should be able to do it more or less like this:

type Config struct { ... Parser inpparser.Settings ... } ... parser, err := config.Parser.Create(reader) ...

mergify · 2021-06-14T17:08:25Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 25249_awss3_add_parsers upstream/25249_awss3_add_parsers
git merge upstream/master
git push upstream 25249_awss3_add_parsers

andrewkroh · 2021-06-23T13:54:07Z

Just noting that if this doesn't merge for 7.14 that it will need to be updated to reflect that this is a breaking config change since the multiline option is already present (new for 7.14).

leehinman · 2021-06-29T14:10:39Z

needs to be rebased onto #26541 when that is merged

- switches multiline configuration to parsers - JSON parsing is independent Closes elastic#25249

andrewkroh

LGTM

…25873) * change multiline configuration in awss3 input to parsers - switches multiline configuration to parsers - JSON parsing is independent Closes #25249 (cherry picked from commit beaa972)

…25873) (#26586) * change multiline configuration in awss3 input to parsers - switches multiline configuration to parsers - JSON parsing is independent Closes #25249 (cherry picked from commit beaa972) Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>

* master: (25 commits) fix: Force PLATFORMS environment variable when we build Elastic Agent dependencies on arm64 (elastic#26415) macos for metricbeat to run in the extended meta-stage (elastic#26573) Packaging: add arm7 platform in the main pipeline (elastic#26575) [Heartbeat] Skip flakey timer queue test (elastic#26592) Update to "read_pipeline" permission (elastic#26465) (elastic#26580) API keys do not reflect the need for read_pipeline (elastic#26466) (elastic#26582) Add Fleet agent.id to Agent monitoring data (elastic#26548) Add kinesis metricset (elastic#25989) Refactor of system/memory metricset (elastic#26334) Introduce httpcommon package in libbeat (add support for Proxy) (elastic#25219) [Filebeat] change multiline configuration in awss3 input to parsers (elastic#25873) docs: Hint for the error "Error extracting container id" (elastic#25824) [Docs] Fixed metricbeat redis exported field CPU descriptions (elastic#25846) (elastic#26496) Update urllib to 1.26.5. (elastic#26380) Update golang.org/x/crypto (elastic#26448) [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816) Move parsers outside of filestream input so others can use them as well (elastic#26541) [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508) [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620) Logging code cleanup related to Nomad auto-discovery (elastic#26498) ...

jsoriano · 2021-06-30T15:00:56Z

x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc

+        match:   after
+----
+
+See the available parser settings in detail below.


@kvch would it be possible/make sense to have a shared asciidoc that can be included in all the inputs with parsers?

leehinman added Filebeat Filebeat cleanup Team:Security-External Integrations backport-v7.14.0 Automated backport with mergify labels May 26, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 26, 2021

leehinman force-pushed the 25249_awss3_add_parsers branch from ab5e9c8 to 901b2cb Compare May 26, 2021 15:47

leehinman force-pushed the 25249_awss3_add_parsers branch 3 times, most recently from 3381e44 to 8bf160f Compare May 28, 2021 16:17

leehinman marked this pull request as ready for review May 28, 2021 18:26

leehinman requested a review from kvch May 28, 2021 18:26

urso reviewed May 28, 2021

View reviewed changes

andrewkroh mentioned this pull request Jun 23, 2021

[Filebeat] Support multiline options in aws s3 input #25249

Closed

leehinman added 2 commits June 29, 2021 10:07

change multiline configuration in awss3 input to parsers

e8f2109

- switches multiline configuration to parsers - JSON parsing is independent Closes elastic#25249

switch to shared parsers implementation

8c88824

leehinman force-pushed the 25249_awss3_add_parsers branch from 8bf160f to 8c88824 Compare June 29, 2021 17:58

andrewkroh approved these changes Jun 29, 2021

View reviewed changes

leehinman merged commit beaa972 into elastic:master Jun 29, 2021

mergify bot mentioned this pull request Jun 29, 2021

[7.x](backport #25873) [Filebeat] change multiline configuration in awss3 input to parsers #26586

Merged

kvch mentioned this pull request Jun 30, 2021

Adopt parsers in all Filebeat inputs #26130

Closed

21 tasks

jsoriano reviewed Jun 30, 2021

View reviewed changes

leehinman deleted the 25249_awss3_add_parsers branch August 17, 2021 21:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] change multiline configuration in awss3 input to parsers #25873

[Filebeat] change multiline configuration in awss3 input to parsers #25873

leehinman commented May 26, 2021 •

edited

Loading

elasticmachine commented May 26, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

elasticmachine commented May 28, 2021

urso May 28, 2021

mergify bot commented Jun 14, 2021

andrewkroh commented Jun 23, 2021 •

edited

Loading

leehinman commented Jun 29, 2021

andrewkroh left a comment

jsoriano Jun 30, 2021

[Filebeat] change multiline configuration in awss3 input to parsers #25873

[Filebeat] change multiline configuration in awss3 input to parsers #25873

Conversation

leehinman commented May 26, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented May 26, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented May 28, 2021

urso May 28, 2021

Choose a reason for hiding this comment

mergify bot commented Jun 14, 2021

andrewkroh commented Jun 23, 2021 • edited Loading

leehinman commented Jun 29, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

jsoriano Jun 30, 2021

Choose a reason for hiding this comment

leehinman commented May 26, 2021 •

edited

Loading

elasticmachine commented May 26, 2021 •

edited by jenkins-beats-ci bot

Loading

andrewkroh commented Jun 23, 2021 •

edited

Loading