Cherry-pick #24763 to 7.x: Add support for parsers in filestream input #25224

kvch · 2021-04-22T08:33:20Z

Cherry-pick of PR #24763 to 7.x branch. Original message:

This PR adds support for pasers in the filestream input.

Example configuration that aggregates fives lines into a single event and parses the JSON contents:

- type: filestream
  enabled: true
  paths:
  - test.log
  parsers:
  - multiline:
      type: count
      count_lines: 5
      skip_newline: true
  - ndjson:
      fields_under_root: true

This PR adds support for pasers in the `filestream` input. Example configuration that aggregates fives lines into a single event and parses the JSON contents: ```yaml - type: filestream enabled: true paths: - test.log parsers: - multiline: type: count count_lines: 5 skip_newline: true - ndjson: fields_under_root: true ``` (cherry picked from commit 30331bc)

elasticmachine · 2021-04-22T08:33:24Z

Pinging @elastic/agent (Team:Agent)

elasticmachine · 2021-04-22T10:20:17Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25224 opened
Start Time: 2021-04-22T08:34:14.369+0000
Duration: 105 min 25 sec
Commit: 3b40e99

Test stats 🧪

Test	Results
Failed	1
Passed	47163
Skipped	5190
Total	52354

Trends 🧪

Test errors

Expand to view the tests failures

`Build&Test / metricbeat-pythonIntegTest / test_node_stats – metricbeat.module.logstash.test_logstash.Test`

Expand to view the error details

 AssertionError: Element counts were not equal: First has 1, Second has 0:  'logstash' First has 0, Second has 1:  'error'

Expand to view the stacktrace

 self = <test_logstash.Test testMethod=test_node_stats>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, "integration test")
    def test_node_stats(self):
        """
        logstash node_stats metricset test
        """
>       self.check_metricset("logstash", "node_stats", self.get_hosts(), self.FIELDS)

module/logstash/test_logstash.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/metricbeat.py:105: in check_metricset
    self.assertCountEqual(self.de_dot(fields), evt.keys())
E   AssertionError: Element counts were not equal:
E   First has 1, Second has 0:  'logstash'
E   First has 0, Second has 1:  'error'

Steps errors

Expand to view the steps failures

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

Took 5 min 10 sec . View more details on here
Description: make -C generator/_templates/metricbeat test test-package

`generator-beat-test - make -C generator/_templates/beat test test-package`

Took 7 min 19 sec . View more details on here
Description: make -C generator/_templates/beat test test-package

`metricbeat-pythonIntegTest - mage pythonIntegTest`

Took 31 min 0 sec . View more details on here
Description: mage pythonIntegTest

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 2'

Log output

Expand to view the last 100 lines of log output

[2021-04-22T10:18:54.631Z] 9.66s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_022_zscaler
[2021-04-22T10:18:54.631Z] 9.53s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_096_gcp
[2021-04-22T10:18:54.631Z] 9.51s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_118_squid
[2021-04-22T10:18:54.631Z] 9.49s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_315_cisco
[2021-04-22T10:18:54.631Z] 9.44s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_370_oracle
[2021-04-22T10:18:54.631Z] ================ 402 passed, 389 warnings in 1810.26s (0:30:10) ================
[2021-04-22T10:18:54.889Z] >> python test: Integration Testing Complete
[2021-04-22T10:18:58.745Z] Cleaning up /var/lib/jenkins/workspace/PR-25224-1-e33f09b7-a4f0-4677-83f2-e2924a22c4ae
[2021-04-22T10:18:58.745Z] Client: Docker Engine - Community
[2021-04-22T10:18:58.745Z]  Version:           20.10.3
[2021-04-22T10:18:58.745Z]  API version:       1.41
[2021-04-22T10:18:58.745Z]  Go version:        go1.13.15
[2021-04-22T10:18:58.745Z]  Git commit:        48d30b5
[2021-04-22T10:18:58.745Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-04-22T10:18:58.745Z]  OS/Arch:           linux/amd64
[2021-04-22T10:18:58.745Z]  Context:           default
[2021-04-22T10:18:58.745Z]  Experimental:      true
[2021-04-22T10:18:58.745Z] 
[2021-04-22T10:18:58.745Z] Server: Docker Engine - Community
[2021-04-22T10:18:58.745Z]  Engine:
[2021-04-22T10:18:58.745Z]   Version:          20.10.3
[2021-04-22T10:18:58.745Z]   API version:      1.41 (minimum version 1.12)
[2021-04-22T10:18:58.745Z]   Go version:       go1.13.15
[2021-04-22T10:18:58.745Z]   Git commit:       46229ca
[2021-04-22T10:18:58.745Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-22T10:18:58.745Z]   OS/Arch:          linux/amd64
[2021-04-22T10:18:58.745Z]   Experimental:     false
[2021-04-22T10:18:58.745Z]  containerd:
[2021-04-22T10:18:58.745Z]   Version:          1.4.4
[2021-04-22T10:18:58.745Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-22T10:18:58.745Z]  runc:
[2021-04-22T10:18:58.745Z]   Version:          1.0.0-rc93
[2021-04-22T10:18:58.745Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-22T10:18:58.745Z]  docker-init:
[2021-04-22T10:18:58.745Z]   Version:          0.19.0
[2021-04-22T10:18:58.745Z]   GitCommit:        de40ad0
[2021-04-22T10:18:58.745Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-22T10:18:58.745Z] Unable to find image 'alpine:3.4' locally
[2021-04-22T10:18:59.313Z] 3.4: Pulling from library/alpine
[2021-04-22T10:18:59.881Z] c1e54eec4b57: Pulling fs layer
[2021-04-22T10:19:00.141Z] c1e54eec4b57: Verifying Checksum
[2021-04-22T10:19:00.141Z] c1e54eec4b57: Download complete
[2021-04-22T10:19:00.400Z] c1e54eec4b57: Pull complete
[2021-04-22T10:19:00.400Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-04-22T10:19:00.400Z] Status: Downloaded newer image for alpine:3.4
[2021-04-22T10:19:02.319Z] Change permissions with write access of all files inside the specific folder
[2021-04-22T10:19:03.737Z] Running in /var/lib/jenkins/workspace/PR-25224-1-e33f09b7-a4f0-4677-83f2-e2924a22c4ae/src/github.com/elastic/beats/build
[2021-04-22T10:19:04.085Z] + rm -rf ve
[2021-04-22T10:19:04.085Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-04-22T10:19:04.524Z] + python .ci/scripts/pre_archive_test.py
[2021-04-22T10:19:07.059Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-04-22T10:19:07.112Z] Running in /var/lib/jenkins/workspace/PR-25224-1-e33f09b7-a4f0-4677-83f2-e2924a22c4ae/src/github.com/elastic/beats/build
[2021-04-22T10:19:07.177Z] Recording test results
[2021-04-22T10:19:10.090Z] [Checks API] No suitable checks publisher found.
[2021-04-22T10:19:10.664Z] + go clean -modcache
[2021-04-22T10:19:14.280Z] Cleaning up /var/lib/jenkins/workspace/PR-25224-1-e33f09b7-a4f0-4677-83f2-e2924a22c4ae
[2021-04-22T10:19:14.281Z] Client: Docker Engine - Community
[2021-04-22T10:19:14.281Z]  Version:           20.10.3
[2021-04-22T10:19:14.281Z]  API version:       1.41
[2021-04-22T10:19:14.281Z]  Go version:        go1.13.15
[2021-04-22T10:19:14.281Z]  Git commit:        48d30b5
[2021-04-22T10:19:14.281Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-04-22T10:19:14.281Z]  OS/Arch:           linux/amd64
[2021-04-22T10:19:14.281Z]  Context:           default
[2021-04-22T10:19:14.281Z]  Experimental:      true
[2021-04-22T10:19:14.281Z] 
[2021-04-22T10:19:14.281Z] Server: Docker Engine - Community
[2021-04-22T10:19:14.281Z]  Engine:
[2021-04-22T10:19:14.281Z]   Version:          20.10.3
[2021-04-22T10:19:14.281Z]   API version:      1.41 (minimum version 1.12)
[2021-04-22T10:19:14.281Z]   Go version:       go1.13.15
[2021-04-22T10:19:14.281Z]   Git commit:       46229ca
[2021-04-22T10:19:14.281Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-22T10:19:14.281Z]   OS/Arch:          linux/amd64
[2021-04-22T10:19:14.281Z]   Experimental:     false
[2021-04-22T10:19:14.281Z]  containerd:
[2021-04-22T10:19:14.281Z]   Version:          1.4.4
[2021-04-22T10:19:14.281Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-22T10:19:14.281Z]  runc:
[2021-04-22T10:19:14.281Z]   Version:          1.0.0-rc93
[2021-04-22T10:19:14.281Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-22T10:19:14.281Z]  docker-init:
[2021-04-22T10:19:14.281Z]   Version:          0.19.0
[2021-04-22T10:19:14.281Z]   GitCommit:        de40ad0
[2021-04-22T10:19:14.281Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-22T10:19:22.401Z] Change permissions with write access of all files inside the specific folder
[2021-04-22T10:19:23.019Z] Running in /var/lib/jenkins/workspace/PR-25224-1-e33f09b7-a4f0-4677-83f2-e2924a22c4ae
[2021-04-22T10:19:29.416Z] + gsutil --version
[2021-04-22T10:19:30.848Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-22T10:19:31.213Z] + gcloud auth activate-service-account --key-file ****
[2021-04-22T10:19:31.780Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-22T10:19:32.212Z] + gsutil -m -q cp -a public-read eC1wYWNrL2ZpbGViZWF0LWJ1aWxkM2I0MGU5OTJlYzRlYWY2MjAwM2ExN2M0NjM1MjI3YTAyZTQ2NjQ4Nw gs://beats-ci-temp/ci/cache/
[2021-04-22T10:19:38.125Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-22T10:19:38.269Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25224/src/github.com/elastic/beats
[2021-04-22T10:19:38.622Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25224
[2021-04-22T10:19:38.785Z] [INFO] getVaultSecret: Getting secrets
[2021-04-22T10:19:38.882Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-22T10:19:39.882Z] + chmod 755 generate-build-data.sh
[2021-04-22T10:19:39.882Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25224/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25224/runs/1 FAILURE 6325241
[2021-04-22T10:19:40.133Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25224/runs/1/steps/?limit=10000 -o steps-info.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	1
Passed	47163
Skipped	5190
Total	52354

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / metricbeat-pythonIntegTest / test_node_stats – metricbeat.module.logstash.test_logstash.Test

kvch added [zube]: In Review backport Team:Elastic-Agent Label for the Agent team labels Apr 22, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 22, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 22, 2021

kvch merged commit e3ebe2c into elastic:7.x Apr 22, 2021

zube bot added [zube]: Done and removed [zube]: In Review labels Apr 22, 2021

zube bot removed the [zube]: Done label Jul 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry-pick #24763 to 7.x: Add support for parsers in filestream input #25224

Cherry-pick #24763 to 7.x: Add support for parsers in filestream input #25224

kvch commented Apr 22, 2021 •

edited by zube bot

Loading

elasticmachine commented Apr 22, 2021

elasticmachine commented Apr 22, 2021

Build stats

Test stats 🧪

Trends 🧪

`Build&Test / metricbeat-pythonIntegTest / test_node_stats – metricbeat.module.logstash.test_logstash.Test`

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`metricbeat-pythonIntegTest - mage pythonIntegTest`

`Error signal`

Test stats 🧪

Genuine test errors

Cherry-pick #24763 to 7.x: Add support for parsers in filestream input #25224

Cherry-pick #24763 to 7.x: Add support for parsers in filestream input #25224

Conversation

kvch commented Apr 22, 2021 • edited by zube bot Loading

elasticmachine commented Apr 22, 2021

elasticmachine commented Apr 22, 2021

💔 Tests Failed

Build stats

Test stats 🧪

Trends 🧪

Test errors

Build&Test / metricbeat-pythonIntegTest / test_node_stats – metricbeat.module.logstash.test_logstash.Test

Steps errors

generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package

generator-beat-test - make -C generator/_templates/beat test test-package

metricbeat-pythonIntegTest - mage pythonIntegTest

Error signal

Log output

🐛 Flaky test report

Test stats 🧪

Genuine test errors

kvch commented Apr 22, 2021 •

edited by zube bot

Loading

`Build&Test / metricbeat-pythonIntegTest / test_node_stats – metricbeat.module.logstash.test_logstash.Test`

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`metricbeat-pythonIntegTest - mage pythonIntegTest`

`Error signal`