[Ingest Manager] Send datastreams fields #20402

michalpristas · 2020-08-03T14:01:28Z

What does this PR do?

Adds datastream.* fields as for request described here elastic/integrations#213 and elastic/elasticsearch#60592

Why is it important?

Desciben in an issue here elastic/integrations#213 and elastic/elasticsearch#60592

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

cc @ruflin

elasticmachine · 2020-08-03T14:01:31Z

Pinging @elastic/ingest-management (Team:Ingest Management)

blakerouse

Missed some in the monitoring.

blakerouse · 2020-08-03T14:06:30Z

x-pack/elastic-agent/pkg/agent/operation/monitoring.go

+							"target": "datastream",
+							"fields": map[string]interface{}{
+								"type":      "logs",
+								"dataset":   "elastic.agent",


This is wrong, should be fmt.Sprintf("elastic.agent.%s", name)

@blakerouse updated

blakerouse · 2020-08-03T14:06:54Z

x-pack/elastic-agent/pkg/agent/operation/monitoring.go

+								"namespace": "default",
+							},
+						},
+					},
 					{
 						"add_fields": map[string]interface{}{


You missed adding it to getMonitoringMetricbeatConfig

thanks, i totally missed that

elasticmachine · 2020-08-03T14:16:21Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #20402 updated]
Start Time: 2020-08-05T07:58:39.780+0000
Duration: 33 min 10 sec

blakerouse

Thanks for the fixes. Looks good!

ruflin · 2020-08-03T14:51:45Z

@michalpristas did not test it locally but remember sometimes the processors for the monitoring data went missing. Just double checking.

ruflin · 2020-08-03T15:10:49Z

@michalpristas Just saw that this PR is against master. Most importantly we will need the change in 7.9. 7.10 will be only datastreams so not a direct backport. Perhaps better to point it to 7.9 directly?

[Ingest Manager] Send datastreams fields (elastic#20402)

[Ingest Manager] Send datastreams fields (#20402)

…ne-2.0 * upstream/master: [docs] Promote ingest management to beta (elastic#20295) Upgrade elasticsearch client library used in tests (elastic#20405) Disable logging when pulling on python integration tests (elastic#20397) Remove pillow from testing requirements.txt (elastic#20407) [Filebeat][ATP Module]Setting user agent field required by the API (elastic#20440) [Ingest Manager] Send datastreams fields (elastic#20402) Add event.ingested to all Filebeat modules (elastic#20386) [Elastic Agent] Fix agent control socket path to always be less than 107 characters (elastic#20426) Improve cgroup_regex docs with examples (elastic#20425) Makes `metrics` config option required in app_insights (elastic#20406) Ensure install scripts only install if needed (elastic#20349) Update container name for the azure filesets (elastic#19899) Group same timestamp metrics values in app_insights metricset (elastic#20403) add_process_metadata processor adds container id even if process metadata not accessible (elastic#19767) Support "cluster" scope in Metricbeat elasticsearch module (elastic#18547) [Filebeat][SophosXG Module] Renaming module and fileset (elastic#20396) Update Suricata dashboards (elastic#20394) [Elastic Agent] Improve version, restart, enroll CLI commands (elastic#20359) Prepare home directories for docker images in a different stage (elastic#20356)

…allation * upstream/master: (23 commits) [docs] Promote ingest management to beta (elastic#20295) Upgrade elasticsearch client library used in tests (elastic#20405) Disable logging when pulling on python integration tests (elastic#20397) Remove pillow from testing requirements.txt (elastic#20407) [Filebeat][ATP Module]Setting user agent field required by the API (elastic#20440) [Ingest Manager] Send datastreams fields (elastic#20402) Add event.ingested to all Filebeat modules (elastic#20386) [Elastic Agent] Fix agent control socket path to always be less than 107 characters (elastic#20426) Improve cgroup_regex docs with examples (elastic#20425) Makes `metrics` config option required in app_insights (elastic#20406) Ensure install scripts only install if needed (elastic#20349) Update container name for the azure filesets (elastic#19899) Group same timestamp metrics values in app_insights metricset (elastic#20403) add_process_metadata processor adds container id even if process metadata not accessible (elastic#19767) Support "cluster" scope in Metricbeat elasticsearch module (elastic#18547) [Filebeat][SophosXG Module] Renaming module and fileset (elastic#20396) Update Suricata dashboards (elastic#20394) [Elastic Agent] Improve version, restart, enroll CLI commands (elastic#20359) Prepare home directories for docker images in a different stage (elastic#20356) New multiline mode in Filebeat: while_pattern (elastic#19662) ...

[Ingest Manager] Send datastreams fields (elastic#20402)

basic switch

f5b4c4d

michalpristas added needs_backport PR is waiting to be backported to other branches. Team:Ingest Management Ingest Management:beta1 Group issues for ingest management beta1 labels Aug 3, 2020

michalpristas self-assigned this Aug 3, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Aug 3, 2020

michalpristas changed the title ~~basic switch~~ [Ingest Manager] Send datastreams fields Aug 3, 2020

blakerouse requested changes Aug 3, 2020

View reviewed changes

comments

4754c9b

blakerouse approved these changes Aug 3, 2020

View reviewed changes

ruflin mentioned this pull request Aug 3, 2020

Add datastream fields to all datasets elastic/integrations#213

Merged

michalpristas added 4 commits August 4, 2020 10:54

Merge branch 'master' of github.com:elastic/beats into agent-datastream

3a05cc6

Merge branch 'master' of github.com:elastic/beats into agent-datastream

b4fb88d

updated spelling

65c1452

removed sending of dataset

2d66356

ruflin approved these changes Aug 5, 2020

View reviewed changes

michalpristas merged commit fc0e192 into elastic:master Aug 5, 2020

michalpristas added a commit to michalpristas/beats that referenced this pull request Aug 5, 2020

[Ingest Manager] Send datastreams fields (elastic#20402)

0ac46ce

[Ingest Manager] Send datastreams fields (elastic#20402)

michalpristas mentioned this pull request Aug 5, 2020

Cherry-pick #20402 to 7.x: Send datastreams fields #20442

Merged

6 tasks

michalpristas added a commit that referenced this pull request Aug 5, 2020

[Ingest Manager] Send datastreams fields (#20402) (#20442)

a3b2d53

[Ingest Manager] Send datastreams fields (#20402)

melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020

[Ingest Manager] Send datastreams fields (elastic#20402)

bc90153

[Ingest Manager] Send datastreams fields (elastic#20402)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Send datastreams fields #20402

[Ingest Manager] Send datastreams fields #20402

michalpristas commented Aug 3, 2020 •

edited

Loading

elasticmachine commented Aug 3, 2020

blakerouse left a comment

blakerouse Aug 3, 2020

michalpristas Aug 3, 2020

blakerouse Aug 3, 2020

michalpristas Aug 3, 2020

elasticmachine commented Aug 3, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

blakerouse left a comment

ruflin commented Aug 3, 2020

ruflin commented Aug 3, 2020

[Ingest Manager] Send datastreams fields #20402

[Ingest Manager] Send datastreams fields #20402

Conversation

michalpristas commented Aug 3, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Aug 3, 2020

blakerouse left a comment

Choose a reason for hiding this comment

blakerouse Aug 3, 2020

Choose a reason for hiding this comment

michalpristas Aug 3, 2020

Choose a reason for hiding this comment

blakerouse Aug 3, 2020

Choose a reason for hiding this comment

michalpristas Aug 3, 2020

Choose a reason for hiding this comment

elasticmachine commented Aug 3, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

blakerouse left a comment

Choose a reason for hiding this comment

ruflin commented Aug 3, 2020

ruflin commented Aug 3, 2020

michalpristas commented Aug 3, 2020 •

edited

Loading

elasticmachine commented Aug 3, 2020 •

edited by jenkins-beats-ci bot

Loading