Remove the base package from the default packages #70437

ruflin · 2020-07-01T13:24:52Z

As the base assets will be shipped by ES directly, the base package is not needed anymore. elastic/elasticsearch#57629

In the future we might reintroduce it to update the installed assets.

As the base assets will be shipped by ES directly, the base package is not needed anymore. elastic/elasticsearch#57629 In the future we might reintroduce it to update the installed assets.

elasticmachine · 2020-07-01T13:24:55Z

Pinging @elastic/ingest-management (Team:Ingest Management)

ruflin · 2020-07-01T13:25:17Z

This PR should go green / be working as soon as the most recent snapshots are available from Elasticsearch.

ruflin · 2020-07-01T13:25:51Z

@jonathan-buttner For awareness

ph · 2020-07-01T14:33:33Z

@ruflin This would go in 7.x as well, 7.x PR is merged, I have added a reminder to myself to make sure its merged on monday

kibanamachine · 2020-07-01T15:15:01Z

💛 Build succeeded, but was flaky

continuous-integration/kibana-ci/pull-request
Commit: 4c559bc
Flaky suites:
- xpack-kibana-ciGroup6

Test Failures

Kibana Pipeline / kibana-xpack-agent / X-Pack API Integration Tests.x-pack/test/api_integration/apis/endpoint/resolver·ts.apis Endpoint plugin Resolver related events route endpoint events should not find any events

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] moving index [ilm-history-2-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [ilm-history-ilm-policy]
[00:00:00]         │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] moving index [ilm-history-2-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [ilm-history-ilm-policy]
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook
[00:12:44]           └-: Endpoint plugin
[00:12:44]             └-> "before all" hook
[00:12:44]             └-> "before all" hook
[00:12:49]               │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] updated role [fleet_enroll]
[00:12:49]               │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] updated user [fleet_enroll]
[00:12:51]             └-: Resolver
[00:12:51]               └-> "before all" hook
[00:12:51]               └-> "before all" hook
[00:12:51]                 │ info [endpoint/resolver/api_feature] Loading "mappings.json"
[00:12:51]                 │ info [endpoint/resolver/api_feature] Loading "data.json.gz"
[00:12:51]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [endgame-4.21.0-000001] creating index, cause [api], templates [], shards [5]/[0], mappings [_doc]
[00:12:51]                 │ info [endpoint/resolver/api_feature] Created index "endgame-4.21.0-000001"
[00:12:51]                 │ debg [endpoint/resolver/api_feature] "endgame-4.21.0-000001" settings {"index":{"lifecycle":{"name":"endgame_policy-4.21.0","rollover_alias":"endgame-4.21.0"},"mapping":{"ignore_malformed":"true","total_fields":{"limit":"10000"}},"number_of_replicas":"0","number_of_shards":"5","refresh_interval":"5s"}}
[00:12:51]                 │ info [endpoint/resolver/api_feature] Indexed 156 docs into "endgame-4.21.0-000001"
[00:12:51]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [logs-endpoint.events.process-default] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[00:12:51]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [.ds-logs-endpoint.alerts-default-000001] creating index, cause [initialize_data_stream], templates [logs-endpoint.alerts], shards [1]/[1], mappings [_doc]
[00:12:51]                 │ info [o.e.c.m.MetadataCreateDataStreamService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] adding data stream [logs-endpoint.alerts-default]
[00:12:51]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [.ds-logs-endpoint.alerts-default-000001/oaBpqwtTRg-Ds8k-qvB5Tw] update_mapping [_doc]
[00:12:51]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [logs-endpoint.events.process-default/PEYfwloKSGu_zCbABKODaQ] create_mapping
[00:12:51]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [logs-endpoint.events.process-default/PEYfwloKSGu_zCbABKODaQ] update_mapping [_doc]
[00:12:51]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [logs-endpoint.events.process-default/PEYfwloKSGu_zCbABKODaQ] update_mapping [_doc]
[00:12:51]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [logs-endpoint.events.process-default/PEYfwloKSGu_zCbABKODaQ] update_mapping [_doc]
[00:12:51]               └-: related events route
[00:12:51]                 └-> "before all" hook
[00:12:52]                 └-: endpoint events
[00:12:52]                   └-> "before all" hook
[00:12:52]                   └-> should not find any events
[00:12:52]                     └-> "before each" hook: global before each
[00:12:52]                     │ proc [kibana]   log   [14:26:49.439] [warning][plugins][resolver][securitySolution] { Error: [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.id] in order to load field data by uninverting the inverted index. Note that this can use significant memory.
[00:12:52]                     │ proc [kibana]     at respond (/dev/shm/workspace/install/kibana-7/node_modules/elasticsearch/src/lib/transport.js:349:15)
[00:12:52]                     │ proc [kibana]     at checkRespForFailure (/dev/shm/workspace/install/kibana-7/node_modules/elasticsearch/src/lib/transport.js:306:7)
[00:12:52]                     │ proc [kibana]     at HttpConnector.<anonymous> (/dev/shm/workspace/install/kibana-7/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
[00:12:52]                     │ proc [kibana]     at IncomingMessage.wrapper (/dev/shm/workspace/install/kibana-7/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
[00:12:52]                     │ proc [kibana]     at IncomingMessage.emit (events.js:203:15)
[00:12:52]                     │ proc [kibana]     at endReadableNT (_stream_readable.js:1145:12)
[00:12:52]                     │ proc [kibana]     at process._tickCallback (internal/process/next_tick.js:63:19)
[00:12:52]                     │ proc [kibana]   status: 400,
[00:12:52]                     │ proc [kibana]   displayName: 'BadRequest',
[00:12:52]                     │ proc [kibana]   message:
[00:12:52]                     │ proc [kibana]    '[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.id] in order to load field data by uninverting the inverted index. Note that this can use significant memory.',
[00:12:52]                     │ proc [kibana]   path: '/logs-endpoint.events.*/_search',
[00:12:52]                     │ proc [kibana]   query: {},
[00:12:52]                     │ proc [kibana]   body:
[00:12:52]                     │ proc [kibana]    { error:
[00:12:52]                     │ proc [kibana]       { root_cause: [Array],
[00:12:52]                     │ proc [kibana]         type: 'search_phase_execution_exception',
[00:12:52]                     │ proc [kibana]         reason: 'all shards failed',
[00:12:52]                     │ proc [kibana]         phase: 'query',
[00:12:52]                     │ proc [kibana]         grouped: true,
[00:12:52]                     │ proc [kibana]         failed_shards: [Array],
[00:12:52]                     │ proc [kibana]         caused_by: [Object] },
[00:12:52]                     │ proc [kibana]      status: 400 },
[00:12:52]                     │ proc [kibana]   statusCode: 400,
[00:12:52]                     │ proc [kibana]   response:
[00:12:52]                     │ proc [kibana]    '{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.id] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logs-endpoint.events.process-default","node":"DysfrGxpQ0a_q59DTj1HiQ","reason":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.id] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.id] in order to load field data by uninverting the inverted index. Note that this can use significant memory.","caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.id] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}},"status":400}',
[00:12:52]                     │ proc [kibana]   toString: [Function],
[00:12:52]                     │ proc [kibana]   toJSON: [Function] }
[00:12:52]                     └- ✖ fail: "apis Endpoint plugin Resolver related events route endpoint events should not find any events"
[00:12:52]                     │

Stack Trace

Error: expected 200 "OK", got 500 "Internal Server Error"
    at Test._assertStatus (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:268:12)
    at Test._assertFunction (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:283:11)
    at Test.assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:173:18)
    at assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:131:12)
    at /dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:128:5
    at Test.Request.callback (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/index.js:718:3)
    at parser (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/index.js:906:18)
    at IncomingMessage.res.on (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/parsers/json.js:19:7)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

Kibana Pipeline / kibana-xpack-agent / X-Pack API Integration Tests.x-pack/test/api_integration/apis/endpoint/resolver·ts.apis Endpoint plugin Resolver "after all" hook in "Resolver"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] moving index [ilm-history-2-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [ilm-history-ilm-policy]
[00:00:00]         │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] moving index [ilm-history-2-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [ilm-history-ilm-policy]
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook
[00:12:44]           └-: Endpoint plugin
[00:12:44]             └-> "before all" hook
[00:12:44]             └-> "before all" hook
[00:12:49]               │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] updated role [fleet_enroll]
[00:12:49]               │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] updated user [fleet_enroll]
[00:12:51]             └-: Resolver
[00:12:51]               └-> "before all" hook
[00:12:51]               └-> "before all" hook
[00:12:51]                 │ info [endpoint/resolver/api_feature] Loading "mappings.json"
[00:12:51]                 │ info [endpoint/resolver/api_feature] Loading "data.json.gz"
[00:12:51]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [endgame-4.21.0-000001] creating index, cause [api], templates [], shards [5]/[0], mappings [_doc]
[00:12:51]                 │ info [endpoint/resolver/api_feature] Created index "endgame-4.21.0-000001"
[00:12:51]                 │ debg [endpoint/resolver/api_feature] "endgame-4.21.0-000001" settings {"index":{"lifecycle":{"name":"endgame_policy-4.21.0","rollover_alias":"endgame-4.21.0"},"mapping":{"ignore_malformed":"true","total_fields":{"limit":"10000"}},"number_of_replicas":"0","number_of_shards":"5","refresh_interval":"5s"}}
[00:12:51]                 │ info [endpoint/resolver/api_feature] Indexed 156 docs into "endgame-4.21.0-000001"
[00:12:51]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [logs-endpoint.events.process-default] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[00:12:51]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [.ds-logs-endpoint.alerts-default-000001] creating index, cause [initialize_data_stream], templates [logs-endpoint.alerts], shards [1]/[1], mappings [_doc]
[00:12:51]                 │ info [o.e.c.m.MetadataCreateDataStreamService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] adding data stream [logs-endpoint.alerts-default]
[00:12:51]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [.ds-logs-endpoint.alerts-default-000001/oaBpqwtTRg-Ds8k-qvB5Tw] update_mapping [_doc]
[00:12:51]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [logs-endpoint.events.process-default/PEYfwloKSGu_zCbABKODaQ] create_mapping
[00:12:51]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [logs-endpoint.events.process-default/PEYfwloKSGu_zCbABKODaQ] update_mapping [_doc]
[00:12:51]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [logs-endpoint.events.process-default/PEYfwloKSGu_zCbABKODaQ] update_mapping [_doc]
[00:12:51]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-centos-tests-xl-1593609921143291950] [logs-endpoint.events.process-default/PEYfwloKSGu_zCbABKODaQ] update_mapping [_doc]
[00:12:52]               └-> "after all" hook
[00:12:52]                 └- ✖ fail: "apis Endpoint plugin Resolver "after all" hook in "Resolver""
[00:12:52]                 │

Stack Trace

{ ResponseError: resource_not_found_exception
    at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'ResponseError',
  meta:
   { body: { error: [Object], status: 404 },
     statusCode: 404,
     headers:
      { 'content-type': 'application/json; charset=UTF-8',
        'content-length': '281' },
     warnings: null,
     meta:
      { context: null,
        request: [Object],
        name: 'elasticsearch-js',
        connection: [Object],
        attempts: 0,
        aborted: false } } }

Build metrics

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

ruflin · 2020-07-02T11:13:55Z

@elasticmachine merge upstream

jonathan-buttner · 2020-07-02T13:14:38Z

💛 Build succeeded, but was flaky

continuous-integration/kibana-ci/pull-request

Commit: 4c559bc

Flaky suites:

xpack-kibana-ciGroup6

Test Failures

Kibana Pipeline / kibana-xpack-agent / X-Pack API Integration Tests.x-pack/test/api_integration/apis/endpoint/resolver·ts.apis Endpoint plugin Resolver related events route endpoint events should not find any events
Kibana Pipeline / kibana-xpack-agent / X-Pack API Integration Tests.x-pack/test/api_integration/apis/endpoint/resolver·ts.apis Endpoint plugin Resolver "after all" hook in "Resolver"

Build metrics

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Well that test failure is discouraging 🤔 removing the base package shouldn't have anything to do with that... I'll look into it

ruflin · 2020-07-02T13:42:42Z

@jonathan-buttner I'll assigned you as "Assignee" to this PR so you can follow up.

As the base assets will be shipped by ES directly, the base package is not needed anymore. elastic/elasticsearch#57629 In the future we might reintroduce it to update the installed assets. Co-authored-by: Nicolas Ruflin <spam@ruflin.com>

Remove the base package from the default packages

4c559bc

As the base assets will be shipped by ES directly, the base package is not needed anymore. elastic/elasticsearch#57629 In the future we might reintroduce it to update the installed assets.

ruflin added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.9.0 Team:Fleet Team label for Observability Data Collection Fleet team Ingest Management:beta1 labels Jul 1, 2020

ruflin requested a review from a team July 1, 2020 13:24

ph approved these changes Jul 1, 2020

View reviewed changes

jonathan-buttner approved these changes Jul 2, 2020

View reviewed changes

ruflin assigned ruflin and jonathan-buttner and unassigned ruflin Jul 2, 2020

jonathan-buttner merged commit 5b8fb95 into elastic:master Jul 2, 2020

jonathan-buttner mentioned this pull request Jul 2, 2020

[7.x] Remove the base package from the default packages (#70437) #70645

Merged

ruflin deleted the remove-base-package branch July 2, 2020 21:09

skh mentioned this pull request Jul 9, 2020

[Ingest Manager] Disable base as a required package #69483

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove the base package from the default packages #70437

Remove the base package from the default packages #70437

ruflin commented Jul 1, 2020

elasticmachine commented Jul 1, 2020

ruflin commented Jul 1, 2020

ruflin commented Jul 1, 2020

ph commented Jul 1, 2020

kibanamachine commented Jul 1, 2020

Standard Out

Stack Trace

Standard Out

Stack Trace

ruflin commented Jul 2, 2020

jonathan-buttner commented Jul 2, 2020 •

edited

Loading

💛 Build succeeded, but was flaky

Test Failures

Build metrics

ruflin commented Jul 2, 2020

Remove the base package from the default packages #70437

Remove the base package from the default packages #70437

Conversation

ruflin commented Jul 1, 2020

elasticmachine commented Jul 1, 2020

ruflin commented Jul 1, 2020

ruflin commented Jul 1, 2020

ph commented Jul 1, 2020

kibanamachine commented Jul 1, 2020

💛 Build succeeded, but was flaky

Test Failures

Standard Out

Stack Trace

Standard Out

Stack Trace

Build metrics

ruflin commented Jul 2, 2020

jonathan-buttner commented Jul 2, 2020 • edited Loading

💛 Build succeeded, but was flaky

Test Failures

Build metrics

ruflin commented Jul 2, 2020

jonathan-buttner commented Jul 2, 2020 •

edited

Loading