Add default composable templates for new indexing strategy

[dakrone]

This commit adds the component and composable templates, as well as ILM policies, for the new
default indexing strategy. It installs:

• logs-default-mappings (component)
• logs-default-settings (component)
• logs-default-policy (ilm policy)
• logs-default-template (composable template)
• metrics-default-mappings (component)
• metrics-default-settings (component)
• metrics-default-policy (ilm policy)
• metrics-default-template (composable template)

These templates and policies are managed by a new x-pack module, stack, and can be disabled by
setting stack.templates.enabled to false.

These ensure that patterns for the logs-*-* and metrics-*-* indices are set up to create data
streams with the proper mappings and settings.

This also makes changes to the IndexTemplateRegistry to support installing component and
composable templates (previously it supported only legacy templates).

Resolves #56709

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Indices APIs)

[dakrone]

@ph @ruflin I requested reviews for you since this uses the templates/policies from Fleet, can you ensure that these are in fact the correct mappings, settings, and policies that we want to use?

[dakrone]

@gwbrown pinging you also since you were interested in how I changed IndexTemplateRegistry to work for composable and component templates

[dakrone]

@elasticmachine update branch

[ruflin]

@dakrone Left quite a few comments on the templates. These should not block the PR itself as we can still follow up making adjustments to the templates.

One thing I was wondering about is on what happens if Elasticsearch is updated and all the assets are already there. Will ES update some of them or leave them as is?

[jpountz]

I think we should discuss the implications of making changes to these Elasticsearch default templates. ECS is not set in stone, there are changes that happen from time to time, e.g. a field gets renamed. How would it play with this approach? If the field gets renamed when Elasticsearch upgrades, then wouldn't it fail existing shippers that send documents with the old field name rather than the new one (since our instructions recommend upgrading Elasticsearch first)?

[jpountz]

So @ruflin just explained to me how our packages are going to deal with backward compatibility by having ingest pipelines that will reformat documents that use an old format (e.g. think of a renamed field). Should we introduce an index pipeline here too? It would be initially empty, but this might be important to reserve namespace? Or alternatively should we make the mappings minimal so that we could realistically expect to never have to change the default mappings, e.g. only define a @timestamp field?

[dakrone]

@elasticmachine update branch

[dakrone]

One thing I was wondering about is on what happens if Elasticsearch is updated and all the assets are already there. Will ES update some of them or leave them as is?

ES uses the version in the template and the version of the stack module to determine whether the templates need to be updated, if the stack module were updated (for instance, we made changes to the templates), then it replaces them if the existing version is null or < STACK_VERSION

[dakrone]

Should we introduce an index pipeline here too? It would be initially empty, but this might be important to reserve namespace?

I'm not sure we should define the namespace yet, since we don't know whether the pipeline would be generic (both metrics & logs) or specific, so maybe we could leave it until we actually need it?

Or alternatively should we make the mappings minimal so that we could realistically expect to never have to change the default mappings, e.g. only define a @timestamp field?

Interesting thought, I'll add this to the "to discuss" list

@ruflin @jpountz thanks for taking a look, I wrote down a bunch of things to be discussed and I'll put together a document and time to discuss them (and incorporate any other feedback in this PR there)

[ph]

@jsvd @acchen97 As discussed in a previous document this is the default templates we would like to ship with ES that follow the indexing strategy document.

[ruflin]

@dakrone Why is this set to 100? Shouldn't we set it to 1? @jonathan-buttner FYI

[dakrone]

@ruflin hah, you already asked that in #57629 (comment) :) We need space in order for templates to be allowed at a lower priority than these built-in ones (for example, someone installing a template for * at priority 10)

[ruflin]

🤦

[jtibshirani]

Apologies for the very late question -- I was wondering why 'mappings' and 'settings' are pulled out into separate components, as opposed to just being defined on the template itself? Do we anticipate that users will reuse these components in their own templates?

Also is there a reason the settings and mappings live in different components? I would guess that you would reuse both the mappings and settings together.

[dakrone]

@jtibshirani I pulled them out to follow the behavior that they were currently in for the ingest manager, maybe @ph or @ruflin can comment about why they were separate?

On the other hand, it is nice to be able to substitute your own mappings for logs and still re-use the settings for the bundled templates, for example someone could use: composed_of: [logs-mappings, my-custom-mappings, logs-settings] where they have more control over the fine-grained merge order when these are separated. I think users may still want to use the settings (for example, lifecycle policy) without necessarily using all the mappings?

[ruflin]

@jtibshirani The use case mentioned by @dakrone is exactly why mappings and settings are separate. We expect users to reused the same settings and we also do it by default. The packages we install use the "gobal" settings by default as a component.