[RFC] Label fields for additional types - Stage 0

[ebeahan]

Summary

Stage 0 proposal for adding nested fields to the labels object:

• labels.* fields will remain type keyword
• labels.long.* fields will be type long
• labels.double.* fields will be type double
• labels.boolean.* fields will be type boolean

Stage 0 (Strawperson) Criteria:

• Discuss with domain or subject matter experts the utility of these changes
• Discuss with the ECS team whether these changes seem appropriate for ECS

Preview of markdown proposal

[axw]

• A flattened object will index all leaf values to keyword. Numeric flattened field support is not yet available in Elasticsearch (elastic/elasticsearch#61550)

• Kibana has limited autocompletion and KQL support for flattened fields (elastic/kibana#25820).

Is waiting for Kibana support an option? I would prefer to change labels to flattened if it's an option.

Numeric labels are very similar to a use case that APM and Metrics has (dotted metric names where field names are not known ahead of time), described in elastic/elasticsearch#63530. The last comment on that issue suggests that `flattened will also be the answer, but I believe the Elasticsearch team are still investigating other options.

I would kinda prefer not to encode the type in the field name. In APM we're planning to take advantage of elastic/elasticsearch#69948 to dynamically set the field type without affecting field names. APM will install a set of well-known dynamic templates (e.g. "latency_histogram"), and will then specify that when indexing metrics whose names and types are not known ahead of time. Maybe we could formalise this in ECS?

[ebeahan]

Is waiting for Kibana support an option? I would prefer to change labels to flattened if it's an option.

I considered if switching the type to flattened for labels as an upcoming breaking change in the next ECS major could make sense. I hoped to switch the type to flattened with only keyword support initially and then gain support for numerics once added in ES without having to make update the type for labels.

After better realizing the current Kibana limitations with flattened, I think it's better to wait until the Kibana support improves.

In APM we're planning to take advantage of elastic/elasticsearch#69948 to dynamically set the field type without affecting field names. APM will install a set of well-known dynamic templates (e.g. "latency_histogram"), and will then specify that when indexing metrics whose names and types are not known ahead of time. Maybe we could formalise this in ECS?

Nice, thanks for linking! I hadn't seen that feature yet.

I do like the idea of standardizing the match_mapping_types naming within ECS.

[ebeahan]

I'm going to pause work on this proposal for now and close the PR.

I still think these two items are worth additional discussion, but I'd rather have a focused discussion on each separately.

• defined dynamic templates pairing with match_mapping_types
• Migrating labels to use flattened, as the flattened type keeps evolving

With some of these other initiatives, what's proposed here doesn't feel like the best direction right now.