Stop Kibana from generating unique X-Opaque-ID values when not received from the client.

[pgayvallet]

We've been historically generating unique x-opaque-id values for each Kibana request, and transmitting those values to elasticsearch every time a request is performed using a scoped client.

This was done with the assumption that this value had no semantics or no functional usage in elasticsearch. However, this assumption was wrong, as this opaqueId value is now used for some features such as deprecation logs deduplication.

We need to stop generating values for x-opaque-id / (also exposed as request.id) when the client initiating the request against Kibana does not provide one, and not propagate the value to elasticsearch.

Now that ES has support for the traceparent header (elastic/elasticsearch#74210), we can use the trace.id value as correlation id for Kibana features that require so.

Note that if this x-opaque-id/request.id value is supposed to be mostly internal to core, it's currently used by the audit logging as their correlation value between Kibana and Elasticsearch audit logs. We will need to adapt Kibana's audit logging to stop overriding trace.id with request.id and update their documentation to reflect the fact that the correlation value will now be trace.id

Also, for this traceparent header to be sent to elasticsearch, the APM agent needs to be enabled, as least in contextPropagationOnly mode (which is the default configuration). We need to figure out if this limitation is acceptable and if documenting it is good enough, or if we may want to consider this feature of the APM agent as an implementation detail and stop allowing customer to disable it.

More context in the original issue: #120124

cc @elastic/kibana-security

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

A draft PR doing most of the technical changes is available here: #120124

We still need a decision regarding

Also, for this traceparent header to be sent to elasticsearch, the APM agent needs to be enabled, as least in contextPropagationOnly mode (which is the default configuration). We need to figure out if this limitation is acceptable and if documenting it is good enough, or if we may want to consider this feature of the APM agent as an implementation detail and stop allowing customer to disable it.

We may want to perform load testing to determine the impact of the contextPropagationOnly compared to a fully disabled APM agent to decide if forcing it to always be enabled is acceptable.

We can probably tweak the load tests from elastic/kibana-load-testing#221 to check the perf impact of contextPropagationOnly

[lukeelmers]

We can probably tweak the load tests from elastic/kibana-load-testing#221 to check the perf impact of contextPropagationOnly

@dmlemeshko I know you've been looking into perf impacts of enabling APM, would you be able to help us assess the impact of this particular option?

[dmlemeshko]

@lukeelmers
Sorry for late response, I will definitely help as much as I can. To check the impact I need either a branch I can use for testing (and compare with main) or required Kibana configuration changes I need to apply.

Please just open an issue with details and will try to do it asap.

[pgayvallet]

@dmlemeshko I opened elastic/kibana-load-testing#245. Please tell me if you need more information

[pgayvallet]

@dmlemeshko did you find an opportunity to take a look at elastic/kibana-load-testing#245?

[pgayvallet]

I performed the benchmarking and posted the results here: #129585 (comment)

TLDR: there is a significant difference between APM being fully disabled and being enabled in contextPropagationOnly mode, which means forcing APM to be enabled in contextPropagationOnly doesn't seems like a viable option in the scope of this issue.

So I guess we're back to the initial plan to simply document that features relying on trace.id such as audit logging require APM to not be disabled to work.

@lukeelmers wdyt?

[jportner]

So I guess we're back to the initial plan to simply document that features relying on trace.id such as audit logging require APM to not be disabled to work.

@lukeelmers wdyt?

Since I was subscribed to this issue: I'll chime in on behalf of Platform Security and say I am fine with this 👍

[lukeelmers]

So I guess we're back to the initial plan to simply document that features relying on trace.id such as audit logging require APM to not be disabled to work.

@lukeelmers wdyt?

I'll chime in on behalf of Platform Security and say I am fine with this

++ Agreed, I think this is acceptable given these findings, as long as Platform Security is comfortable with it.

[pgayvallet]

So #123197 will need to update the audit logging documentation that was added in #123757 to inform that APM needs to be enabled at least for context propagation for the feature to function.