Add support for exporting es apm data to otlp-collector

[LHB6540]

Component(s)

No response

Is your feature request related to a problem? Please describe.

My organization's original APM efforts were all based on Elastic APM, and when we tried to migrate to OTLP, some parts went very smoothly, such as the Java application, while other languages encountered problems, such as PHP, whose auto-detection required us to upgrade PHP Recent major version updates and code changes. After evaluation, we decided to retain Elastic APM, but our current visualization work has shifted to LOKI, Prometheus, and Grafana stacks. APM is left alone in Elastic APM, which makes our work experience feel more fragmented.

Describe the solution you'd like

Therefore, a good solution is to add support for Elastic APM to OTLP conversion in otlp-collector, so that we can collect data to Tempo, and all observability work will be completed in Grafana. I noticed that opentelemetry-collector can already convert OTLP to ES APM data, I'm not sure if this part of the job should be done by opentelemetry-collector, so I raised the same issues with ES APM. Sorry if this has caused you the wrong confusion.

Describe alternatives you've considered

One alternative that has been considered is to enable Grafana to parse the Trace data of ES APM. Grafana already has an Elasticsearch data source, but it does not support drawing as a Trace view. I have opened issues under Grafana's repository.
grafana/grafana#77907

Additional context

No response

[crobert-1]

Hello @LHB6540, I'm having a bit of a hard time understanding your use case and request. It looks like your goal is to no longer use Elasticsearch for your APM data. Are you trying to extract existing traces from Elasticsearch? Is that the correct summary here? Am I missing something?

[LHB6540]

Hello @LHB6540, I'm having a bit of a hard time understanding your use case and request. It looks like your goal is to no longer use Elasticsearch for your APM data. Are you trying to extract existing traces from Elasticsearch? Is that the correct summary here? Am I missing something?

Yes, I plan not to use Elastic to process data anymore, but the workload of migrating the collector or code integration framework from Elastic Agent to otlp is huge. I hope otlp-collector can receive the data collected by Elastic Agent and then output it to a storage or processing system that supports otlp.

[crobert-1]

I have two suggestions here:

1. Add Elasticsearch's data format to the encoding extension. I assume this is the best option as it gives extra flexibility in using different components with this data format.
2. Add trace support to the Elasticsearch receiver. This may not work exactly though as the receiver currently is querying Elasticsearch for performance metrics, not actually getting data stored in the platform.

I'll defer to others at this point to give feedback on which would be preferable, or if there's another option I'm missing.

[github-actions]

Pinging code owners for receiver/elasticsearch: @djaglowski @BinaryFissionGames. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for extension/encoding: @atoulme @dao-jun @dmitryax @MovieStoreGuy @VihasMakwana. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/elasticsearch: @djaglowski @BinaryFissionGames
• extension/encoding: @atoulme @dao-jun @dmitryax @MovieStoreGuy @VihasMakwana

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/elasticsearch: @djaglowski @BinaryFissionGames
• extension/encoding: @atoulme @dao-jun @dmitryax @MovieStoreGuy @VihasMakwana

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.