[exporter/elasticsearch] Regression: Duplicate Key in JSON After #31694

[BenEfrati]

Component(s)

exporter/elasticsearch

What happened?

Description

Previously, the request to ELK contained a well-structured JSON object with unique keys:

{
  ...
  "host": {
    "name": "hostname"
  },
  "os": {
    "description": "Windows 11 10.0",
    "type": "windows"
  },
  ...
}

However, after the change, the JSON format has been modified, resulting in a duplicate host key:

{
  ...
  "host": {
    "hostname": "hostname",
    "os": {
      "full": "Windows 11 10.0",
      "platform": "windows"
    }
  },
  ...
  "host": {  // Duplicate key
    "os": {
      "type": "windows"
    }
  },
  ...
}

This duplicate key renders the JSON invalid, leading to parsing errors from ELK:

{
  "error": {
    "reason": "failed to parse",
    "type": "mapper_parsing_exception"
  },
  "status": 400
}

Steps to Reproduce

add the following resource attribute to OTLP request
host.name: Str(hostname)
os.description: Str(Windows 11 10.0)
os.type: Str(windows)

Expected Result

ELK successfully parses the JSON data and processes it without errors.

Actual Result

ELK encounters a parsing error due to the duplicate host key, and the data is not processed successfully.

Collector version

0.101.0

Environment information

Environment

Not relevant

OpenTelemetry Collector configuration

exporters:
  elasticsearch/logs:
    ...
    mapping:
      mode: ecs
    ....

Log output

2024-05-27T06:13:19.135Z	debug	elasticsearchexporter@v0.101.0/elasticsearch_bulk.go:45	Request roundtrip completed.	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch/logs", "path": "/_bulk", "method": "POST", "duration": 0.057272404, "status": "200 OK"}
2024-05-27T06:13:19.136294606Z 2024-05-27T06:13:19.136Z	error	elasticsearchexporter@v0.101.0/elasticsearch_bulk.go:318	Drop docs: failed to index: struct { Type string "json:\"type\""; Reason string "json:\"reason\"" }{Type:"mapper_parsing_exception", Reason:"failed to parse"}	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch/logs", "status": 400}
2024-05-27T06:13:19.136315906Z github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*worker).flush
2024-05-27T06:13:19.136322306Z 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.101.0/elasticsearch_bulk.go:318
2024-05-27T06:13:19.136327106Z github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*worker).run
2024-05-27T06:13:19.136332306Z 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.101.0/elasticsearch_bulk.go:304
2024-05-27T06:13:19.136337506Z github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newBulkIndexer.func1
2024-05-27T06:13:19.136342306Z 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.101.0/elasticsearch_bulk.go:220

Additional context

#31694 (comment)

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @ycombinator @carsonip

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Thanks for the detailed explanation and investigation, @BenEfrati!

Removing needs triage as this is a valid bug, and has a clear reference to where the bug occurs.