PUT index requests with mappings objects always blindly excise

[sanathbhat]

What is the bug?

PUT index requests with a mappings object are always indiscriminately subject to excision of the top level object within the mappings object.

How can one reproduce the bug?

With traffic capture and replayer modules running, make a put index request to source cluster via capture proxy server with the following sample body:

{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "field1": {
        "type": "text"
      },
      "field2": {
        "type": "keyword"
      }
    }
  }
}

This would create the index on the source cluster as expected but the target cluster returns the following error:

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters: [field1 : {type=text}] [field2 : {type=keyword}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [field1 : {type=text}] [field2 : {type=keyword}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters: [field1 : {type=text}] [field2 : {type=keyword}]"}},"status":400}

On checking the logs, it can be seen that the target cluster received the request body shown in Target request: section below, instead of the original, where the properties object has been excised and its contents directly pushed into its parent mappings. The likely cause of this can be traced to these lines of code: https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/transform/JsonTypeMappingTransformer.java#L77-L87

What is the expected behavior?

The above behavior was probably targeted at some open source version compatibility issue as a proof-of-concept solution but causes the replayer module to indiscriminately modify all put index request bodies making some of them erroneous and thereby failing to create indices on the target cluster. The expected behavior is that the index should be correctly created in both the source and target clusters wherein the replayer module carefully performs any request body transformations in an intelligent way only if required.

What is your host/environment?

Source/target cluster platforms: OCI
Source cluster version: OS 2.3
Target cluster version: OS 2.8
Oracle Linux host running capture proxy and replayer modules

Do you have any screenshots?

Decoded source/target request-response bodies from above screenshot:
Source request:

{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "field1": {
        "type": "text"
      },
      "field2": {
        "type": "keyword"
      }
    }
  }
}

Source response:

{"acknowledged":true,"shards_acknowledged":true,"index":"test-index-new-2"}

Target request:

{"settings":{"number_of_shards":1,"number_of_replicas":1},"mappings":{"field1":{"type":"text"},"field2":{"type":"keyword"}}}

Target response:

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [field1 : {type=text}] [field2 : {type=keyword}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [field1 : {type=text}] [field2 : {type=keyword}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [field1 : {type=text}] [field2 : {type=keyword}]"}},"status":400}

Do you have any additional context?

N/A

[gregschohn]

Thanks for the details!
You’re correct that the transformation support was more for POC. How we manage transformations need to be reconsidered.

We have this task on our board to make the transformer load via ServiceLoader instead of indiscriminately running the same one (or shipping a catchall transformation) for every installation.

In the meantime, removing the JsonTypeMappingTransformer() from this line entirely will suppress the changes that you’re noticing. The sooner that that goes out, the sooner we’ll start to find out what transformations we do need (other than host and auth transformations) and those findings would help us figure out how to manage transformations more intelligently.

[sumobrian]

@sanathbhat, for your continued testing, I suggest removing the JsonTypeMappingTransformer() and in the meantime, we would welcome your feedback on developing a transformation framework focused on creating extensible transformations as discussed in the Jira task referenced above (https://opensearch.atlassian.net/browse/MIGRATIONS-1378). This is targeted for Q1, 2024.

[sumobrian]

Issue confirmed on ES 7.10.2 to OS 2.7 migration. Replayer output:

{
  "targetResponse": {
    "access-control-allow-origin": "*",
    "Connection": "keep-alive",
    "response_time_ms": 16,
    "HTTP-Version": "HTTP/1.1",
    **"Status-Code": "400",**
    "Content-Length": "528",
    "body": {
      "error": {
        "root_cause": [
          {
            "type": "mapper_parsing_exception",
            "reason": "Root mapping definition has unsupported parameters:  [field1 : {type=text}] [field2 : {type=keyword}]"
          }
        ],
        "type": "mapper_parsing_exception",
        "reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [field1 : {type=text}] [field2 : {type=keyword}]",
        "caused_by": {
          "type": "mapper_parsing_exception",
          "reason": "Root mapping definition has unsupported parameters:  [field1 : {type=text}] [field2 : {type=keyword}]"
        }
      },
      "status": 400
    },
    "Reason-Phrase": "Bad Request",
    "Date": "Tue, 24 Oct 2023 04:29:44 GMT",
    "Content-Type": "application/json; charset=UTF-8"
  },
  "sourceRequest": {
    "Authorization": "*****",
    "Accept": "*/*",
    "User-Agent": "curl/7.81.0",
    "Request-URI": "/bjp-test",
    "Host": "capture-proxy-es:9200",
    "Method": "PUT",
    "HTTP-Version": "HTTP/1.1",
    "Content-Length": "227",
    "body": {
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 1
      },
      "mappings": {
        "properties": {
          "field1": {
            "type": "text"
          },
          "field2": {
            "type": "keyword"
          }
        }
      }
    },
    "Content-Type": "application/json"
  },
                                                                                                                                                                          160175,13     99%
          "type": "mapper_parsing_exception",
          "reason": "Root mapping definition has unsupported parameters:  [field1 : {type=text}] [field2 : {type=keyword}]"
        }
      },
      "status": 400
    },
    "Reason-Phrase": "Bad Request",
    "Date": "Tue, 24 Oct 2023 04:29:44 GMT",
    "Content-Type": "application/json; charset=UTF-8"
  },
  "sourceRequest": {
    "Authorization": "*****",
    "Accept": "*/*",
    "User-Agent": "curl/7.81.0",
    "Request-URI": "/bjp-test",
    "Host": "capture-proxy-es:9200",
    "Method": "PUT",
    "HTTP-Version": "HTTP/1.1",
    "Content-Length": "227",
    "body": {
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 1
      },
      "mappings": {
        "properties": {
          "field1": {
            "type": "text"
          },
          "field2": {
            "type": "keyword"
          }
        }
      }
    },
    "Content-Type": "application/json"
  },
  "connectionId": "org.opensearch.migrations.replay.datatypes.PojoTrafficStreamKey@6dcab9d9.0",
  "targetRequest": {
    "Authorization": "*****",
    "Accept": "*/*",
    "User-Agent": "curl/7.81.0",
    "Request-URI": "/bjp-test",
    "Method": "PUT",
    "HTTP-Version": "HTTP/1.1",
    "Content-Length": "124",
    "body": "{\"settings\":{\"number_of_shards\":1,\"number_of_replicas\":1},\"mappings\":{\"field1\":{\"type\":\"text\"},\"field2\":{\"type\":\"keyword\"}}}",
    "Content-Type": "application/json"
  },
  "sourceResponse": {
    "content-length": "67",
    "content-type": "application/json; charset=UTF-8",
    "response_time_ms": 103,
    "HTTP-Version": "HTTP/1.1",
    "Status-Code": "200",
    "body": {
      "acknowledged": true,
      "shards_acknowledged": true,
      "index": "bjp-test"
    },
    "Reason-Phrase": "OK"
  }
}

[gregschohn]

mainline now has a flag for --transformer-config. By default now, when that flag is not set, no transformations will be performed. We're actively working on building out some sample transforms now. For more details on how to use them, including how to supply your own scripts using some scripting languages, please see the TrafficReplayer README.

Since the root cause of this issue was that the replayer applied a transformation when there should have been none and that's now the default behavior, I'm marking this as fixed. Thanks for the feedback & let us know what else you find!