[RFC] Suggest/Infer Mapping Based on Inspection

[macohen]

Summary

When building a product or full-text search based application in OpenSearch, careful consideration and thought needs to be put into building the index based on the data being ingested. Often, a document to be indexed is derived from several different sources (e.g. databases, tables, or files). In OpenSearch, the best practice is to denormalize this data into a single document. This proposal is to build tooling that inspects source data and generates a suggested OpenSearch index mapping file so that search application builders do not need to start from scratch and also do not need to default to dynamic mapping.

General Use Case

1. Upload a CSV, JSON document, or SQL query against a source
2. Optionally specify a primary or unique key
3. Mapping file generator then:
   4. sorts the data based on the key provided
   5. loops through X number of rows to generate a suggestion for mapping file including nested documents, datatypes,
   analyzers, and autosuggest.
4. Search Application Builder can take the mapping file and tweak/correct the mappings.

Questions

1. Is there an existing component, like data-prepper, that can be used to support this case?
2. Many organizations reuse datatypes, so we would want to learn from existing mappings. How can this be done?