[EPIC] Zero-ETL - OpenSearch Table

[penghuo]

This is meta issue to track the tasks of opensearch-project/OpenSearch#14524

Milestone I - Direct access OpenSearch index in Spark SQL

In milestone I, we want to achieve (1) Support direct query opensearch index use case. (2) Improve performance. #334. The task includes

• OpenSearch domain as Catalog in Spark #395
• OpenSearch index shard based parallel processing #396
• Support search_after on read and deprecated scroll. #494
• Enhancement - Describe OpenSearch table should return partition info #529
• Support read OpenSearch using PIT #430
• OpenSearch DataType map to Spark DataType #505
• Performance Test - Evaluate OpenSearch read performance #403
• OpenSearch DataType extension in Spark - UDT
• OpenSearch Function extension in Spark - UDF
• Blog - OpenSearch Table in Spark

[dai-chen]

Another requirement is to support writing to an OpenSearch table, allowing users to use the INSERT statement. This would enable users to control when and what data is loaded into the OpenSearch index.

For example:

# First batch
INSERT INTO os_table SELECT ... FROM logs WHERE id BETWEEN 1 AND 100;

# Second batch
INSERT INTO os_table SELECT ... FROM logs WHERE id BETWEEN 101 AND 200;

This is not a use case for Flint's materialized view feature. Currently there is no way to achieve in Flint Spark.