Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix index name with uppercase letter issue #60

Merged

Conversation

dai-chen
Copy link
Collaborator

@dai-chen dai-chen commented Oct 4, 2023

Description

OpenSearch has restriction on index name and only allows lowercase letter. Because Flint index is not registered to Glue/Hive metastore, table name and index name given in create statement is encoded into OpenSearch index name.

Without the fix

SQL statement such as CREATE SKIPPING INDEX ON myGlue.default.alb_logs will fail due to capital G and flint_myGlue_default_alb_logs_skipping_index index name generated.

With the fix

For any API, do this lowercase conversion in low-level FlintOpenSearchClient since this is restriction in OpenSearch store.

Documentation

Updated user manual with example and the limitation caused by this:

  1. Naming convention: https://github.com/dai-chen/opensearch-spark/blob/fix-uppercase-index-name-issue/docs/index.md#opensearch
  2. Limitation: https://github.com/dai-chen/opensearch-spark/blob/fix-uppercase-index-name-issue/docs/index.md#flint-index-naming

Refactoring

Potential refactor for FlintOpenSearchClient code in future:

  # Current
  @Override public void createIndex(String indexName, FlintMetadata metadata) {
    String osIndexName = toLowercase(indexName);
    try (RestHighLevelClient client = createClient()) {
        ...

  # Approach 1 by lambda
    return performAction(indexName, (client, osIndexName) -> ... );

  # Approach 2 by fluent API
    return withFlintIndex(indexName)
        .perform((client, osIndexName) -> ... );

  # Approach 3 by AOP
  class SomeClientWrapperForAop
      public void createIndex(String indexName, FlintMetadata metadata) {
          client.createIndex(toLowercase(indexName), metadata)
      ...

Issues Resolved

N/A

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
@dai-chen dai-chen added the bug Something isn't working label Oct 4, 2023
@dai-chen dai-chen self-assigned this Oct 4, 2023
@dai-chen dai-chen merged commit d3f72e6 into opensearch-project:main Oct 4, 2023
4 checks passed
@dai-chen dai-chen deleted the fix-uppercase-index-name-issue branch October 4, 2023 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants