Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix column name with special character when create materialized view #98

Merged

Conversation

dai-chen
Copy link
Collaborator

@dai-chen dai-chen commented Oct 24, 2023

Description

Spark StructType.fromDDL() requires all columns quoted if any special character. Previously all table/index/column names after Flint parser has been unquoted. Quote them before calling fromDDL method and finally it is unquoted in OpenSearch doc as before.

Reproducing the bug

CREATE MATERIALIZED VIEW test
AS
SELECT startTime AS `start.time`
...

start.time string not null,count long not null
----^^^

at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseTableSchema(ParseDriver.scala:76)
at org.apache.spark.sql.types.StructType$.fromDDL(StructType.scala:543)
at org.opensearch.flint.spark.FlintSparkIndex$.generateSchemaJSON(FlintSparkIndex.scala:143
at org.opensearch.flint.spark.mv.FlintSparkMaterializedView.metadata(FlintSparkMaterializedView.scala:58)
   ...

Issues Resolved

#84

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
@dai-chen dai-chen added the bug Something isn't working label Oct 24, 2023
@dai-chen dai-chen self-assigned this Oct 24, 2023
@dai-chen dai-chen marked this pull request as ready for review October 25, 2023 00:34
@dai-chen dai-chen merged commit 62c11df into opensearch-project:main Oct 25, 2023
4 checks passed
@dai-chen dai-chen deleted the fix-quoted-identifier-issue branch October 25, 2023 01:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants