Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Incremental refresh index on Hive source table #91

Closed
dai-chen opened this issue Oct 23, 2023 · 3 comments
Closed

[FEATURE] Incremental refresh index on Hive source table #91

dai-chen opened this issue Oct 23, 2023 · 3 comments
Labels
enhancement New feature or request wontfix This will not be worked on

Comments

@dai-chen
Copy link
Collaborator

dai-chen commented Oct 23, 2023

Is your feature request related to a problem?

Currently Spark structured streaming only supports Spark Data Source table. For Hive table, it throws exception when starting streaming job.

What solution would you like?

Opt-1: Support streaming on Hive table either enhance stream source operator or convert Hive table internally (not sure if possible)
Opt-2: Give user clear message simply or how to create Spark DS table guide upon error. Related to #65

@dai-chen dai-chen added enhancement New feature or request untriaged labels Oct 23, 2023
@dai-chen dai-chen self-assigned this Oct 23, 2023
@dai-chen dai-chen removed their assignment Oct 31, 2023
@dai-chen dai-chen added the 0.2 label Nov 13, 2023
@dai-chen
Copy link
Collaborator Author

A new option is this may be supported by Lazy Build idea in #118 if the proposal feasible.

@dai-chen
Copy link
Collaborator Author

Need to double check if all or only certain Hive table cannot be supported by Spark structured streaming.

@penghuo
Copy link
Collaborator

penghuo commented Feb 15, 2024

no plan to support hive table as streaming source in short term. more context of hive table issue with spark structured streaming query in #65 (comment).

@penghuo penghuo removed the 0.2 label Feb 15, 2024
@dai-chen dai-chen added the wontfix This will not be worked on label Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request wontfix This will not be worked on
Development

No branches or pull requests

2 participants