Add RFC for Presto -Native TPC-DS Connector #28
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pdabre12
force-pushed
the
native-tpcds-connector
branch
2 times, most recently
from
cf621f2
to
353cbed
Compare
pdabre12
force-pushed
the
native-tpcds-connector
branch
from
353cbed
to
de2d633
Compare
pdabre12
changed the title
| ### [Optional] Goals | ||
|
|
||
| 1. Add a TPC-DS connector to generate TPC-DS data in Presto native. | ||
| 2. Write end-to-end tests in Presto native with TPC-DS tables and conduct microbenchmarks in Velox. |
There was a problem hiding this comment.
We can skip micro-benchmarks in Velox here.
|
|
||
| ## Background | ||
|
|
||
| Currently , Presto does not have a native implementation of the TPC-DS connector. This RFC proposes the addition of a new TPC-DS connector. The new connector can be used as a Presto - Native catalog. |
There was a problem hiding this comment.
Give more background of TPC-DS benchmark, schema and dsdgen program here.
|
|
||
| The Presto - Native TPC-DS connector will be a wrapper for the generator distributed (dsdgen) by the TPC organization from C. This means we need our implementation to have the exact same behavior as the C implementation. DuckDB already has a TPC-DS connector of their own and they have wrapped the C files into C++ files, we are going to use these C++ files in our implementation. | ||
|
|
||
| In the C++ implementation, there are two types of tables: source tables and target tables used for generation. Source table files are prefixed with "s_", while target table files are prefixed with "w_". For instance, there may be files like "s_call_center.c" and "w_call_center.c". It appears that source tables are only utilized when running the "dsdgen" with an update flag, though the exact function of this flag and the purpose of the source tables have not yet been explored. Currently, our focus is solely on implementing functionalities for the target tables (w_ tables). |
There was a problem hiding this comment.
Doesn't Ying want to generate the update data with TPC-DS connector as well ? Might be good to add more information about it here.
|
|
||
| In the C++ implementation, there are two types of tables: source tables and target tables used for generation. Source table files are prefixed with "s_", while target table files are prefixed with "w_". For instance, there may be files like "s_call_center.c" and "w_call_center.c". It appears that source tables are only utilized when running the "dsdgen" with an update flag, though the exact function of this flag and the purpose of the source tables have not yet been explored. Currently, our focus is solely on implementing functionalities for the target tables (w_ tables). | ||
|
|
||
| In the target table files prefixed with “w_”, there are some helper functions(need to be implemented by us) precisely called as “append_row_start“ and “append_row_end“ which help in the row generation. Depending on the schema of the table, there will be “append_ “ functions depending on the data type to be appended. |
There was a problem hiding this comment.
Will be great to give more information about TpcdsSplits and how data generation happens for a table from one split to the next.
|
|
||
| In the target table files prefixed with “w_”, there are some helper functions(need to be implemented by us) precisely called as “append_row_start“ and “append_row_end“ which help in the row generation. Depending on the schema of the table, there will be “append_ “ functions depending on the data type to be appended. | ||
|
|
||
| A new TPC-DS config `tpcds.toggle-char-to-varchar` will be added to toggle the char columns to varchar, addressing the lack of support for the char data type in Presto - Native. This config allows the toggling of the char to varchar when required, ensuring consistency between Presto - Java and Presto - Native. |
There was a problem hiding this comment.
What is the impact of this at the schema level and at the data level ? Please can you elaborate.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.