Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(query): add topn runtime filter in native storage format #9738

Merged
merged 10 commits into from
Jan 27, 2023

Conversation

sundy-li
Copy link
Member

@sundy-li sundy-li commented Jan 26, 2023

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Summary about this PR

If we push down the sort + limit to storage level, we can have TopN opt.

But it must work with runtime predication (prewhere) + Page index optimization , so currently we only have it in native storage format.

Query select * from hits_page where URL like '%google%' order by EventTime desc limit 10; can works 5 times faster.

@vercel
Copy link

vercel bot commented Jan 26, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated
databend ⬜️ Ignored (Inspect) Jan 27, 2023 at 6:54AM (UTC)

@mergify mergify bot added the pr-feature this PR introduces a new feature to the codebase label Jan 26, 2023
@BohuTANG
Copy link
Member

BohuTANG commented Jan 27, 2023

Good idea.
Can we use OrderBy(min-max)+ Limit in reading fuse block data to reduce the block size?

@sundy-li
Copy link
Member Author

Good idea. Can we use OrderBy(min-max)+ Limit in reading fuse block data to reduce the block size?

Unless we have prewhere and page index.

@sundy-li sundy-li marked this pull request as ready for review January 27, 2023 03:24
Co-authored-by: BohuTANG <overred.shuttler@gmail.com>
@BohuTANG BohuTANG merged commit 3ea04f2 into datafuselabs:main Jan 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants