Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: impl layered memtable to reduce duplicated encode during scan #1271

Merged
merged 16 commits into from
Jan 8, 2024

Conversation

Rachelint
Copy link
Contributor

@Rachelint Rachelint commented Oct 19, 2023

Rationale

Conversion from row format in memtable to record batch in datafusion has been found a cpu bottleneck in production. For reduce the cpu cost, I impl the layered memtable framework to support gradually conversion during normal write path(and before flush).

Detailed Changes

  • Impl layered memtable framework
  • Integrate it into the write path.

Test Plan

Test by new ut and it.

@Rachelint Rachelint changed the title feat: impl lazyer memtable to reduce duplicated encode during scan feat: impl lazyered memtable to reduce duplicated encode during scan Oct 19, 2023
@Rachelint Rachelint changed the title feat: impl lazyered memtable to reduce duplicated encode during scan feat: impl layered memtable to reduce duplicated encode during scan Oct 19, 2023
@Rachelint Rachelint force-pushed the optimized-memtable branch 2 times, most recently from 05debba to 9a165a5 Compare October 20, 2023 01:54
@ShiKaiWi
Copy link
Member

@Rachelint Please fill in the content of the PR description.

Copy link
Member

@ShiKaiWi ShiKaiWi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Rachelint Some basic unit tests are needed. And the comments needs polishing.

analytic_engine/src/memtable/layered/mod.rs Outdated Show resolved Hide resolved
analytic_engine/src/memtable/layered/mod.rs Outdated Show resolved Hide resolved
analytic_engine/src/memtable/layered/mod.rs Outdated Show resolved Hide resolved
analytic_engine/src/memtable/layered/mod.rs Show resolved Hide resolved
@Rachelint Rachelint force-pushed the optimized-memtable branch 2 times, most recently from 74849fc to 711feab Compare December 29, 2023 09:18
@Rachelint Rachelint changed the base branch from main to dev December 29, 2023 09:19
@Rachelint Rachelint marked this pull request as ready for review December 29, 2023 09:50
@Rachelint Rachelint force-pushed the optimized-memtable branch 2 times, most recently from 5d4ba1e to afe28ce Compare January 5, 2024 03:31
analytic_engine/src/lib.rs Outdated Show resolved Hide resolved
analytic_engine/src/memtable/layered/iter.rs Show resolved Hide resolved
analytic_engine/src/memtable/layered/mod.rs Outdated Show resolved Hide resolved
analytic_engine/src/memtable/layered/mod.rs Outdated Show resolved Hide resolved
analytic_engine/src/memtable/layered/mod.rs Outdated Show resolved Hide resolved
analytic_engine/src/memtable/test_util.rs Outdated Show resolved Hide resolved
analytic_engine/src/memtable/mod.rs Show resolved Hide resolved
analytic_engine/src/table/data.rs Outdated Show resolved Hide resolved
@ShiKaiWi ShiKaiWi merged commit 53df95a into apache:dev Jan 8, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants