Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: memtable bitmap for null #1046

Merged
merged 7 commits into from
Jul 6, 2023

Conversation

ShiKaiWi
Copy link
Member

@ShiKaiWi ShiKaiWi commented Jul 4, 2023

Rationale

For a table which contains many nullable columns, many nulls may cause huge memory consumption in the memtable, leading to small ssts where null values will be compressed.

Detailed Changes

  • Introduce a bitset for continuous row encoding
  • Use u32 to encode the offest in the encoded bytes rather than u64

Test Plan

Existing unit test. And test it with tsbs's data, here is result with write buffer 32MB:

description version encoded size of one row number of rows in memtable sst's size
20 col (not null) main 325B 70000+ 1.8M
20 col (not null) current 295B 78000 2M
20 col + 200 null col main 2135B 12100 460K
20 col + 200 null col current 332B 70000 1.5M

common_types/src/bitset.rs Outdated Show resolved Hide resolved
common_types/src/row/contiguous.rs Outdated Show resolved Hide resolved
@ShiKaiWi ShiKaiWi force-pushed the feat-memtable-bitmap-for-null branch from 1446144 to 92ce7a0 Compare July 5, 2023 08:51
common_types/src/bitset.rs Show resolved Hide resolved
common_types/src/row/contiguous.rs Outdated Show resolved Hide resolved
common_types/src/row/contiguous.rs Outdated Show resolved Hide resolved
common_types/src/bitset.rs Show resolved Hide resolved
common_types/src/row/contiguous.rs Show resolved Hide resolved
common_types/src/row/contiguous.rs Outdated Show resolved Hide resolved
common_types/src/bitset.rs Show resolved Hide resolved
@ShiKaiWi ShiKaiWi force-pushed the feat-memtable-bitmap-for-null branch from f19988d to ba9fd02 Compare July 6, 2023 10:40
Copy link
Contributor

@jiacai2050 jiacai2050 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jiacai2050 jiacai2050 merged commit c6c306d into apache:main Jul 6, 2023
@ShiKaiWi ShiKaiWi deleted the feat-memtable-bitmap-for-null branch July 6, 2023 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants