Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support columar encoding for datums #1158

Merged
merged 10 commits into from
Aug 21, 2023

Conversation

ShiKaiWi
Copy link
Member

@ShiKaiWi ShiKaiWi commented Aug 16, 2023

Rationale

The encoding format for wal log is not compact, which imposes too much overhead on the underlying wal service.

Detailed Changes

  • Introduce the columnar encoding framework sub-module in the codec module.
  • Provide some basic examples of implementation for ValuesEncoder/ValuesDecoder.

Test Plan

New unit tests.

Future works

Implement specific encoders for every data type.

@ShiKaiWi ShiKaiWi marked this pull request as draft August 16, 2023 10:26
@ShiKaiWi ShiKaiWi marked this pull request as ready for review August 17, 2023 03:36
components/codec/src/columnar/mod.rs Outdated Show resolved Hide resolved
components/codec/src/columnar/mod.rs Show resolved Hide resolved
components/codec/src/columnar/mod.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@jiacai2050 jiacai2050 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShiKaiWi ShiKaiWi merged commit 05d5d4f into apache:main Aug 21, 2023
5 checks passed
@ShiKaiWi ShiKaiWi deleted the improve-wal-log-size branch August 21, 2023 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants