Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: separate metadata from parquet's kv_metadata #1120

Merged
merged 49 commits into from
Aug 24, 2023
Merged

feat: separate metadata from parquet's kv_metadata #1120

merged 49 commits into from
Aug 24, 2023

Conversation

tanruixiang
Copy link
Member

@tanruixiang tanruixiang commented Aug 1, 2023 •
edited by jiacai2050
Loading

Rationale

Close #955

Detailed Changes

  • Introduce another independent file to store metadata of sst

Test Plan

  • UT
    • test_parquet_build_and_read tests write and read.
    • test_arrow_meta_data tests compatible with older version.
  • Manually
    • Upgrade from old deployments
    • Start a new deploy, and run tsbs

@tanruixiang tanruixiang changed the title feat: separate metadata from parquet's kv_metadata feat: separate metadata from parquet's kv_metadata (WIP) Aug 1, 2023
@tanruixiang tanruixiang changed the title feat: separate metadata from parquet's kv_metadata (WIP) feat: separate metadata from parquet's kv_metadata Aug 8, 2023
@tanruixiang tanruixiang marked this pull request as ready for review August 8, 2023 06:14
@tanruixiang tanruixiang requested a review from jiacai2050 August 8, 2023 09:12
jiacai2050
jiacai2050 reviewed Aug 9, 2023
View reviewed changes
analytic_engine/src/sst/meta_data/cache.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/meta_data/cache.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
analytic_engine/src/table/sst_util.rs Outdated Show resolved Hide resolved
jiacai2050
jiacai2050 reviewed Aug 10, 2023
View reviewed changes
analytic_engine/src/sst/parquet/writer.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/writer.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/writer.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
@jiacai2050 jiacai2050 mentioned this pull request Aug 23, 2023
Merged
jiacai2050
jiacai2050 approved these changes Aug 24, 2023
View reviewed changes
Copy link
Contributor

@jiacai2050 jiacai2050 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jiacai2050 jiacai2050 merged commit fc0f314 into apache:main Aug 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jiacai2050 jiacai2050 jiacai2050 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Decrease SST metadata size
2 participants
@tanruixiang @jiacai2050