Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: write hybrid storage format #185

Merged
merged 15 commits into from
Aug 19, 2022
Merged

feat: write hybrid storage format #185

merged 15 commits into from
Aug 19, 2022

Conversation

jiacai2050
Copy link
Contributor

@jiacai2050 jiacai2050 commented Aug 10, 2022 •
edited
Loading

Which issue does this PR close?

part of #77

Rationale for this change

See issue above

What changes are included in this PR?

  • And a new table option stroage_format, read from env now
  • Write record as hybrid format, read will be supported in future PR.

Are there any user-facing changes?

How does this change test

Unit tests

@ShiKaiWi ShiKaiWi self-requested a review August 11, 2022 01:44
@waynexia waynexia self-requested a review August 12, 2022 03:31
@jiacai2050 jiacai2050 changed the title feat: add hybrid storage format convert feat: write hybrid storage format Aug 16, 2022
@jiacai2050 jiacai2050 marked this pull request as ready for review August 16, 2022 01:22
@jiacai2050 jiacai2050 force-pushed the feat-hybrid-format branch 2 times, most recently from d21da8b to 8df7357 Compare August 16, 2022 04:03
waynexia
waynexia reviewed Aug 16, 2022
View reviewed changes
analytic_engine/src/sst/parquet/encoding.rs Show resolved Hide resolved
common_types/src/schema.rs Show resolved Hide resolved
common_types/src/schema.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Show resolved Hide resolved
chunshao90
chunshao90 reviewed Aug 16, 2022
View reviewed changes
analytic_engine/src/sst/parquet/encoding.rs Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
@jiacai2050 jiacai2050 mentioned this pull request Aug 17, 2022
Closed
@jiacai2050 jiacai2050 force-pushed the feat-hybrid-format branch 3 times, most recently from 365dd17 to 97fa3e2 Compare August 18, 2022 09:09
waynexia
waynexia reviewed Aug 18, 2022
View reviewed changes
common_types/src/datum.rs Outdated Show resolved Hide resolved
common_types/src/datum.rs Show resolved Hide resolved
common_types/src/schema.rs Show resolved Hide resolved
common_types/src/schema.rs Outdated Show resolved Hide resolved
ShiKaiWi
ShiKaiWi reviewed Aug 18, 2022
View reviewed changes
analytic_engine/src/sst/builder.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/builder.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
common_types/src/schema.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
ShiKaiWi
ShiKaiWi reviewed Aug 18, 2022
View reviewed changes
analytic_engine/src/sst/parquet/reader.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
waynexia
waynexia reviewed Aug 18, 2022
View reviewed changes
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/encoding.rs Outdated Show resolved Hide resolved
@jiacai2050 jiacai2050 mentioned this pull request Aug 18, 2022
Merged
ShiKaiWi
ShiKaiWi reviewed Aug 18, 2022
View reviewed changes
analytic_engine/src/sst/parquet/hybrid.rs Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
analytic_engine/src/sst/parquet/hybrid.rs Outdated Show resolved Hide resolved
@waynexia waynexia added the A-analytic-engine Area: Analytic Engine label Aug 18, 2022
@jiacai2050 jiacai2050 force-pushed the feat-hybrid-format branch 2 times, most recently from 879495d to 93cfa2d Compare August 18, 2022 15:09
@waynexia waynexia merged commit a7c44db into apache:main Aug 19, 2022
@waynexia
Copy link
Member

🎉🎉

@jiacai2050 jiacai2050 mentioned this pull request Aug 19, 2022
Merged
@jiacai2050 jiacai2050 added this to the Release v0.3 milestone Aug 26, 2022
chunshao90 pushed a commit to chunshao90/ceresdb that referenced this pull request May 15, 2023
@jiacai2050
feat: write hybrid storage format (apache#185)
1cefda7 
* add convert_to_hybrid

* add parquet encoder to keep schema consistent between writer and record

* use buffer to build list array

* fields support all primitive array

* support bitmap

* check tsid in try_new

* fix CR, add more docs

* fix clippy

* rename field column to non key column

* fix clippy

* fix wrong data type size

* fix reviews

* fix naming convertion, add is_collapsible_column method

* fix clippy

* null bimap init to 1 to avoid unnecessory operation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chunshao90 chunshao90 chunshao90 left review comments

@ShiKaiWi ShiKaiWi ShiKaiWi left review comments

@waynexia waynexia waynexia approved these changes

Assignees
No one assigned
Labels
A-analytic-engine Area: Analytic Engine
Projects
None yet
Milestone
Release v0.3
Development

Successfully merging this pull request may close these issues.
4 participants
@jiacai2050 @waynexia @ShiKaiWi @chunshao90