Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CH-123] Support short/byte/binary/decimal/array/map/struct #128

Merged
merged 63 commits into from
Oct 14, 2022
Merged

[CH-123] Support short/byte/binary/decimal/array/map/struct #128

merged 63 commits into from
Oct 14, 2022

Conversation

taiyang-li
Copy link

@taiyang-li taiyang-li commented Sep 19, 2022 •
edited
Loading

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Support CH column to Spark row for complex types

Detailed Steps:

  • Calculate backing buffer length for complex types(array/map/tuple)
  • Implement array writer
  • Implement map writer
  • Implement tuple writer
  • Implement decimal writer
  • Implement array reader
  • Implement map reader
  • Implement tuple reader
  • Implement decimal reader
  • Add tests for above reader and writer
  • Add benchmarks for above reader and writer
  • Complex type transformation between substrait and CH.
  • Complex type transformation between Spark and substrait in gluten.
  • Improve the performance of above writers(BM_CHColumnToSparkRow_Lineitem: from 328ms to 260ms)
  • Improve the performance of above readers(BM_SparkRowToCHColumn_Lineitem: from 359ms to 305ms)
  • implement short/byte/binary

Benchmark baseline branch: https://github.com/bigo-sg/clickhouse/tree/baseline

@kyligence-git
Copy link
Collaborator

Can one of the admins verify this patch?

This was referenced Sep 19, 2022
Closed
Closed
@taiyang-li taiyang-li mentioned this pull request Sep 23, 2022
Closed
@taiyang-li taiyang-li changed the title support conversion between CH column and Spark row for decimal/array/map/struct support conversion between CH column and Spark row for short/byte/binary/decimal/array/map/struct Oct 9, 2022
@taiyang-li taiyang-li changed the title support conversion between CH column and Spark row for short/byte/binary/decimal/array/map/struct Support short/byte/binary/decimal/array/map/struct Oct 9, 2022
@taiyang-li taiyang-li changed the title Support short/byte/binary/decimal/array/map/struct [CH-123] Support short/byte/binary/decimal/array/map/struct Oct 9, 2022
@liuneng1994
Copy link
Collaborator

test this please with 402

3 similar comments
@liuneng1994
Copy link
Collaborator

test this please with 402

@liuneng1994
Copy link
Collaborator

test this please with 402

@liuneng1994
Copy link
Collaborator

test this please with 402

@taiyang-li
Copy link
Author

test this please with 402

2 similar comments
@taiyang-li
Copy link
Author

test this please with 402

@taiyang-li
Copy link
Author

test this please with 402

@taiyang-li
Copy link
Author

test this please with 402

@taiyang-li
Copy link
Author

test this please with 402

1 similar comment
@taiyang-li
Copy link
Author

test this please with 402

liuneng1994
liuneng1994 approved these changes Oct 14, 2022
View reviewed changes
Copy link
Collaborator

@liuneng1994 liuneng1994 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liuneng1994 liuneng1994 merged commit 729900c into Kyligence:clickhouse_backend Oct 14, 2022
liuneng1994 added a commit that referenced this pull request Oct 17, 2022
@liuneng1994
Revert "[CH-123] Support short/byte/binary/decimal/array/map/struct (#…
89d6bb8 
…128)"

This reverts commit 729900c.
@liuneng1994 liuneng1994 mentioned this pull request Oct 17, 2022
Merged
liuneng1994 added a commit that referenced this pull request Oct 17, 2022
@liuneng1994
Revert "[CH-123] Support short/byte/binary/decimal/array/map/struct (#…
8ba0bed 
…128)" (#159)

This reverts commit 729900c.
@taiyang-li taiyang-li mentioned this pull request Oct 20, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lgbo-ustc lgbo-ustc lgbo-ustc left review comments

@liuneng1994 liuneng1994 liuneng1994 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@taiyang-li @kyligence-git @liuneng1994 @lgbo-ustc