Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

implement JSON type in BE #3085

Merged
merged 13 commits into from
Jan 30, 2022
Merged

implement JSON type in BE #3085

merged 13 commits into from
Jan 30, 2022

Conversation

murphyatwork
Copy link
Contributor

@murphyatwork murphyatwork commented Jan 25, 2022 •
edited
Loading

What type of PR is this：

  • bug
  • feature: implement JSON type in BE
  • enhancement
  • others

Summary

  • Add JSON type as PrimitiveType and FieldType
  • Implement JSON binary serialization with support of velocypack
  • Implement type casting between SQL type(int, bool, string, float) and JSON type
  • Support data loading from .json file

Not included

@imay
Copy link
Contributor

imay commented Jan 25, 2022

Better to split the thirdparty changes into another PR.
After that PR is merged, thirdparty can be rebuild.
Then other changes that depends on it can compile successfully.

@murphyatwork murphyatwork mentioned this pull request Jan 26, 2022
Merged
4 tasks
@murphyatwork
Copy link
Contributor Author

Better to split the thirdparty changes into another PR. After that PR is merged, thirdparty can be rebuild. Then other changes that depends on it can compile successfully.

Submit another PR #3089, including thridparty library and type define.

@murphyatwork murphyatwork changed the title implement JSON type in BE [WIP] implement JSON type in BE Jan 26, 2022
@murphyatwork murphyatwork force-pushed the mofei_json_be branch 3 times, most recently from 534e8d4 to 9bb35de Compare January 27, 2022 12:08
@murphyatwork murphyatwork changed the title [WIP] implement JSON type in BE implement JSON type in BE Jan 28, 2022
kangkaisen
kangkaisen reviewed Jan 28, 2022
View reviewed changes
be/src/storage/types.h Outdated Show resolved Hide resolved
kangkaisen
kangkaisen previously approved these changes Jan 29, 2022
View reviewed changes
satanson
satanson reviewed Jan 29, 2022
View reviewed changes
be/src/column/column_visitor_mutable.h
@@ -40,6 +40,7 @@ class ColumnVisitorMutable {
virtual Status visit(vectorized::HyperLogLogColumn* column);
virtual Status visit(vectorized::BitmapColumn* column);
virtual Status visit(vectorized::PercentileColumn* column);
virtual Status visit(vectorized::JsonColumn* column);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what the difference betweens JsonColumn and ObjectColumn?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, JsonColumn just inherits ObjectColumn, and overwrites some put_mysql_buffer, hash methods.

Even more important, JsonColumn needs to implement its own serialization and access algorithms, more closely to columnar storage, not only ObjectColumn.

be/src/column/json_column.cpp Outdated Show resolved Hide resolved
be/src/column/json_column.cpp
for (uint32_t i = from; i < to; i++) {
JsonValue* json = get_object(i);
int64_t h = json->hash();
hash[i] = HashUtil::fnv_hash(&h, sizeof(h), hash[i]);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fnv_hash is used for bucket shuffling, json->hash() already compute out a hash value, so use the high or low 32 bit is just ok. HashUtil::fnv_hash is needless.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fnv_hash needs to take the input hash into account, so I could not use json->hash as the result?

be/src/common/status.h Outdated Show resolved Hide resolved
be/src/exprs/vectorized/cast_expr.cpp Outdated Show resolved Hide resolved
be/src/exprs/vectorized/cast_expr.cpp Show resolved Hide resolved
be/src/exprs/vectorized/cast_expr.cpp Outdated Show resolved Hide resolved
stdpain
stdpain previously approved these changes Jan 30, 2022
View reviewed changes
Seaven
Seaven previously approved these changes Jan 30, 2022
View reviewed changes
satanson
satanson previously approved these changes Jan 30, 2022
View reviewed changes
@murphyatwork murphyatwork dismissed stale reviews from satanson, Seaven, and stdpain via 4e3fc0f January 30, 2022 07:02
@stdpain stdpain merged commit d445118 into StarRocks:main Jan 30, 2022
jaogoy pushed a commit to jaogoy/starrocks that referenced this pull request Nov 15, 2023
@amber-create
modified file_path descriptions in BROKERLOAD.md in 2.4 (StarRocks#3085)
030166b 
* Update STREAM LOAD.md

* Update BROKER LOAD.md

* modified file_path descriptions in BROKERLOAD.md
@murphyatwork murphyatwork deleted the mofei_json_be branch July 23, 2024 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stdpain stdpain stdpain approved these changes

@Seaven Seaven Seaven approved these changes

@satanson satanson Awaiting requested review from satanson

@kangkaisen kangkaisen Awaiting requested review from kangkaisen

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@murphyatwork @imay @satanson @Seaven @kangkaisen @stdpain