Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat] Add support to use hbase online store #126

Merged
merged 14 commits into from
Sep 20, 2024
Merged

[feat] Add support to use hbase online store #126

merged 14 commits into from
Sep 20, 2024

Conversation

shydefoo
Copy link
Contributor

@shydefoo shydefoo commented Sep 2, 2024

Summary

This PR adds support to use hbase as the online store.

  • It modifies the spark-job and caraml-serving to connect to hbase (for both stream and batch ingestion), but reuses most of the existing logic to pull data from bigtable
  • It piggybacks the changes from feat: add HBase SDK for serving #127 to use hbase sdk for bigtable but adds some extra new modifications to communicate with hbase

@shydefoo shydefoo changed the title [feat] Add support to connect to hbase [feat] Add support for spark jobs to connect to hbase Sep 2, 2024
Use of deprecated classes result in ImmutableHTableDescriptor returned which throws java.lang.UnsupportedOperationException: HTableDescriptor is read-only error
* Use offset and length to get rowCell values because hbase server
  returns slightly different response structure than bigtable
* This is also applied when looking up the avro schema
@shydefoo shydefoo changed the title [feat] Add support for spark jobs to connect to hbase [feat] Add support to use hbase online store Sep 9, 2024
@shydefoo shydefoo changed the base branch from main to bayu/hbase September 9, 2024 02:52
@shydefoo shydefoo changed the base branch from bayu/hbase to main September 9, 2024 02:53
@shydefoo shydefoo changed the base branch from main to bayu/hbase September 17, 2024 07:38
@shydefoo shydefoo marked this pull request as ready for review September 17, 2024 07:56
# # topology refresh period in seconds
# refreshPeriodSecond: 30
#
# redis:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it expected for redis and redisCluster commented?


featuresCF.setMaxVersions(1)

// TODO: Set compression type for column family
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems this comment can be removed, or is it expected for future notes?

@shydefoo shydefoo merged commit 7292b48 into bayu/hbase Sep 20, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants