Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBase 2.4 #10

Closed
wants to merge 8 commits into from
Closed

HBase 2.4 #10

wants to merge 8 commits into from

Conversation

ianawilson
Copy link

@ianawilson ianawilson commented Dec 27, 2021

First of all, and a bit of a sidebar, I have a growing suspicion that this fork doesn't need to exist. It seems that its main purpose is to match dependency versions with EMR, but it's possible to do that in our own pom files using the properties that we're modifying in this. I'm going to experiment with that later.


The purpose of this PR is to match our hbase-server version with the jar provided on EMR. While our HBase is listed as 2.2.6, the library code we have from Amazon is for 2.4.1+. There was a breaking change introduced in HBase 2.4.1 which is present in the jars labeled 2.2.6, and this mismatch prevents bulkLoadThinRows() from working.

Specifically what is going on is that a couple of helpers moved from HStore to StoreUtils, eg getChecksumType(). This change was introduced in HBase 2.4.1 via this PR: apache/hbase#2800. If you search that PR's files, you'll see that getChecksumType() moved.

hbase-connectors relies on getChecksumType() and a few other helpers in bulkLoadThinRows(), which we use for the Competitor Historical Migration jobs. With these being out of sync, we can't run the bulk load.

To fix that, this fast forwards and brings in apache#88.


This also includes reverting a few things in our local fork that I don't think make much sense:

  1. We were hard coding the version everywhere instead of using ${revision} and setting it once in the parent properties
  2. The version properties were being modified in the pom and the Jenkinsfile

@ianawilson ianawilson force-pushed the hbase24 branch 2 times, most recently from 2efa19f to 207adad Compare December 28, 2021 23:54
@ianawilson
Copy link
Author

Closing this in favor of #11 which I think creates less noise and doesn't lose upstream commits in a rebase.

@ianawilson ianawilson closed this Jan 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

5 participants