-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement using Postgres instead of MariaDB for hive-metastore #982
Comments
Cloud answer: You would have to either use PostgreSQL or MySQL managed as currently the only ATO'd. I think AAW has a preference for PostgreSQL ^^_ Therefore we will go with PostgreSQL |
It was chosen that since Minio will soon be remove, this task will be implemented with Azure blob storage account instead of s3. |
Following a discussion, it was decided to use the binami image. To test, we need the same trino steps as local development. Note: how to get the access-token: Note 2: to get the trino.example.com, need to add it to /etc/hosts, have the metallb stuff running in your cluster |
Currently implemented using the image from bitnami but that does not include a hive-metastore component. |
The idea I am currently following is to replace all mentions of mysql by postgresql. Some of the sites I looked at to follow : |
Currently, the postgresql is there, but it does not connect with the metastore. From using k9s, I cannot see the metastore in either the svc, statefulset, pods or helm chart. All of which i expected to see an instance of hive-metastore in addition to hive-metastore-postgres Also, when using the Despite port 9083 being associated with the service |
We seem to have hive-metastore now, but it doesnt seem to really start.
Similarly, when shelling into the hive pod and trying to curl the trino-coordinator service, :
|
I had to add jdk 11 aka Java to the docker image because the schematool -initSchema -dbType postgresand the startmetastore could be executed.
Unfortunately there seems to be multiple issues with this.
Similarly but with different outcome, the other one:
|
Fixed the driver, with the correct version. It seems when the
are executed, it immediately tries to create a database, which makes sense. But it tries to do it with data i am not sure where it came. 'metastore-db' database, as well as something called Derby.
And the second one:
Which might just be because the init script was already run. instead i ran this:
It complains about a missing file. |
On June 4th 2022, there was an update of the bitnami repo.
When using the Chart 10.2.1 it gives this error. I tried updating for a newer version, be it 10.21.0 (which was updated yesterday) but it gives the same error. RESOLVED: Delete the cluster ( |
Yesterday we managed to get the docker image built in a way that worked.
of course the configmap had to be updated for this change as well. Which will need to be done in a way to not be hardcoded code.
|
The changes now work. I will link the repo in which our current docker image for bitnami with hive-standalone can be found. |
To discuss with Cloud. See the repercussions.
The text was updated successfully, but these errors were encountered: