[Feature][Hive JDBC Source] Support Hive JDBC Source Connector #5424

NickCodeJourney · 2023-09-04T14:40:54Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

This commit includes the addition of Hive JDBC dependencies in the pom.xml file to enable the JDBC connector support Hive. The Hive dialect, type mapper and row converter were added accordingly. This update allows users to connect and interact with Hive directly using the JDBC connector."

EricJoy2048

Please add document for this connector reference https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/source/Mysql.md

EricJoy2048 · 2023-09-06T03:28:56Z

Please add e2e for this connector reference https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlIT.java

This commit introduces a new documentation file under `/docs/en/connector-v2/source/` explaining how to use the Hive JDBC source connector. The included information covers the supported engines, key features, support DataSource info, data type mapping, source options and database dependency, along with examples for how to use the connector with different configurations. This documentation is intended to help users to understand and effectively use the Hive JDBC source connector in their data pipeline.

NickCodeJourney · 2023-09-06T13:27:16Z

...ava/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/hive/HiveTypeMapper.java

+ case HIVE_NUMERIC:
+ if (precision > 0) {
+ return new DecimalType(precision, metadata.getScale(colIndex));
+ }
+ LOG.warn("decimal did define precision,scale, will be Decimal(38,18)");
+ return new DecimalType(38, 18);


I think it should maintain similar data types

Introduces JDBC Hive Integration Tests and new configuration settings for the 'connector-jdbc-e2e' module in the SeaTunnel v2 project. These additions are designed to enhance testing and configuration handling capabilities for connections to Hive databases. The new JdbcHiveIT class implements Hive-specific configurations and integration test cases. The jdbc_hive_source.conf file contains a sample configuration setup for a Hive JDBC source connection.

Fixed the case sensitivity issue in jdbc config filename in e2e-test. Changed "/jdbc_HIVE_source_and_sink.conf" to "/jdbc_hive_source_and_sink.conf". Linux servers are case sensitive hence the filename should match exact case to avoid file not found exceptions while running end-to-end testing.

...ava/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/hive/HiveTypeMapper.java

...jdbc-e2e-part-3/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcHiveIT.java

This commit adds a BYTE_TYPE mapping to the HiveTypeMapper class for the HIVE_TINYINT field. This change allows mapping of HIVE_TINYINT to BasicType.BYTE_TYPE, ensuring type consistency.

zhilinli123 · 2023-09-13T07:58:45Z

...v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-3/src/test/resources/jdbc_hive_source.conf

+ Jdbc {
+ url = "jdbc:hive2://localhost:10000/default"
+ user = "root"
+ driver = "org.apache.hive.jdbc.HiveDriver"


We currently do not have hive E2e tests, can your E2e directly cover jdbc tests and hive thrift mode read and write, metastore
https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/source/Hive.md,https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Hive.md

https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/source/Hive.md,https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Hive.md

jdbc_hive and hive are difference connector, It would be better to separate the e2e of the two connectors. We are missing the e2e for the hive connector.

@NickYoungPeng If you are willing, you can provide a separate PR to add the e2e for the hive connector.

# Conflicts: # seatunnel-connectors-v2/connector-jdbc/pom.xml

Removed redundant file and tidied up pom.xml. Removed the file jdbc_hive_source.conf, added needed dependencies for Hive in pom.xml, and removed an unused module from connector-file pom. Improved tests by updating test configuration to better demonstrate features and modified corresponding test class accordingly. Changes aim to enhance test readability and correctness.

NickCodeJourney · 2023-09-15T02:30:56Z

I want Hive JDBC to support Kerberos authentication. Do you have any suggestions?

My 1st idea is to enhance the SimpleJdbcConnectionProvider. This would involve modifying the JdbcConnectionConfig class to support Hive JDBC Kerberos authentication.
My 2ed idea is Create a new JdbcConnectionProvider, such as HiveJdbcConnectionProvider, and inject it through SPI (Service Provider Interface).

Updated the HIVE_IMAGE from "youngyangp/hive_3.1.2_arm:1.0.0" to "apache/hive:3.1.3" in the JdbcHiveIT test. Also, introduced "SERVICE_NAME" as an environment variable to the Hive server container. The HIVE_IMAGE was updated for using an official Apache hive image. The environment variable is crucial for service internal recognition.

Uniformly formatted the environment variables for better code readability. The change is on line 160 from '.withEnv("SERVICE_NAME","hiveserver2")' to '.withEnv("SERVICE_NAME", "hiveserver2")'. The whitespace after file.

Added a new parameter, 'auto_commit', to the jdbc_hive configuration file to improve database transaction management. This is pivotal for applications that require a high level of data integrity and consistency, or for those that need to manage complex, multi-step transactions.

Updated hive.jdbc.version to 3.1.3 in connector-jdbc/pom.xml to have access to the latest features, bug fixes and improvements. Also, removed the 'provided' scope from the same file as it is not necessary and its presence could lead to potential issues with the dependency resolution during the build process.

EricJoy2048 · 2023-09-18T08:35:01Z

I want Hive JDBC to support Kerberos authentication. Do you have any suggestions?

My 1st idea is to enhance the SimpleJdbcConnectionProvider. This would involve modifying the JdbcConnectionConfig class to support Hive JDBC Kerberos authentication.

My 2ed idea is Create a new JdbcConnectionProvider, such as HiveJdbcConnectionProvider, and inject it through SPI (Service Provider Interface).

It seems that the second option is more suitable.

This commit enables kerberos authentication while establishing JDBC connections. The JdbcOptions class has been updated to include new options such as 'use_kerberos', 'kerberos_principal', etc to configure Kerberos settings. A new connection provider 'HiveJdbcConnectionProvider' has been added which takes care of connection creation when using Kerberos. This change also makes the connection provider more flexible as it can now be selected based on the JDBC dialect."

Updated the dependency to the oceanbase-client from the com.oceanbase group in the connector-jdbc pom.xml file. This change is necessary to maintain API consistency and efficiency.

...in/java/org/apache/seatunnel/connectors/seatunnel/jdbc/exception/JdbcConnectorErrorCode.java

Introduced new error codes JDBC-07 and JDBC-08 in connection with the JDBC in the Error Quick Reference Manual. These additions are important to help users understand and diagnose the new issues related to unsupported jdbc type and failed Kerberos authentication respectively.

Updated the default krb5 file path in jdbc configuration from '/etc/krb5.conf' to '/seatunnel/krb5.conf' or use the default path '/etc/krb5.conf'. This change was made to provide flexibility if the krb5.conf file is not found at the default location. The documentation has been updated to reflect this change

docs/en/connector-v2/source/Hive-jdbc.md

Included definite support information for Hive version 3.1.3 and 3.1.2 in JDBC Hive Source Connector doc, while stating that other versions need testing. This provides clarity on version compatibility, helping users determine if their Hive version is definitely compatible or requires testing.

zhilinli123 · 2023-10-31T03:34:53Z

fix ci

Carl-Zhou-CN

+1

...r-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/utils/HiveJdbcUtils.java

yangpeng added 2 commits September 4, 2023 22:38

NickCodeJourney marked this pull request as draft September 4, 2023 14:41

NickCodeJourney marked this pull request as ready for review September 4, 2023 14:41

🎨 Refactor HiveTypeMapper for JDBC connector

5072173

EricJoy2048 reviewed Sep 6, 2023

View reviewed changes

NickCodeJourney commented Sep 6, 2023

View reviewed changes

yangpeng added 2 commits September 7, 2023 01:55

EricJoy2048 reviewed Sep 8, 2023

View reviewed changes

🎨 Add BYTE_TYPE mapping in HiveTypeMapper

ab7125b

This commit adds a BYTE_TYPE mapping to the HiveTypeMapper class for the HIVE_TINYINT field. This change allows mapping of HIVE_TINYINT to BasicType.BYTE_TYPE, ensuring type consistency.

zhilinli123 reviewed Sep 13, 2023

View reviewed changes

yangpeng added 2 commits September 14, 2023 21:42

Merge remote-tracking branch 'origin1/dev' into hiveJdbc

6bd08a0

# Conflicts: # seatunnel-connectors-v2/connector-jdbc/pom.xml

yangpeng and others added 6 commits September 15, 2023 15:26

🎨 Refactor styling in JdbcHiveIT file

4d31e34

Uniformly formatted the environment variables for better code readability. The change is on line 160 from '.withEnv("SERVICE_NAME","hiveserver2")' to '.withEnv("SERVICE_NAME", "hiveserver2")'. The whitespace after file.

Merge branch 'apache:dev' into hivejdbc

f37c375

Merge branch 'apache:dev' into hivejdbc

0b3a237

EricJoy2048 added this to the 2.3.4 milestone Sep 25, 2023

NickCodeJourney and others added 5 commits September 25, 2023 16:59

Merge branch 'apache:dev' into hivejdbc

6c6f15d

Merge branch 'apache:dev' into hivejdbc

15428f6

Merge remote-tracking branch 'origin/hivejdbc' into hivejdbc-1

26faef0

NickCodeJourney and others added 3 commits October 27, 2023 13:50

Merge branch 'dev' into hivejdbc

23eaedb

Merge branch 'apache:dev' into hivejdbc

cc1c38f

Update dependency in connector-jdbc pom.xml

c601e84

Updated the dependency to the oceanbase-client from the com.oceanbase group in the connector-jdbc pom.xml file. This change is necessary to maintain API consistency and efficiency.

EricJoy2048 previously approved these changes Oct 30, 2023

View reviewed changes

github-actions bot added approved reviewed labels Oct 30, 2023

Carl-Zhou-CN reviewed Oct 30, 2023

View reviewed changes

...in/java/org/apache/seatunnel/connectors/seatunnel/jdbc/exception/JdbcConnectorErrorCode.java Show resolved Hide resolved

NickCodeJourney dismissed EricJoy2048’s stale review via 1db6ce0 October 30, 2023 06:10

github-actions bot removed approved reviewed labels Oct 30, 2023

NickCodeJourney and others added 2 commits October 30, 2023 14:24

Merge branch 'dev' into hivejdbc

a31931a

NickCodeJourney requested review from Carl-Zhou-CN and EricJoy2048 October 30, 2023 06:28

zhilinli123 reviewed Oct 30, 2023

View reviewed changes

docs/en/connector-v2/source/Hive-jdbc.md Show resolved Hide resolved

NickCodeJourney added 2 commits October 31, 2023 09:55

Merge remote-tracking branch 'origin/hivejdbc' into hiveJdbc

b8d57b2

NickCodeJourney requested a review from zhilinli123 October 31, 2023 01:56

Merge branch 'apache:dev' into hivejdbc

027b58d

zhilinli123 approved these changes Nov 1, 2023

View reviewed changes

github-actions bot added the reviewed label Nov 1, 2023

EricJoy2048 approved these changes Nov 1, 2023

View reviewed changes

github-actions bot added the approved label Nov 1, 2023

Carl-Zhou-CN approved these changes Nov 1, 2023

View reviewed changes

hailin0 approved these changes Nov 7, 2023

View reviewed changes

hailin0 merged commit a64e177 into apache:dev Nov 7, 2023
8 checks passed

NickCodeJourney deleted the hivejdbc branch November 7, 2023 06:22

Hisoka-X reviewed Apr 8, 2024

View reviewed changes

...r-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/utils/HiveJdbcUtils.java Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Hive JDBC Source] Support Hive JDBC Source Connector #5424

[Feature][Hive JDBC Source] Support Hive JDBC Source Connector #5424

NickCodeJourney commented Sep 4, 2023 •

edited

Loading

EricJoy2048 left a comment

EricJoy2048 commented Sep 6, 2023

NickCodeJourney Sep 6, 2023

zhilinli123 Sep 13, 2023

EricJoy2048 Sep 13, 2023

NickCodeJourney commented Sep 15, 2023 •

edited

Loading

EricJoy2048 commented Sep 18, 2023

zhilinli123 commented Oct 31, 2023

Carl-Zhou-CN left a comment

[Feature][Hive JDBC Source] Support Hive JDBC Source Connector #5424

[Feature][Hive JDBC Source] Support Hive JDBC Source Connector #5424

Conversation

NickCodeJourney commented Sep 4, 2023 • edited Loading

Purpose of this pull request

Check list

EricJoy2048 left a comment

Choose a reason for hiding this comment

EricJoy2048 commented Sep 6, 2023

NickCodeJourney Sep 6, 2023

Choose a reason for hiding this comment

zhilinli123 Sep 13, 2023

Choose a reason for hiding this comment

EricJoy2048 Sep 13, 2023

Choose a reason for hiding this comment

NickCodeJourney commented Sep 15, 2023 • edited Loading

EricJoy2048 commented Sep 18, 2023

zhilinli123 commented Oct 31, 2023

Carl-Zhou-CN left a comment

Choose a reason for hiding this comment

NickCodeJourney commented Sep 4, 2023 •

edited

Loading

NickCodeJourney commented Sep 15, 2023 •

edited

Loading