[Connector-V2][JDBC-connector] support Jdbc dm #2377

laglangyue · 2022-08-06T10:19:20Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

laglangyue · 2022-08-06T10:31:38Z

I imitate MySqlCatalog to DMCatalog, but I don't know it work.
and the workflows is failed, I need some time to solve it，and I try to learn know how to do it rencently.

CalvinKirs · 2022-08-08T01:06:17Z

If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

CalvinKirs · 2022-08-08T01:06:29Z

@ic4y PTAL,thx

ic4y · 2022-08-08T02:28:02Z

I imitate MySqlCatalog to DMCatalog, but I don't know it work. and the workflows is failed, I need some time to solve it，and I try to learn know how to do it rencently.

Ask any questions at any time.

laglangyue · 2022-08-08T05:37:48Z

If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

ok,I will lately

laglangyue · 2022-08-08T11:48:18Z

I imitate MySqlCatalog to DMCatalog, but I don't know it work. and the workflows is failed, I need some time to solve it，and I try to learn know how to do it rencently.

Ask any questions at any time.

TypeMapper is in the method
org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.MySqlCatalog#fromJdbcType

but also in this method
org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.mysql.MySqlTypeMapper

Any difference?

laglangyue · 2022-08-08T16:28:58Z

another question, Mysql has the catalog named MysqlCatalog ,and why does PostgreSQL have no catalog such as PostGresqlCatalog?

ic4y · 2022-08-09T02:27:30Z

I imitate MySqlCatalog to DMCatalog, but I don't know it work. and the workflows is failed, I need some time to solve it，and I try to learn know how to do it rencently.

Ask any questions at any time.

TypeMapper is in the method org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.MySqlCatalog#fromJdbcType

but also in this method org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.mysql.MySqlTypeMapper

Any difference?

There is really no difference between the two, This is indeed repeated in the implementation of the catalog. However, the catalog has not been started yet, so it is not necessary to implement the catalog related things.

Refer to the implementation under the jdbc.internal.dialect package.

ic4y · 2022-08-09T02:30:47Z

another question, Mysql has the catalog named MysqlCatalog ,and why does PostgreSQL have no catalog such as PostGresqlCatalog?

Because only the catalog of mysql is currently implemented, pg has not yet been implemented. DM can also temporarily not implement catalog

laglangyue · 2022-08-09T05:10:05Z

DM can also temporarily not implement catalog

thx，I get , I will modify my code this night

.../java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/dm/DmdbTypeMapper.java

pom.xml

...org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/dm/DmdbJdbcRowConverter.java

ic4y · 2022-08-11T08:00:31Z

Have you tested XA-Transaction?, This will be used in exactly-once Sink. And add how to configure exactly-once sink in the document

laglangyue · 2022-08-11T13:05:57Z

Have you tested XA-Transaction?, This will be used in exactly-once Sink. And add how to configure exactly-once sink in the document

Thx for your guide, I will do it this weekend.

ic4y · 2022-08-15T03:01:09Z

@laglangyue Wait for the JDBC e2e test (#2321) to merge. You need to add test for DM in jdbc e2e test

laglangyue · 2022-08-20T09:28:25Z

I have encountered a difficulty. The PostgreSQL driver is directly introduced and introduced into the Lib of connector-v2. I am afraid this is not appropriate.

In the E2E test, I need a driver in the docker. I can manually copy it to the Lib folder of connector-v2-dist, but it will fail if in the CI environment or someone else.

Therefore, I wonder whether an additional module, similar to e2e-driver-jar-dist, is required to be provided to E2E test and copied to docker.

hailin0 · 2022-08-21T06:58:32Z

The PostgreSQL driver is directly introduced and introduced into the Lib of connector-v2. I am afraid this is not appropriate.

@CalvinKirs Do you have any suggestions

laglangyue · 2022-08-23T13:46:22Z

I have 2 ideas to solve it.

maven-dependency-plugin generate the dir such lib include all dirver-jar, and copy it to container, like above.
Run the command in container wget https://repo1.maven.org/maven2/com/dameng/DmJdbcDriver18/8.1.2.141/DmJdbcDriver18-8.1.2.141.jar,and copy it to lib.

In our pom just need to provided scope
@CalvinKirs

CalvinKirs · 2022-08-29T05:51:58Z

In fact, we could have a dist directory dedicated to these jars if we really needed to. (if necessary), but it seems that they are all jdbc driven, as a temporary solution, you can also use the second one. Of course, how you do it is up to you.

laglangyue · 2022-09-03T12:34:22Z

laglangyue · 2022-09-05T13:18:05Z

Please fixed checkstyle problem and conflct

done

pom.xml

Hisoka-X · 2022-09-06T03:05:01Z

@ic4y Hi, PTAL

laglangyue · 2022-09-06T13:49:11Z

@ic4y @Hisoka-X @CalvinKirs I has rebased it to dev, and change dependencies,Please review again.

tools/dependencies/known-dependencies.txt

seatunnel-dist/release-docs/licenses/LICENSE-DMjdbc.txt

ic4y · 2022-09-07T03:18:28Z

...onnector-jdbc-flink-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/jdbc/JdbcDmdbIT.java

+        assertHasData(SOURCE_TABLE);
+        Container.ExecResult execResult = executeSeaTunnelFlinkJob("/jdbc/jdbc_dm_source_and_sink.conf");
+        Assertions.assertEquals(0, execResult.getExitCode());
+        assertHasData(SINK_TABLE);


Suggest check whether the data in the two tables are the same

I Ihave debugged to check. Verifying the data consistency between sink and source is a public Issue.. I hope to discuss with you in another PR, such as how to preset data and assert verification. I think this is required for all E2E use cases.
Currently, only DM has preset SQL for all fields, and other E2E has only a few fields

...onnector-jdbc-spark-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/jdbc/JDBCDmdbIT.java

seatunnel-examples/seatunnel-spark-connector-v2-example/pom.xml

seatunnel-examples/seatunnel-flink-connector-v2-example/pom.xml

seatunnel-e2e/seatunnel-spark-connector-v2-e2e/pom.xml

...onnector-jdbc-spark-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/jdbc/JDBCDmdbIT.java

...onnector-jdbc-flink-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/jdbc/JdbcDmdbIT.java

...onnector-jdbc-spark-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/jdbc/JDBCDmdbIT.java

...onnector-jdbc-flink-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/jdbc/JdbcDmdbIT.java

...onnector-jdbc-spark-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/jdbc/JDBCDmdbIT.java

ic4y · 2022-09-08T10:58:43Z

LGTM

EricJoy2048

+1

* [Connector-V2][JDBC-connector] Add DM source and sink connector

laglangyue changed the title ~~Jdbc dm~~ [Connector-V2][JDBC-connector] support Jdbc dm Aug 6, 2022

laglangyue force-pushed the jdbc_dm branch from be8e278 to 3d56c06 Compare August 6, 2022 12:45

CalvinKirs added the connectors-v2 label Aug 8, 2022

ic4y mentioned this pull request Aug 8, 2022

[Umbrella][Connector] New SeaTunnel API Connectors #1946

Closed

laglangyue force-pushed the jdbc_dm branch from ea9a0a6 to 61ea6bf Compare August 8, 2022 13:48

laglangyue force-pushed the jdbc_dm branch 2 times, most recently from df82462 to 2b2ce89 Compare August 10, 2022 14:11

ic4y reviewed Aug 11, 2022

View reviewed changes

laglangyue force-pushed the jdbc_dm branch from 793894c to c042f8e Compare August 20, 2022 02:35

laglangyue force-pushed the jdbc_dm branch from 00a0648 to c4054f6 Compare August 23, 2022 13:52

laglangyue force-pushed the jdbc_dm branch 2 times, most recently from df5ae70 to e53e5ee Compare September 3, 2022 07:04

Hisoka-X reviewed Sep 6, 2022

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

Hisoka-X requested a review from ic4y September 6, 2022 03:05

laglangyue force-pushed the jdbc_dm branch from db67c53 to 70fb0b8 Compare September 6, 2022 13:03

ic4y reviewed Sep 7, 2022

View reviewed changes

hailin0 reviewed Sep 7, 2022

View reviewed changes

laglangyue force-pushed the jdbc_dm branch from aa40c75 to 6696c43 Compare September 7, 2022 18:28

laglangyue and others added 9 commits September 8, 2022 12:53

[Connector-V2][JDBC-connector] support dm

561631d

delete the update for start.sh

72bdc99

checkstyle

7d120d1

update doc

42ab6e7

update pom

10415fe

update pom

24cacae

update dm e2e

2c3dc72

update dm e2e

452b6e5

fix checkstyle and dead link

678538e

laglangyue force-pushed the jdbc_dm branch from d255175 to 678538e Compare September 8, 2022 04:54

fix connector v2

53b137d

Hisoka-X approved these changes Sep 8, 2022

View reviewed changes

EricJoy2048 approved these changes Sep 9, 2022

View reviewed changes

EricJoy2048 merged commit 7278209 into apache:dev Sep 9, 2022

laglangyue added a commit to laglangyue/seatunnel that referenced this pull request Sep 11, 2022

[Connector-V2][JDBC-connector] support Jdbc dm (apache#2377)

e449039

* [Connector-V2][JDBC-connector] Add DM source and sink connector

laglangyue deleted the jdbc_dm branch September 11, 2022 06:01

lhyundeadsoul pushed a commit to lhyundeadsoul/incubator-seatunnel that referenced this pull request Sep 13, 2022

[Connector-V2][JDBC-connector] support Jdbc dm (apache#2377)

de48f77

* [Connector-V2][JDBC-connector] Add DM source and sink connector

MRYOG pushed a commit to MRYOG/incubator-seatunnel that referenced this pull request Sep 16, 2022

[Connector-V2][JDBC-connector] support Jdbc dm (apache#2377)

a50fe6f

* [Connector-V2][JDBC-connector] Add DM source and sink connector

TyrantLucifer pushed a commit to TyrantLucifer/incubator-seatunnel that referenced this pull request Sep 18, 2022

[Connector-V2][JDBC-connector] support Jdbc dm (apache#2377)

2723a81

* [Connector-V2][JDBC-connector] Add DM source and sink connector

EricJoy2048 mentioned this pull request Oct 8, 2022

[Umbrella][Connector-V2] SeaTunnel Connector V2 Tracking Table #3018

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Connector-V2][JDBC-connector] support Jdbc dm #2377

[Connector-V2][JDBC-connector] support Jdbc dm #2377

laglangyue commented Aug 6, 2022 •

edited

Loading

laglangyue commented Aug 6, 2022

CalvinKirs commented Aug 8, 2022

CalvinKirs commented Aug 8, 2022

ic4y commented Aug 8, 2022

laglangyue commented Aug 8, 2022

laglangyue commented Aug 8, 2022

laglangyue commented Aug 8, 2022

ic4y commented Aug 9, 2022

ic4y commented Aug 9, 2022

laglangyue commented Aug 9, 2022

ic4y commented Aug 11, 2022

laglangyue commented Aug 11, 2022

ic4y commented Aug 15, 2022

laglangyue commented Aug 20, 2022 •

edited

Loading

hailin0 commented Aug 21, 2022

laglangyue commented Aug 23, 2022 •

edited

Loading

CalvinKirs commented Aug 29, 2022

laglangyue commented Sep 3, 2022

laglangyue commented Sep 5, 2022

Hisoka-X commented Sep 6, 2022

laglangyue commented Sep 6, 2022

ic4y Sep 7, 2022

laglangyue Sep 7, 2022

ic4y commented Sep 8, 2022

EricJoy2048 left a comment

[Connector-V2][JDBC-connector] support Jdbc dm #2377

[Connector-V2][JDBC-connector] support Jdbc dm #2377

Conversation

laglangyue commented Aug 6, 2022 • edited Loading

Purpose of this pull request

Check list

laglangyue commented Aug 6, 2022

CalvinKirs commented Aug 8, 2022

CalvinKirs commented Aug 8, 2022

ic4y commented Aug 8, 2022

laglangyue commented Aug 8, 2022

laglangyue commented Aug 8, 2022

laglangyue commented Aug 8, 2022

ic4y commented Aug 9, 2022

ic4y commented Aug 9, 2022

laglangyue commented Aug 9, 2022

ic4y commented Aug 11, 2022

laglangyue commented Aug 11, 2022

ic4y commented Aug 15, 2022

laglangyue commented Aug 20, 2022 • edited Loading

hailin0 commented Aug 21, 2022

laglangyue commented Aug 23, 2022 • edited Loading

CalvinKirs commented Aug 29, 2022

laglangyue commented Sep 3, 2022

laglangyue commented Sep 5, 2022

Hisoka-X commented Sep 6, 2022

laglangyue commented Sep 6, 2022

ic4y Sep 7, 2022

Choose a reason for hiding this comment

laglangyue Sep 7, 2022

Choose a reason for hiding this comment

ic4y commented Sep 8, 2022

EricJoy2048 left a comment

Choose a reason for hiding this comment

laglangyue commented Aug 6, 2022 •

edited

Loading

laglangyue commented Aug 20, 2022 •

edited

Loading

laglangyue commented Aug 23, 2022 •

edited

Loading