[Feature][Connector-V2] StarRocks-sink support schema evolution #8082

jw-itq · 2024-11-19T06:04:55Z

Feature #8034

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

…ing with Elasticsearch's automatic index creation apache#7430

…utomatic index creation conflict apache#7430

…utomatically creating indexes based on templates apache#7430

…utomatic index creation conflict apache#7430

Hisoka-X · 2024-11-20T06:36:35Z

cc @hailin0 @dailai

hailin0 · 2024-11-21T02:04:12Z

...-e2e/src/test/java/org/apache/seatunnel/e2e/connector/starrocks/StarRocksSchemaChangeIT.java

+    }
+
+    @TestTemplate
+    public void testStarRocksSinkWithSchemaEvolutionCase(TestContainer container)


reference

seatunnel/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/MysqlCDCWithSchemaChangeIT.java

Line 119 in 69cd4ae

public void testMysqlCdcWithSchemaEvolutionCase(TestContainer container)

jw-itq · 2024-11-25T09:11:51Z

@hailin0 hi,can you help me see the reason why CI did not pass,thanks

sunxiaojian · 2024-11-26T07:57:16Z

@hailin0 hi,can you help me see the reason why CI did not pass,thanks

@jw-itq https://productionresultssa16.blob.core.windows.net/actions-results/2b6ed37b-6849-4740-af19-49a0177aea17/workflow-job-run-01458a8d-fc45-5e2e-2042-b9dae5cb9dfb/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-11-26T07%3A59%3A03Z&sig=zo%2BARhStTeBEmkHQdBA7HDK4L%2F%2Bc8THmpctUwF9j4RE%3D&ske=2024-11-26T17%3A39%3A07Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-11-26T05%3A39%3A07Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-11-04&sp=r&spr=https&sr=b&st=2024-11-26T07%3A48%3A58Z&sv=2024-11-04

jw-itq · 2024-11-26T09:01:33Z

@hailin0 hi,can you help me see the reason why CI did not pass,thanks

@jw-itq https://productionresultssa16.blob.core.windows.net/actions-results/2b6ed37b-6849-4740-af19-49a0177aea17/workflow-job-run-01458a8d-fc45-5e2e-2042-b9dae5cb9dfb/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-11-26T07%3A59%3A03Z&sig=zo%2BARhStTeBEmkHQdBA7HDK4L%2F%2Bc8THmpctUwF9j4RE%3D&ske=2024-11-26T17%3A39%3A07Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-11-26T05%3A39%3A07Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-11-04&sp=r&spr=https&sr=b&st=2024-11-26T07%3A48%3A58Z&sv=2024-11-04

@sunxiaojian thank you very much! I know how to view the logs now. I didn't find any errors in the action interface before

hailin0 · 2024-11-28T04:16:11Z

.../main/java/org/apache/seatunnel/connectors/seatunnel/starrocks/sink/StarRocksSinkWriter.java

+        }
+
+        try {
+            Connection conn =


Why not close connection ?

Why not close connection ?

sorry, I'll optimize it

hailin0 · 2024-11-28T04:16:25Z

.../main/java/org/apache/seatunnel/connectors/seatunnel/starrocks/sink/StarRocksSinkWriter.java

+        if (event instanceof AlterTableColumnsEvent) {
+            AlterTableColumnsEvent alterTableColumnsEvent = (AlterTableColumnsEvent) event;


reference
#8158

ok,thanks！

.../main/java/org/apache/seatunnel/connectors/seatunnel/starrocks/sink/StarRocksSinkWriter.java

hailin0 · 2024-12-02T07:07:26Z

Please fix the ci error

jw-itq · 2024-12-02T07:09:04Z

Please fix the ci error

ok!

...2e/seatunnel-connector-v2-e2e/connector-starrocks-e2e/src/test/resources/ddl/add_columns.sql

...seatunnel-connector-v2-e2e/connector-starrocks-e2e/src/test/resources/ddl/change_columns.sql

...e/seatunnel-connector-v2-e2e/connector-starrocks-e2e/src/test/resources/ddl/drop_columns.sql

...seatunnel-connector-v2-e2e/connector-starrocks-e2e/src/test/resources/ddl/modify_columns.sql

...unnel-e2e/seatunnel-connector-v2-e2e/connector-starrocks-e2e/src/test/resources/ddl/shop.sql

...va/org/apache/seatunnel/connectors/seatunnel/starrocks/datatypes/StarRocksTypeConverter.java

jw-itq · 2024-12-03T13:42:59Z

StarRocksTypeConverter referenced from #8087

dailai

LGTM

jw-itq added 14 commits August 22, 2024 19:10

[Fix] [sink elasticsearch] Fix the issue of sink-es saveMode conflict…

d272789

…ing with Elasticsearch's automatic index creation apache#7430

[Fix] [sink elasticsearch] Fix the issue of sink-es saveMode and es a…

4ebc66b

…utomatic index creation conflict apache#7430

[Bug] [sink elasticsearch] the savemode of sink-es conficts with es a…

6810d80

…utomatically creating indexes based on templates apache#7430

[Fix] [sink elasticsearch] Fix the issue of sink-es saveMode and es a…

ac50c3e

…utomatic index creation conflict apache#7430

[Fix] [sink elasticsearch] Fix the issue of sink-es saveMode and es a…

6e8ccc9

…utomatic index creation conflict apache#7430

[Doc] Add IGNORE savemode type into docment apache#7443

3936038

Merge branch 'apache:dev' into dev

3bc24c2

Merge branch 'apache:dev' into dev

c26e89d

Merge branch 'apache:dev' into dev

b5f5162

Merge branch 'apache:dev' into dev

0347da2

Merge branch 'apache:dev' into dev

4bce27f

Merge branch 'apache:dev' into dev

d4993f3

Merge branch 'apache:dev' into dev

5d61e76

support schema evolution

2723cb4

github-actions bot added connectors-v2 e2e starrocks labels Nov 19, 2024

jw-itq added 10 commits November 19, 2024 16:57

change package name

0b33432

fix

b00c5d0

fix

5665556

fix

12c8e68

fix

abd304e

fix e2e test

a947891

fix e2e test

e17e1aa

fix e2e test

3a05bf6

log level

bb429c4

change log level

8f8e143

fix fe port

0693041

hailin0 reviewed Nov 21, 2024

View reviewed changes

This was referenced Nov 24, 2024

[Feature][Jdbc] Support sink ddl for postgresql #8113

Closed

[Feature][Clickhouse] Support sink ddl #8061

Open

[Feature][kafka] Support sink ddl #8135

Closed

jw-itq added 2 commits November 25, 2024 12:07

optimized code

36a892c

update doc

43a02ac

github-actions bot added the document label Nov 25, 2024

fix the createWriter method return type

46ab3b3

hailin0 reviewed Nov 28, 2024

View reviewed changes

fix code

5d2fabe

hailin0 reviewed Nov 28, 2024

View reviewed changes

.../main/java/org/apache/seatunnel/connectors/seatunnel/starrocks/sink/StarRocksSinkWriter.java Outdated Show resolved Hide resolved

fix code

7ab43fd

jw-itq added 3 commits December 2, 2024 15:09

Merge branch 'dev' into starrock-sink-ddl3

92bb68a

fix code style

d007452

fix ci

bf6b696

dailai reviewed Dec 3, 2024

View reviewed changes

optimize sql files and add UT for typeconverter

a061913

dailai approved these changes Dec 4, 2024

View reviewed changes

github-actions bot added approved reviewed labels Dec 4, 2024

hailin0 approved these changes Dec 4, 2024

View reviewed changes

hailin0 merged commit d33b0da into apache:dev Dec 4, 2024
6 checks passed

hailin0 mentioned this pull request Dec 9, 2024

[Feature][Connector-V2[Doris]Support sink ddl #8250

Merged

4 tasks

hailin0 mentioned this pull request Dec 16, 2024

SeaTunnel community meeting Topic collect #8309

Closed

3 tasks

hailin0 mentioned this pull request Jan 14, 2025

SeaTunnel community meeting Topic collect #8517

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2] StarRocks-sink support schema evolution #8082

[Feature][Connector-V2] StarRocks-sink support schema evolution #8082

jw-itq commented Nov 19, 2024

Hisoka-X commented Nov 20, 2024

hailin0 Nov 21, 2024

jw-itq Nov 21, 2024

jw-itq commented Nov 25, 2024

sunxiaojian commented Nov 26, 2024 •

edited

Loading

jw-itq commented Nov 26, 2024 •

edited

Loading

hailin0 Nov 28, 2024

jw-itq Nov 28, 2024

hailin0 Nov 28, 2024

jw-itq Nov 28, 2024

hailin0 commented Dec 2, 2024

jw-itq commented Dec 2, 2024

jw-itq commented Dec 3, 2024

dailai left a comment

		if (event instanceof AlterTableColumnsEvent) {
		AlterTableColumnsEvent alterTableColumnsEvent = (AlterTableColumnsEvent) event;

[Feature][Connector-V2] StarRocks-sink support schema evolution #8082

[Feature][Connector-V2] StarRocks-sink support schema evolution #8082

Conversation

jw-itq commented Nov 19, 2024

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Hisoka-X commented Nov 20, 2024

hailin0 Nov 21, 2024

Choose a reason for hiding this comment

jw-itq Nov 21, 2024

Choose a reason for hiding this comment

jw-itq commented Nov 25, 2024

sunxiaojian commented Nov 26, 2024 • edited Loading

jw-itq commented Nov 26, 2024 • edited Loading

hailin0 Nov 28, 2024

Choose a reason for hiding this comment

jw-itq Nov 28, 2024

Choose a reason for hiding this comment

hailin0 Nov 28, 2024

Choose a reason for hiding this comment

jw-itq Nov 28, 2024

Choose a reason for hiding this comment

hailin0 commented Dec 2, 2024

jw-itq commented Dec 2, 2024

jw-itq commented Dec 3, 2024

dailai left a comment

Choose a reason for hiding this comment

sunxiaojian commented Nov 26, 2024 •

edited

Loading

jw-itq commented Nov 26, 2024 •

edited

Loading