From 7c3bef6a8064745b716b3d82b6f9fa0ec7bc0720 Mon Sep 17 00:00:00 2001 From: fffy <63960540@qq.com> Date: Wed, 17 Jul 2024 16:11:32 +0800 Subject: [PATCH 1/3] Support the version mapping between pipeline connectors and flink --- .../docs/connectors/pipeline-connectors/overview.md | 8 ++++++++ .../docs/connectors/pipeline-connectors/overview.md | 8 ++++++++ 2 files changed, 16 insertions(+) diff --git a/docs/content.zh/docs/connectors/pipeline-connectors/overview.md b/docs/content.zh/docs/connectors/pipeline-connectors/overview.md index 80c8251b416..fea91b9e66e 100644 --- a/docs/content.zh/docs/connectors/pipeline-connectors/overview.md +++ b/docs/content.zh/docs/connectors/pipeline-connectors/overview.md @@ -38,6 +38,14 @@ Flink CDC 提供了可用于 YAML 作业的 Pipeline Source 和 Sink 连接器 | [Paimon]({{< ref "docs/connectors/pipeline-connectors/paimon" >}}) | Sink |
  • [Paimon](https://paimon.apache.org/): 0.6, 0.7, 0.8 | | [StarRocks]({{< ref "docs/connectors/pipeline-connectors/starrocks" >}}) | Sink |
  • [StarRocks](https://www.starrocks.io/): 2.x, 3.x | +## Supported Flink Versions +下表展示了Flink® CDC管道连接器和Flink® 之间的版本映射 + +| Flink® CDC Version | Flink® Version | Pipeline Source | Pipeline Sink | 备注 | +|:-----------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------:|:-----------------:|:-------------------------------:|:------------------------------:| +| 3.0.x | 1.17.\*, 1.18.\* | Mysql | StarRocks,Doris | | +| 3.1.x | 1.17.\*, 1.18.\*, 1.19.\* | Mysql | StarRocks,Doris,Paimon,Kafka | 仅flink-cdc 3.1.1支持flink 1.19.* | + ## Develop Your Own Connector 如果现有的连接器无法满足您的需求，您可以自行开发自己的连接器，以将您的外部系统集成到 Flink CDC 数据管道中。查阅 [Flink CDC APIs]({{< ref "docs/developer-guide/understand-flink-cdc-api" >}}) 了解如何开发您自己的连接器。 diff --git a/docs/content/docs/connectors/pipeline-connectors/overview.md b/docs/content/docs/connectors/pipeline-connectors/overview.md index 2d536a68f6e..8fbf7403d44 100644 --- a/docs/content/docs/connectors/pipeline-connectors/overview.md +++ b/docs/content/docs/connectors/pipeline-connectors/overview.md @@ -41,6 +41,14 @@ definition. | [Paimon]({{< ref "docs/connectors/pipeline-connectors/paimon" >}}) | Sink |
  • [Paimon](https://paimon.apache.org/): 0.6, 0.7, 0.8 | | [StarRocks]({{< ref "docs/connectors/pipeline-connectors/starrocks" >}}) | Sink |
  • [StarRocks](https://www.starrocks.io/): 2.x, 3.x | +## Supported Flink Versions +The following table shows the version mapping between Flink® CDC Pipeline Connectors and Flink® + +| Flink® CDC Version | Flink® Version | Pipeline Source | Pipeline Sink | Notes | +|:-----------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------:|:-----------------:|:-------------------------------:|:-------------------------------------------:| +| 3.0.x | 1.17.\*, 1.18.\* | Mysql | StarRocks,Doris | | +| 3.1.x | 1.17.\*, 1.18.\*, 1.19.\* | Mysql | StarRocks,Doris,Paimon,Kafka | only flink-cdc 3.1.1 support flink 1.19.* | + ## Develop Your Own Connector If provided connectors cannot fulfill your requirement, you can always develop From 28589cae35229c9493aea919ccdcef892caf6bb9 Mon Sep 17 00:00:00 2001 From: fffy <63960540@qq.com> Date: Wed, 17 Jul 2024 16:35:59 +0800 Subject: [PATCH 2/3] check some code --- .../docs/connectors/pipeline-connectors/overview.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/content.zh/docs/connectors/pipeline-connectors/overview.md b/docs/content.zh/docs/connectors/pipeline-connectors/overview.md index fea91b9e66e..986cea4b971 100644 --- a/docs/content.zh/docs/connectors/pipeline-connectors/overview.md +++ b/docs/content.zh/docs/connectors/pipeline-connectors/overview.md @@ -39,12 +39,12 @@ Flink CDC 提供了可用于 YAML 作业的 Pipeline Source 和 Sink 连接器 | [StarRocks]({{< ref "docs/connectors/pipeline-connectors/starrocks" >}}) | Sink |
  • [StarRocks](https://www.starrocks.io/): 2.x, 3.x | ## Supported Flink Versions -下表展示了Flink® CDC管道连接器和Flink® 之间的版本映射 +下表展示了 Flink® CDC 管道连接器和 Flink® 之间的版本映射 -| Flink® CDC Version | Flink® Version | Pipeline Source | Pipeline Sink | 备注 | -|:-----------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------:|:-----------------:|:-------------------------------:|:------------------------------:| -| 3.0.x | 1.17.\*, 1.18.\* | Mysql | StarRocks,Doris | | -| 3.1.x | 1.17.\*, 1.18.\*, 1.19.\* | Mysql | StarRocks,Doris,Paimon,Kafka | 仅flink-cdc 3.1.1支持flink 1.19.* | +| Flink® CDC Version | Flink® Version | Pipeline Source | Pipeline Sink | 备注 | +|:-----------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------:|:-----------------:|:-------------------------------:|:---------------------------------:| +| 3.0.x | 1.17.\*, 1.18.\* | Mysql | StarRocks,Doris | | +| 3.1.x | 1.17.\*, 1.18.\*, 1.19.\* | Mysql | StarRocks,Doris,Paimon,Kafka | 仅 flink-cdc 3.1.1 支持 flink 1.19.* | ## Develop Your Own Connector From 755bf9362909d6f874195f7da7a96a61236a081a Mon Sep 17 00:00:00 2001 From: fffy <63960540@qq.com> Date: Thu, 5 Sep 2024 14:59:03 +0800 Subject: [PATCH 3/3] suit 3.2 version --- .../docs/connectors/pipeline-connectors/overview.md | 10 ++++++---- .../docs/connectors/pipeline-connectors/overview.md | 9 +++++---- .../src/test/resources/docker/server/my.cnf | 2 ++ 3 files changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/content.zh/docs/connectors/pipeline-connectors/overview.md b/docs/content.zh/docs/connectors/pipeline-connectors/overview.md index 986cea4b971..56edf1b8808 100644 --- a/docs/content.zh/docs/connectors/pipeline-connectors/overview.md +++ b/docs/content.zh/docs/connectors/pipeline-connectors/overview.md @@ -41,10 +41,12 @@ Flink CDC 提供了可用于 YAML 作业的 Pipeline Source 和 Sink 连接器 ## Supported Flink Versions 下表展示了 Flink® CDC 管道连接器和 Flink® 之间的版本映射 -| Flink® CDC Version | Flink® Version | Pipeline Source | Pipeline Sink | 备注 | -|:-----------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------:|:-----------------:|:-------------------------------:|:---------------------------------:| -| 3.0.x | 1.17.\*, 1.18.\* | Mysql | StarRocks,Doris | | -| 3.1.x | 1.17.\*, 1.18.\*, 1.19.\* | Mysql | StarRocks,Doris,Paimon,Kafka | 仅 flink-cdc 3.1.1 支持 flink 1.19.* | +| Flink® CDC Version | Flink® Version | Pipeline Source | Pipeline Sink | 备注 | +|:-----------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------:|:---------------:|:------------------------------------------:|:----------------------------------| +| 3.0.x | 1.17.\*, 1.18.\* | Mysql | StarRocks,Doris | | +| 3.1.x | 1.17.\*, 1.18.\*, 1.19.\* | Mysql | StarRocks,Doris,Paimon,Kafka | 仅 flink-cdc 3.1.1 支持 flink 1.19.* | +| 3.2.x | 1.17.\*, 1.18.\*, 1.19.\* | Mysql | StarRocks,Doris,Paimon,Kafka,ElasticSearch | | + ## Develop Your Own Connector diff --git a/docs/content/docs/connectors/pipeline-connectors/overview.md b/docs/content/docs/connectors/pipeline-connectors/overview.md index 8fbf7403d44..19568673182 100644 --- a/docs/content/docs/connectors/pipeline-connectors/overview.md +++ b/docs/content/docs/connectors/pipeline-connectors/overview.md @@ -44,10 +44,11 @@ definition. ## Supported Flink Versions The following table shows the version mapping between Flink® CDC Pipeline Connectors and Flink® -| Flink® CDC Version | Flink® Version | Pipeline Source | Pipeline Sink | Notes | -|:-----------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------:|:-----------------:|:-------------------------------:|:-------------------------------------------:| -| 3.0.x | 1.17.\*, 1.18.\* | Mysql | StarRocks,Doris | | -| 3.1.x | 1.17.\*, 1.18.\*, 1.19.\* | Mysql | StarRocks,Doris,Paimon,Kafka | only flink-cdc 3.1.1 support flink 1.19.* | +| Flink® CDC Version | Flink® Version | Pipeline Source | Pipeline Sink | Notes | +|:-----------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------:|:-----------------:|:------------------------------------------:|:-----------------------------------------:| +| 3.0.x | 1.17.\*, 1.18.\* | Mysql | StarRocks,Doris | | +| 3.1.x | 1.17.\*, 1.18.\*, 1.19.\* | Mysql | StarRocks,Doris,Paimon,Kafka | only flink-cdc 3.1.1 support flink 1.19.* | +| 3.2.x | 1.17.\*, 1.18.\*, 1.19.\* | Mysql | StarRocks,Doris,Paimon,Kafka,ElasticSearch | | ## Develop Your Own Connector diff --git a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/resources/docker/server/my.cnf b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/resources/docker/server/my.cnf index e6ba682c94f..627e1443fa6 100644 --- a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/resources/docker/server/my.cnf +++ b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/resources/docker/server/my.cnf @@ -36,6 +36,8 @@ skip-name-resolve #socket=/var/lib/mysql/mysql.sock secure-file-priv=/var/lib/mysql user=mysql +binlog_transaction_compression=on +binlog_transaction_compression_level_zstd=3 # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0