Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Umbrella] Add sort connectors which support Flink 1.15 #8336

Open
14 of 16 tasks
Tracked by #2154
EMsnap opened this issue Jun 27, 2023 · 5 comments
Open
14 of 16 tasks
Tracked by #2154

[Umbrella] Add sort connectors which support Flink 1.15 #8336

EMsnap opened this issue Jun 27, 2023 · 5 comments
Labels
good first issue Good for newcomers stage/stale Issues or PRs that had no activity for a long time type/umbrella

Comments

@EMsnap
Copy link
Contributor

EMsnap commented Jun 27, 2023

Describe the proposal

Currently there are two sort connectors that can be running on both flink1.13 and flink1.15 which are starrocks and PostGreSQL

We should provide flink 1.15 support on all connectors that can only run on flink 1.13

They are :

  1. mysql-cdc
  2. elasticSearch
  3. hdfs
  4. iceberg
  5. jdbc
  6. kafka
  7. pulsar
  8. tubemq
  9. mongoDb-cdc
  10. sqlServer-cdc
  11. redis
  12. doris
  13. hbase
  14. hudi
  15. kudu
  16. oracle-cdc
  17. hive

How to tackle down

Take hudi connector as an Example

  1. Please make sure that you have read and understand this post : https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sourcessinks/
  2. Please make sure that you have read and understand every line of code in hudi connector on flink 1.13 (If things are not clear to you please present a issue or discussion on https://github.com/apache/inlong)
  3. Most of the systems have their flink connectors on their github page, for instance hudi (https://github.com/apache/hudi/blob/master/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java), we will import this dependency and add our inlong-specific features (metric, audit) on them.
  4. Make sure you add license after modify the TableFactory in /licenses/inlong-sort-connectors
  5. Run some tests on flink1.15 and make sure your connector works
  6. Modify inlong-distribution and make sure the connectors will be compiled just fine

Make reference for these two connectors:
#8235
#8193

Task list

InLong Component

InLong Sort

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

@EMsnap EMsnap added this to the 1.8.0 milestone Jun 27, 2023
@EMsnap EMsnap changed the title [Umbrella] Add sort connector on Flink 1.15 [Umbrella] Add sort connectors on Flink 1.15 Jun 27, 2023
@EMsnap EMsnap changed the title [Umbrella] Add sort connectors on Flink 1.15 [Umbrella] Add sort connectors which support Flink 1.15 Jun 27, 2023
@dockerzhang dockerzhang mentioned this issue Jun 28, 2023
6 tasks
@hnrainll
Copy link
Contributor

hi, @EMsnap i want add kafka connector on flink 1.15.
The issue I submitted is #8358

@dockerzhang
Copy link
Contributor

@hnrainll thanks, #8358 was assigned to you.

@fancycoderzf
Copy link
Contributor

I want add TubeMQ connector on flink 1.15.
The issue I submitted is #8922

Copy link

This issue is stale because it has been open for 60 days with no activity.

@github-actions github-actions bot added the stage/stale Issues or PRs that had no activity for a long time label Feb 27, 2024
@github-actions github-actions bot removed the stage/stale Issues or PRs that had no activity for a long time label Mar 21, 2024
Copy link

This issue is stale because it has been open for 60 days with no activity.

@github-actions github-actions bot added the stage/stale Issues or PRs that had no activity for a long time label Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers stage/stale Issues or PRs that had no activity for a long time type/umbrella
Projects
None yet
Development

No branches or pull requests

4 participants