Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Umbrella][Connector] New SeaTunnel API Connectors #1946

Closed
Hisoka-X opened this issue May 25, 2022 · 195 comments · Fixed by #4164
Closed

[Umbrella][Connector] New SeaTunnel API Connectors #1946

Hisoka-X opened this issue May 25, 2022 · 195 comments · Fixed by #4164
Labels
connectors-v2 good first issue good first issue help wanted volunteer wanted

Comments

@Hisoka-X
Copy link
Member

Hisoka-X commented May 25, 2022 •
edited
Loading

Please Move To #3018

No Connector priority difficulty Source/Sink Contributer Status Issue/PR
1 Console high low Sink @ruanwenjun Done #1864
2 Fake high low Source @ruanwenjun Done #1864
3 Doris high high Source @2013650523 #2536
4 Doris high high Sink @hk-lrzy #2586
5 Druid high middle Source @guanboo #2937
6 Druid high middle Sink @guanboo #2937
7 ElasticSearch high high Source @iture123 #2821
8 ElasticSearch high high Sink @iture123 Done #2330
9 ClickHouse high high Source @Hisoka-X Done #2051
10 ClickHouse high high Sink @Hisoka-X Done #2051
11 JDBC high high Source @ic4y Done #2048
12 JDBC high high Sink @ic4y Done #2048
13 FeiShu low low Source @TyrantLucifer
14 FeiShu low low Sink @TyrantLucifer Done
15 File(Local) high middle Source @TyrantLucifer Done
16 File(Local) high middle Sink @EricJoy2048 Done #2117
17 File(S3) high middle Source @TyrantLucifer
18 File(S3) high middle Sink @TyrantLucifer
19 File(HDFS) high high Source @TyrantLucifer Done
20 File(HDFS) high high Sink @EricJoy2048 Done
21 File(OSS) high middle Source @TyrantLucifer Done
22 File(OSS) high middle Sink @TyrantLucifer Done
23 Hudi high high Source @Emor-nj Done #2147
24 Hudi high high Sink @Emor-nj
25 Iceberg high middle Source @hailin0 Done #2615
26 Iceberg high middle Sink @s7monk
27 Kafka high middle Source @Hisoka-X Done
28 Kafka high middle Sink @ruanwenjun Done
29 Kudu high high Source @2013650523 Done #2254
30 Kudu high high Sink @2013650523 Done #2254
31 MongoDB high middle Source @wuchunfu Done
32 MongoDB high middle Sink @wuchunfu Done
33 Neo4j middle middle Source @getChan #2777
34 Neo4j middle middle Sink @getChan Done #2434
35 Phoenix middle middle Source @531651225 Done #2499
36 Phoenix middle middle Sink @531651225 Done #2499
37 Redis high middle Source @TyrantLucifer Done
38 Redis high middle Sink @TyrantLucifer Done
39 Socket high low Source @zhuangchong Done #1999
40 Socket high low Sink @531651225 Done #2549
41 [JDBC]Tidb high low Source @xbkaishui Abandonment
42 [JDBC]Tidb high low Sink @xbkaishui Abandonment
43 Webhook high middle Sink @TyrantLucifer Done #2348
44 InfluxDB middle middle Source @531651225 #2697
45 InfluxDB middle middle Sink @Jellal-HT
46 Pulsar high middle Source @ashulin Done #1984
47 Pulsar high middle Sink @FlechazoW
48 Email middle low Sink @2013650523 Done #2304
49 Assert low low Sink @lhyundeadsoul Done
50 Redshift high high Source
51 Facebook Marketing middle middle Source
52 HubSpot middle middle Source
53 Instagram middle middle Source
54 Bing ADs middle middle Source
55 Google Analytics middle middle Source @MRYOG
56 Intercom middle middle Source
57 Zendesk middle middle Source
58 TikTok Marketing middle middle Source
59 Salesforce middle middle Source
60 LinkedIn Ads middle middle Source
61 Stripe middle middle Source
62 Sentry middle middle Sink @Saintyang done #2244
63 DingTalk low low Source @MRYOG #2757
63 DingTalk low low Sink @MRYOG Done #2257
64 IoTDB high high Source @CalvinKirs Done
65 IoTDB high high Sink @hailin0 Done
66 TD-engine middle middle Source @lhyundeadsoul #2707
67 TD-engine middle middle Sink @lhyundeadsoul
68 HBase high high Source @zhuzhengjun01
69 HBase high high Sink @zhuzhengjun01
70 Hive high high Source @CalvinKirs Done
71 Hive high high Sink @EricJoy2048 Done
72 HTTP high middle Source @zhuangchong Done #2012
72 StarRocks high high Sink @tonyDong-code
73 StarRocks high high Source @wangw9420
74 ADB PostgreSQL high low Sink @etcZYP
75 Greenplum high high Source @hailin0 Done #2429
76 Greenplum high high Sink @hailin0 Done #2429
77 OceanBase middle high Sink @silenceland
78 OceanBase middle high Source @silenceland
79 DB2 high high Sink @laglangyue #2410
80 DB2 high high Source @laglangyue #2410
81 BigSource middle middle Source
82 Github low low Source @MonsterChenzhuo
83 Enterprise WeChat low low Sink @531651225 Done #2412
84 Slack middle low Sink @Charlie17Li
85 Databricks Lakehouse high high Sink
86 Snowflake middle high Sink
87 Snowflake middle high Source
88 [Jdbc]Sql-Server middle low Sink @liugddx Done #2646
89 [Jdbc]Sql-Server middle low Source @liugddx Done #2646
90 [Jdbc]Oracle high low Sink @liugddx #2550
91 [Jdbc]Oracle high low Source @liugddx #2550
92 [JDBC]Rds high low Sink @s7monk #2829
93 [JDBC]Rds high low Source @s7monk #2829
94 [JDBC]SqlLite middle low Sink @maruko-code
95 [JDBC]SqlLite middle low Source @Caribbeanz
96 [JDBC]DM(达梦) middle low Source @laglangyue done #2377
97 [JDBC]DM(达梦) middle low Sink @laglangyue done #2377
98 Cassandra middle middle Sink @bigdataf
99 Cassandra middle middle Source @bigdataf
100 [File]excel middle low Sink @Bingz2 #2585
101 [File]excel middle low Source @MonsterChenzhuo
102 [File]JSON low low Sink @hailin0 Done
103 [File]JSON low low Source @TyrantLucifer Done
104 MaxCompute middle middle Source @longer-jl
105 MaxCompute middle middle Sink @longer-jl
106 TDSql middle middle Source @dzzxjl
107 OpenMLDB middle high Source @TyrantLucifer
107 OpenMLDB middle high Sink @dlimeng
108 Ftp middle middle Sink @chessplay done #2774
109 Ftp middle middle Source @guanboo done #2774
110 GaussDB middle middle Source @Builder34
111 GaussDB middle middle Sink @Builder34
110 Teradata middle middle Source
111 Teradata middle middle Sink
112 SFTP middle middle Sink @TyrantLucifer
113 SFTP middle middle Source @TyrantLucifer
114 DataHub middle middle Source @selectbook
115 DataHub middle middle Sink @chessplay done
116 SAP HANA middle middle Source
117 SAP HANA middle middle Sink
118 Flink Table Store high high Sink @iture123
119 Flink Table Store high high Source @zhaomin1423
120 Vertica middle middle Source
121 Vertica middle middle Sink
122 Kylin middle middle Source @531651225
123 Kylin middle middle Sink @TaoZex
124 Neocrm middle middle Source
125 TiDB middle middle Source @Xuxiaotuan #2830
126 TiDB middle middle Sink @Xuxiaotuan #2830
127 Sentry middle middle Source
128 PolarDB middle middle Source
129 PolarDB middle middle Sink
130 PolarDB-X middle middle Source
131 PolarDB-X middle middle Sink
132 AnalyticDB middle middle Sink
133 TDSQL middle middle Sink
134 SequoiaDB middle middle Sink
135 TcaplusDB middle middle Source
136 TcaplusDB middle middle Sink
137 GoldenDB middle middle Source
138 GoldenDB middle middle Sink
139 AntDB middle middle Source
140 AntDB middle middle Sink
141 OushuDB middle middle Sink
142 SUNDB middle middle Source
143 SUNDB middle middle Sink
144 UXDB middle middle Source
145 UXDB middle middle Sink
146 DolphinDB middle middle Source
147 DolphinDB middle middle Sink
148 RapidsDB middle middle Source
149 RapidsDB middle middle Sink
150 GreatDB middle middle Source
151 GreatDB middle middle Sink
152 CirroData middle middle Sink
153 Nebula middle middle Sink
154 Gbase 8a middle middle Source
155 KunlunDB middle middle Sink
156 Percona middle middle Source
157 Percona middle middle Sink
158 Splunk middle middle Source
159 Splunk middle middle Sink
160 Amazon DynamoDB middle middle Source
161 Amazon DynamoDB middle middle Sink
162 Microsoft Azure SQL Database middle middle Source
163 Microsoft Azure SQL Database middle middle Sink
164 Neo5j middle middle Source
165 Neo5j middle middle Sink
166 Solr middle middle Sink
167 BigQuery middle middle Source
168 BigQuery middle middle Sink
169 SAP Adaptive Server middle middle Source
170 SAP Adaptive Server middle middle Sink
171 Microsoft Azure Cosmos DB middle middle Source
172 Microsoft Azure Cosmos DB middle middle Sink
173 PostGIS middle middle Source
174 PostGIS middle middle Sink
175 Couchbase middle middle Sink
176 Vika middle middle Sink
177 Gitlab low low Source
@EricJoy2048
Copy link
Member

EricJoy2048 commented May 27, 2022 •
edited
Loading

Great job!
I can work for Hive Sink Connector.

@ashulin
Copy link
Member

ashulin commented May 27, 2022

I can work for Apache pulsar source connector.

@Hisoka-X Hisoka-X pinned this issue May 27, 2022
@ic4y
Copy link
Contributor

ic4y commented May 27, 2022

Great. Can you assign me to MongoDB connector.

@qigengwei
Copy link

Good job!

@ic4y
Copy link
Contributor

ic4y commented May 27, 2022

Sorry i plan to do JDBC Sink first. Please reassign JDBC Sink to me

@ruanwenjun ruanwenjun mentioned this issue May 27, 2022
Merged
3 tasks
@wolfboys
Copy link
Member

wolfboys commented Jun 7, 2022

Hbase, Redis thx.

@EricJoy2048
Copy link
Member

Since most of the work in hive's sink connector is the writing of HDFS files, which is very similar to the work in file sink, I also complete the file sink connector.

@davidzollo davidzollo changed the title [Umbrella] [Feature] New SeaTunnel API Connector [Feature][Connector] New SeaTunnel API Connector Jun 8, 2022
@zhuangchong
Copy link
Contributor

socket source, thx.

#1999

@zhuangchong
Copy link
Contributor

Http source thx.

ic4y pushed a commit to ic4y/incubator-seatunnel that referenced this issue Jun 12, 2022
Add SeaTunnel jdbc sink (apache#1946)
34ecdd5
@ic4y ic4y mentioned this issue Jun 12, 2022
Merged
7 tasks
CalvinKirs pushed a commit that referenced this issue Jun 13, 2022
@ic4y @Hisoka-X
[api-draft][connector] Add SeaTunnel jdbc sink (#1946) (#2009)
890211c 
* Add SeaTunnel jdbc sink (#1946)

* fix code style err

* fix use deprecated method

Co-authored-by: liuli <liuli@analysys.com.cn>
Co-authored-by: Hisoka <1095948736@qq.com>
@ic4y
Copy link
Contributor

ic4y commented Jun 13, 2022

jdbcSink is done, please assign jdbcSource to me. Thanks

@zhuangchong zhuangchong mentioned this issue Jun 14, 2022
Merged
3 tasks
@legendtkl
Copy link
Contributor

Socket sink

@wuchunfu
Copy link
Member

MongoDB thx

@davidzollo
Copy link
Contributor

good job, has assigned to you

MongoDB thx

@davidzollo davidzollo moved this to Todo in SeaTunnel Roadmap Jun 18, 2022
ic4y added a commit to ic4y/incubator-seatunnel that referenced this issue Jun 21, 2022
@ic4y
Add SeaTunnel jdbc sink (apache#1946)
9810016
@ic4y ic4y mentioned this issue Jun 21, 2022
Merged
3 tasks
@hk-lrzy
Copy link
Contributor

hk-lrzy commented Jun 22, 2022

i can work for doris sink first. can you assign it to me?

@iture123
Copy link
Contributor

iture123 commented Jun 22, 2022 •
edited
Loading

I can work for ElasticSearch Source and Sink Connector.

it is too dependent now,because including transport client、RestHighLevelClient and RestLowLevelClient in Seatunnel.We can use RestLowLevelClient to support all versions of ElastichSearch.

@EricJoy2048 EricJoy2048 mentioned this issue Jun 23, 2022
Closed
@EricJoy2048
Copy link
Member

支持ShardingSphere 分库分表自动同步

Can you introduce the idea?

从ShardingSphere 获取底层数据库和表的信息，并行方式同步数据，例如多个不同数据库实例，可以并行方式读写数据，需要考虑表分片设置等信息。

Ok, I will add ShardingSphere Source and ShardingSphere Sink connector.

@EricJoy2048
Copy link
Member

Hi, everyone. We are ready to open a new round of connector tasks. Next, the community will complete the following tasks:

  1. Find the connector that has been assigned but has not submitted pr within 30 days and reset them to allocatable status(If you already work for it and plan to push the pr, please tell me).
  2. Close this issue and create a new issue.

@tonyDong-code
Copy link

Hi, everyone. We are ready to open a new round of connector tasks. Next, the community will complete the following tasks:

1. Find the connector that has been assigned but has not submitted pr within 30 days and reset them to allocatable status(If you already work for it and plan to push the pr, please tell me).

2. Close this issue and create a new issue.

StarRocks Sink ConnectorV2 is createing PR, thanks.

@EricJoy2048
Copy link
Member

Hi, everyone. We are ready to open a new round of connector tasks. Next, the community will complete the following tasks:

1. Find the connector that has been assigned but has not submitted pr within 30 days and reset them to allocatable status(If you already work for it and plan to push the pr, please tell me).

2. Close this issue and create a new issue.

StarRocks Sink ConnectorV2 is createing PR, thanks.

Thanks, I get it.

@EricJoy2048
Copy link
Member

@zhuzhengjun01 Hi, What's the nwes about Hbase Source and Sink connector? Do you need some help?

@EricJoy2048
Copy link
Member

Hello, I am interested in Slack Sink, but I am not clear about what data needs to be written into Slack. I just read the document(slack jdk for Java), and it provides many interfaces that can write data, such as sending messages in a channel, or uploading files, etc.

Can I imitate DingTalk Sink and only implement the funcion for sending messages in the first version?

Hi, @Charlie17Li What's the news about slack sink connector? I will close this issue and create another one. Please tell me If you plan to submit a pr about slack sink connector, I will assign it to you again.

Thank's for your contribution.

@EricJoy2048
Copy link
Member

[JDBC]SqlLite sink, tks

Hi, @maruko-code What's the news about SqlLite connector? I will close this issue and create another one. Please tell me If you plan to submit a pr about slack sink connector, I will assign it to you again.

Thank's for your contribution.

@EricJoy2048
Copy link
Member

sqllite source , thx

Hi, @Caribbeanz What's the news about SqlLite connector? I will close this issue and create another one. Please tell me If you plan to submit a pr about slack sink connector, I will assign it to you again.

Thank's for your contribution.

@EricJoy2048
Copy link
Member

bigdataf

Hi, @bigdataf What's the news about Cassandra connector? I will close this issue and create another one. Please tell me If you plan to submit a pr about slack sink connector, I will assign it to you again.

Thank's for your contribution.

@EricJoy2048
Copy link
Member

tdsql source, thx.

Hi, @dzzxjl What's the news about TDSql connector? I will close this issue and create another one. Please tell me If you plan to submit a pr about slack sink connector, I will assign it to you again.

Thank's for your contribution.

@zhuzhengjun01
Copy link

zhuzhengjun01 commented Oct 8, 2022 via email

@EricJoy2048
Copy link
Member

I'm sorry for the slow progress of the task.If possible,I will continue to promote hbase source sink,thanks！ 获取 Outlook for iOShttps://aka.ms/o0ukef
…
________________________________ 发件人: Eric @.> 发送时间: Saturday, October 8, 2022 11:35:37 AM 收件人: apache/incubator-seatunnel @.> 抄送: oliverjude @.>; Mention @.> 主题: Re: [apache/incubator-seatunnel] [Feature][Connector] New SeaTunnel API Connectors (Issue #1946) @zhuzhengjun01https://github.com/zhuzhengjun01 Hi, What's the nwes about Hbase Source and Sink connector? Do you need some help? ― Reply to this email directly, view it on GitHub<#1946 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AR6PANHOUAH4CUMAV7OFVCLWCDTYTANCNFSM5W3P5GBQ. You are receiving this because you were mentioned.Message ID: @.***>

Ok, I will assign it to you in the new issue.

@liujingwen1225
Copy link

PolarDB can be assigned to me, thx

@EricJoy2048
Copy link
Member

PolarDB can be assigned to me, thx

Ok.

@liugddx
Copy link
Member

liugddx commented Oct 8, 2022

Do gbase 8a first, please assign to me.

@liugddx liugddx mentioned this issue Oct 8, 2022
Closed
3 tasks
@wangw9420
Copy link

StarRocks source can be assigned to me, thx

@Hisoka-X
Copy link
Member Author

Hisoka-X commented Oct 9, 2022 •
edited
Loading

StarRocks source can be assigned to me, thx

@wangw9420 Please reply in ours new issue: #3018 , I will close this later. Thanks!

@Hisoka-X Hisoka-X closed this as completed Oct 9, 2022
Repository owner moved this from Todo to Done in SeaTunnel Roadmap Oct 9, 2022
@EricJoy2048 EricJoy2048 unpinned this issue Oct 9, 2022
@Xuxiaotuan Xuxiaotuan mentioned this issue Oct 24, 2022
Closed
3 tasks
@EricJoy2048 EricJoy2048 changed the title [Feature][Connector] New SeaTunnel API Connectors [Umbrella][Connector] New SeaTunnel API Connectors Nov 9, 2022
laglangyue pushed a commit to laglangyue/seatunnel that referenced this issue Jan 17, 2023
fix [Feature][Connector] New SeaTunnel API Connectors apache#1946 Add…
455012d 
… Druid Source&Sink
@2000liux
Copy link

@s7monk what is the current status of iceberg sink connectors?
@EricJoy2048 @Hisoka-X @liugddx Can I try this feature, can it be assigned to me?

@2000liux 2000liux mentioned this issue Jan 19, 2023
Closed
3 tasks
laglangyue pushed a commit to laglangyue/seatunnel that referenced this issue Jan 21, 2023
fix [Feature][Connector] New SeaTunnel API Connectors apache#1946 Add…
d00acf5 
… Druid Source&Sink
laglangyue pushed a commit to laglangyue/seatunnel that referenced this issue Feb 4, 2023
@laglangyue
fix [Feature][Connector] New SeaTunnel API Connectors apache#1946 Add…
d3512d2 
… Druid Source&Sink
laglangyue pushed a commit to laglangyue/seatunnel that referenced this issue Feb 10, 2023
@laglangyue
fix [Feature][Connector] New SeaTunnel API Connectors apache#1946 Add…
f897ccd 
… Druid Source&Sink
@MonsterChenzhuo MonsterChenzhuo mentioned this issue Feb 19, 2023
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
connectors-v2 good first issue good first issue help wanted volunteer wanted
Projects
SeaTunnel Roadmap
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Feature][ConnectorV2]add file excel sink and source MonsterChenzhuo/incubator-seatunnel