Skip to content
/ flinkx Public
forked from DTStack/chunjun

Based on Apache Flink. support data synchronization/integration and streaming SQL computation.

License

Apache-2.0, Unknown licenses found

Licenses found

Apache-2.0
LICENSE
Unknown
license.txt
Notifications You must be signed in to change notification settings

bearlly/flinkx

 
 

Repository files navigation

FlinkX

License

English | 中文

Communication

  • We are recruiting Big data platform development engineers.If you want more information about the position, please add WeChat ID [ysqwhiletrue] or email your resume to sishu@dtstack.com.

  • We use DingTalk to communicate,You can search the group number [30537511] or scan the QR code below to join the communication group

Introduction

FlinkX 1.12 New Features

FlinkX is a data synchronization tool based on Flink. FlinkX can collect static data, such as MySQL, HDFS, etc, as well as real-time changing data, such as MySQL binlog, Kafka, etc. At the same time, FlinkX is also a computing framework that supports all the syntax and features of native FlinkSql , And provide a large number of cases. FlinkX currently includes the following features:

  • Most plugins support concurrent reading and writing of data, which can greatly improve the speed of reading and writing;

  • Some plug-ins support the function of failure recovery, which can restore tasks from the failed location and save running time; Failure Recovery

  • The source plugin for relational databases supports interval polling. It can continuously collect changing data; Interval Polling

  • Some databases support opening Kerberos security authentication; Kerberos

  • Limit the reading speed of source plugins and reduce the impact on business databases;

  • Save the dirty data when writing data;

  • Limit the maximum number of dirty data;

  • Multiple running modes: Local,Standalone,Yarn Session,Yarn Per;

  • Synchronization tasks support transformer operations that execute flinksql syntax;

  • sql task support is shared with flinkSql's own connectors;

The following databases are currently supported:

Database Type Source Sink Lookup
Batch Synchronization MySQL doc doc doc
TiDB reference mysql reference mysql
Oracle doc doc doc
SqlServer doc doc doc
PostgreSQL doc doc doc
DB2 doc doc doc
ClickHouse doc doc doc
Greenplum doc doc
KingBase doc doc
MongoDB doc doc doc
SAP HANA doc doc
ElasticSearch7 doc doc doc
FTP doc doc
HDFS doc doc
Stream doc doc
Redis doc doc
Hive doc
Solr doc doc
File doc
Stream Synchronization Kafka doc doc
EMQX doc doc
MySQL Binlog doc
Oracle LogMiner doc
Sqlserver CDC doc
Postgres CDC doc

Quick Start

Please click Quick Start

General Configuration

Please click General Configuration

Statistics Metric

Please click Statistics Metric

Iceberg

Please click Iceberg

Kerberos

Please click Kerberos

Questions

Please click Questions

How to contribute FlinkX

Please click Contribution

License

FlinkX is under the Apache 2.0 license. See the LICENSE file for details.

About

Based on Apache Flink. support data synchronization/integration and streaming SQL computation.

Resources

License

Apache-2.0, Unknown licenses found

Licenses found

Apache-2.0
LICENSE
Unknown
license.txt

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Java 98.6%
  • Other 1.4%