Skip to content

Releases: DataLinkDC/dinky

Dinky v0.7.4

20 Aug 16:27
327a90b
Compare
Choose a tag to compare

Feature:
[Feature-1930][client] Analyze custom functions from Flink SQL
[Feature-2126][connector] Add SQLServer Simple Connector in Flink 1.16
[Feature][cdc] Doris auto create table in CDCSOURCE task

Fix:
[Fix][metadata] Fix the issue of no data type when MySQL automatically generates DDL statements
[Fix][cdc] Fix issues with CDCSOURCE
[Fix-1958][cdc] Fix data misalignment occurs after synchronizing data hen PK do not declare at the table head
[Fix-2087][UDF] Fix udf cannot run on flink1.16 and flink 1.17
[Fix][metadata] Fix clickhouse Nullable columns
[Fix-2147][metadata] Fix missing precision and scale when generating MySql DDL
[Fix][metadata] Fix mssql tinyint Type conversion error
[Fix-2176][admin] Fix unknown jobid bug
[Fix-2108][admin] Fix updating the savepoint policy of the job during checkpoint recovery

Optimization & Improve:
[Optimization][cdc] Optimize CDCSOURCE in Flink 1.16
[Optimization-2089][flink] Upgrade Flink version

Contributors:
@aiwenmo
@HamaWhiteGG
@leeoo
@LiuHao0606
@seawenc
@tan-zhuo
@xiaofan2022
@zhengwenhao2017

Dinky v0.7.3

19 Apr 14:27
cdcd50c
Compare
Choose a tag to compare

Feature:
[Feature-1709][core] Support use FlinkSQL CTE to preview data
[Feature-1774][flink] Support Apache Flink 1.17

Fix:
[Fix-797][web] Fix no fragment parameters were passed when getting lineage
[Fix-1731][gateway] Fix deploy flink k8s session error
[Fix-1787][metadata] Fix error in automatic table creation using CDCSOURCE Mysql field unsigned type problem
[Fix-1805][admin] Fix NPE occurred when publishing the job
[Fix][cdcsource] Fix data sending and receiving issues with synchronizing multiple Sinks in CDCSOURCE
[Fix][SqlSegment] Fix the issue of SQL configuration parsing failure in specific scenarios
[Fix-1784][admin] Fix task is deleted but task instances and history are not deleted consecutively
[Fix-1849][client] Fix CDCSOURCE kafkaSink transactionalIdPrefix NullPointerException
[Fix-1847][metadata] Fix postgreSql get DDL script error
[Fix][metadata] Fix generating MySQL DDL statements without quotes for normal default values

Optimization & Improve:
[Optimization-1715][admin] Optimize job monitor reconnect
[Optimization-1776][core,metadata] Fix typo
[Optimization-1657][client] Optimize MysqlJsonDebeziumDeserializationSchema use shaded lib in flink 1.13
[Optimization][client] Support scan.incremental.snapshot.chunk.size
[Optimization][client] Analyze lineage from the original relNode, not the optimized relNode, and add unit tests
[Optimization-1851][flink] Upgrade all versions of Flink to the latest version

Contributors:
@aiwenmo
@AntChen27
@cygnusdark
@HamaWhiteGG
@LiuHao0606

Dinky v0.7.2

28 Feb 16:52
e2982a1
Compare
Choose a tag to compare

Feature:
[Feature-1593] Add CDCSOURCE support sink catalog
[Feature-1608] Support get task lineage by taskid with openapi
[Feature-1641] Add Flink row level permissions
[Feature-1647] Yarn session support applicationId and resource manager config
[Feature-1547] Flink Jar support alert

Fix:
[Fix-1615] Fix submit task error but return succeed when using dolphinscheduler
[Fix-1624] Fix lineage api in data explore module returns empty result
[Fix-1652] Fix the task using the FlinkSQLEnv cannot obtain lineage
[Fix-1662] Fix OpenApi notLoginException
[Fix] Fix wrong Python type
[Fix-1672] Fix failed to get metadata due to jdbc session timeout in CDCSOURCE
[Fix-1638] Fix dlink web could not correct display Boolean type data
[Fix-1686] Fix ZipFile got wrong in K8s native app mode

Optimization & Improve:
[Optimization-1631] Optimize auto.sh and remove useless files
[Optimization-1666] Add job reconnecting status and optimize job monitor
[Optimization-1682] Optimize setting named tinyInt1isBit for mysql metadata

Contributors:
@aiwenmo
@liaotian1005

Dinky v0.7.1

26 Jan 04:07
02f2b95
Compare
Choose a tag to compare

Fix:
[Fix-1494] Fix tinyint(1) convet error
[Fix-1575] Fix unknown column 'tenant_id' in 'field list'
[Fix] Fix i18n
[Fix] Fix UDF function case insensitive
[Fix] Fix the bug that the schema name contains middleLine
[Fix-1556] Fix job submission succeeded but failed
[Fix-1503] Fix use yarn application mode to submit job reported null pointer exception
[Fix-1563] Fix NPE caused by querying data
[Fix] Fix add k8s ClusterConfiguration npe
[Fix] Fix parse type error when use bigint types

Optimization & Improve:
[Optimization-1548] Optimize datasource registration should can save config before validate
[Optimization] Optimize clear console and add clear console at right click
[Optimization] Update org.apache.httpcomponents:httpclient 4.4.1 to 4.5.13

Contributors:
@aiwenmo
@leechor
@zackyoungh
@zhoumengyks
@zhu-mingye

Dinky v0.7.0

24 Nov 15:49
e8b8e6b
Compare
Choose a tag to compare

Feature:
Supports Apache Flink 1.16.0
Add build java udf jar
Support Flink session mode to automatically load udf jars
Support Flink per-job mode to automatically load udf jars
Support Flink application mode to automatically load udf jars
Support python udf online development
Support scala udf online development
Support custom k8sapp submit at studio
Add file upload in flinkjar task
Add Column Lineage By Logic Plan in Flink all version
Flink JDBC support data filter
Flink JDBC scan partition support datetime
Add multi tenant management
Add select tenant at login
Add dolphinScheduler auto create task
Add system log console
Add execution progress console
Add Flink UDF template
Add Presto data source
Add the function of deleting job tree directory level
Add new sink connector 'datastream-doris-ext' for CDCSOURCE to support metadata writing
DataStudio metadata add refesh button
Add database copying
Add frontend internationalization
Add backend internationalization
Add K8S auto deploy application
Add local environment with web UI
Add FlinkSQL built-in date global variable
Add Kerberos Verification on session model
Add CDCSOURCE supports Doris schema evolution
Add user can change password
Add kill cluster
Add deploy session cluster by cluster configuration
Add jmx monitoring
Add clear console

Fix:
Resolve the exception when starting from the specified savepoint
Fix StarRocks databases display
Fix query exceptions caused by the system's failure to automatically clean up the selected table information when switching data sources
Fix Exception in obtaining SQLDDL of view
Fix invalid oracle validationQuery
Fix failed to get schema of PG database metadata
Fix kafka sink properties not enable
Fix jobConfig useAutoCancel parameter pass wrong value
Fix job monitoring bugs caused by multi tenancy
Fix the deletion of a cluster instance causes existing tasks to fail to stop
Fix application mode missing database variable
Fix continuous click task item will open multiply tabs problem
Fix cdcsource kafka product transactionalIdPrefix disable
Fix cluster display nothing when alias is not set
Fix task version query error
Fix The status of per-job and application always is unknown when the job completes
Fix guide page link error
Fix open API does not have tenant
Fix switch editor tab doesn't work

Optimization & Improve:
Add parameter configuration of MySQL CDC
Optimizate datastream kafka-json and datastream starrocks
Support metadata cache to memory or redis
Add uuid after doris label prefix
Optimize tenant selection
Change resource center to authentication center
Add spotless plugin
Optimize SQL files and differentiate versions by directory
Improve the automatic creation of MySQL tables
Improve postgres metadata information
Improve the generation of postgres table building statements
Optimizate Flink Oracle Connector
Optimizate maven assembly and profile
Compatible java 11
Remove duplicated init of datasource
Upgrade mysql-connector-java to 8.0.28
Upgrade Flink 1.14.5 to 1.14.6
Upgrade Guava and Lombok Version
Upgrade jackson and sa-token version
Optimize lineage to support watermark and udf and LOCALTIMESTAMP and cep and ROW_NUMBER
The authentication center is not visible under non-admin users

Document:
Optimizate readme.md
Update website
Optimize deploy doc
Add flink metrics monitor and optimize deploy document

Contributors:
@admxj
@aiwenmo
@billy-xing
@boolean-dev
@chengchuen
@czy868
@dzygcc
@Forus0322
@gujincheng
@HamaWhiteGG
@hxp0618
@ikiler
@leechor
@lewnn
@jinyanhui2008
@nylqd
@ren-jq101
@RookieGao
@siriume
@tgluon
@wellCh4n
@wfmh
@wmtbnbo
@wuzhenhua01
@zackyoungh
@zhongjingq
@zhu-mingye
@ziqiang-wang
@zq0757
@Zzih
@zzzzzzzs

Dinky v0.6.7

06 Sep 16:08
a70e020
Compare
Choose a tag to compare

Feature:
[Feature-775][admin] Add tenant implementation
[Feature-789][admin,web] One click online and offline operation
[Feature-812][admin,web] Add FragmentVariable manager && resourcecenter page
[Feature] Added NameSpaceForm PasswordForm
[Feature-823][web] Render multi tenant forms on login
[Feature-823][web] Multi tenant front end implementation
[Feature-868][common] Added github workflow to check the checkstyle, test and build of each PR
[Feature-861][metadata] Add alibaba druid connection pooling to fix jdbc multi-connection problem
[Feature-890][admin,web] Realization of user empowerment role function
[Feature-907][pom] Change Flink base version into 1.14
[Feature-905][admin,web] Implementation of global variable management
[Feature][client] Add SqlServer CDCSOURCE
[Feature-915][admin,core] Add global variables takes effect in flinksql
[Feature-923][*] Build column lineage base on flink logical plan
[Feature][client] Add posgresql CDCSOURCE
[Feature][test] Modify checkstyle to be a required item
[Feature] Add swagger api doc
[Feature] CDCSOURCE supports multi-sink
[feature][admin] File upload
[Feature-987][admin] ClusterConfig and jar add upload file
[Feature-989][metadata] Add StarRocks datasource
[Feature-946][alert] Alarm after task monitoring retry
[Feature][web] Add data development task information log details button

Fix:
[Fix][admin] Modify the problem of repeated judgment of task monitoring code
[Fix] Fix mail alert params bug
[Fix-804][admin] GetJobInfoDetail and refreshJobInfoDetail maybe return a error result
[Fix-818][connectors] Fix dlink-connector-doris-1.13 when the data is flushed with an exception and no new data is entered subsequently
[Fix-833][client] Error with SQLSinkBuilder.buildRow
[Fix] Fix execute error by flink 1.14.4
[Fix] Fix cluster submission taskId is empty
[Fix] Fix yarn per job can't release resources
[Fix] Fix multi tenant add role and delete role
[Fix] Fix dlink-conector-pulsar-1.14 can't find SubscriptionType
[Fix] Fix the jackjson problem in flink1.14 savepoint
[Fix-803][client] Fix TypeConvert-ColumnType Enumeration usage error
[Fix] The full database sync kafkasinkbuilder does not implement serialization and cause an error
[Fix-840][web] fix registration document type filter condition error
[Fix] Fix yarn perjob/application and k8s application cluster configuration
[Fix] Fix k8s application submit error and add Maximum waiting time
[Fix] Fix the banner of dlink admin app
[Fix][function] Fix udf and udtaf in Flink 1.14
[Fix][app] Fix yarn application task separator error
[Fix][admin] Failed to save job after repairing rename job
[Fix][web] No content when repairing the second pop-up of the submitted history

Optimize:
[Optimization-764][dlink-web,docs] Optimization dlink-web,docs
[Optimization-780][admin] Optimization The task version was not deleted after deleting the task
[Optimization-781][web] Optimization Overflow in job list tree after importing file
[Optimization-801][web] Optimization StudioConsole's StudioProcess
[Optimization-773][dlink-client] Optmizatiion cdcsource filter to process
[Optimization-809][Common] Added ignore of logs folder in git ignore file.
[Optimization-810][Common] Added Maven Wrapper
[Optimization-816][Common] Fix Chinese README link error and add English README
[Optimization] Optimization of multi tenant && Optimize form rendering
[Optimization] Remove sensitive info for more api
[Optimization][admin] Optimized for multi-tenant
[Optimization-819][client] CDCSOURCE with timestamp and timezone
[Optimization-849][client,executor] Replace sql separator and change default sql separator into ;\n
[Optimization][Style] Optimization code style import order
[Optimization][Git] Added .DS_Store git file ignore
[Optimization] Optimization multi tenant of delete roles && optimization multi tenant web render
[Optimization] Optimize user associated character rendering
[Optimization][Style] Added dlink-admin module code style
[Optimization][Style] Added dlink-alert module code style
[Optimization][Style] Added dlink-common module code style
[Optimization][Style] Added dlink-catalog module code style
[Optimization][Style] Added dlink-client module code style
[Optimization][Style] Added dlink-app module code style
[Optimization][metadata] Optimized connection pooling and connection creation
[Optimization][Style] Added dlink-connectors module code style
[Optimization][Style] Added dlink-core module code style
[Optimization][Style] Added dlink-daemon module code style
[Optimization][Style] Added dlink-executor module code style
[Optimization][Style] Added dlink-function and dlink-gateway module code style
[Optimization][Style] Added dlink-metadata module code style
[Optimization][license] Add a license description to the pom file
[Optimization-932][pom] Optimizate package and auto.sh by loading classpaths
[Optimization] dlink-client-hadoop add ServicesResourceTransformer
[Optimization-943][pom] Optimizate config and static dir packaging
[Optimization][style] Configure global checkstyle validation
[Optimization][client] Add sqlserver date type convert
[Optimization][metadata] Optimizate postgresql schema_name query
[Optimization-981][metadata] Doris support more syntax
[Optimization-983][client] Optimizate Doris datastream sink
[Optimization][web] Optimizate Some problems in front-end and some tips
[Optimization-882][web] Collapse all directories by default on datastudio's directory panel
[Optimization-881][datastudio] Optimizate FlinkSql explain exception message
[Optimization-1014][client] Optimizate Doris sink and type convert and upgrade flink to 1.15.2
[Optimization][flink] Upgrade Flink 1.15 to 1.15.2
[Optimization][metadata] Optimize SqlServer field type query

Document:
[Document][docs] Migrate dlink docs from dinky-website to docs
[Document] Dlink add Flink1.15 docs
[Document-769][docs] The whole database synchronization document repair
[Document-766][docs] Add import and export job docs
[Document-793][doc] Optimization some docs
[Document-835][doc] Update the home page and basic information of the document
[Document-832][doc] Add Practice Sharing of FlinkSQL Extending Redis
[Document][docs] Optimizing Deployment Documents

Contributor:
@aiwenmo
@byd-android-2017
@chrofram
@complone
@dylenWu
@dzygcc
@gaogao110
@hxp0618
@hzymarine
@ikiler
@leo65535
@lnnlab
@JanCong
@jinyanhui2008
@mengyejiang
@mydq
@rafaelxie
@sunyanqing01
@tgluon
@Toms1999
@walkhan
@wangzhonggui
@XiaoF-Liu
@zackyoungh
@zhu-mingye
@zhuangchong
@zhujunjieit
@ziqiang-wang

Dinky v0.6.6

23 Jul 15:57
c2f886a
Compare
Choose a tag to compare

Feature:
[Feature-685][web,admin] add job historyversion list of DevOps
[Feature-692][web] Add history version comparison in data development
[Feature][catalog] Add Flink MySql Catalog
[Feature-704][catalog,admin] Add default mysql catalog in FlinkSQLEnv
[Feature][connector] Added version 1.13 Doris connection is hidden by _DORIS_DELETE
[Feature][connector] Add dlink-connector-pulsar
[Feature][web,admin] Select checkpoint restart
[Future][flink] Update flink1.15 to flink1.15.1
[Feature-728][admin,web] Add meta store feature
[Feature-733][admin,web,client] Add Flink meta store info and column details
[Feature-738][*] Add and update Licenses
[Feature-750][admin,web] Add import and export tasks json

Fix:
[Fix][connctor] flink-connector-phoenix and Update PhoenixDynamicTableFactory
[Fix][admin] Fix job instance cause OOM
[Fix][bug] Update flink version and fix Flink and CDC version compatibility bug
[Fix-696][web] Fix the problem of time-consuming parsing errors of task instances
[Fix-709] [catalog] Fix catalog SPI bug and sql bug
[Fix-719][web] Fix checkpoint has error of devops && add savepoints info
[Fix-714][client] Catch translateToPlan exception in SQLSinkBuilder
[Fix-736][client] An exception occurred during the packaging process
[Fix-738][admin] Fix CopyrightUtil
[Fix-741][app] Fix application mode submit failed
[Fix-745][admin] Fix alert instance delete bug
[Fix][admin] Allow circular references config
[Fix-750][admin] Delete unreferenced classes
[Fix][admin] Fix jobhistory field null

Optimize:
[Improvement][jobplan] Prompt returned when failed to obtain the optimized flinksql execution graph
[Optimization][admin] Move sensitive info(password) from api
[Optimization-713][dlink-admin] Fix the problem of incorrect display information of deleted user login
[Optimization-719][web] Optimization checkpoints page

Contributors:
@a279780399
@aiwenmo
@Arnu-
@darren-da
@dzygcc
@Forus0322
@gaogao110
@JPengCheng
@mydq
@syyangs799
@wmtbnbo
@zhu-mingye

Dinky v0.6.5

03 Jul 16:06
3062cec
Compare
Choose a tag to compare

Feature:
[Feature][connector]Add flink connecotr phoenix 1.14
[Feature-609][metadata]Supports generating FlinkSQL with null and not null through metadata
[Feature-635][admin]Add job instance info api
[Feature-644][admin,web]Add Flink snapshot info and JobManager config in DevOps
[Feature-649][admin,web]Add TaskManager info in DevOps
[Feature-654][web]Add task info tab
[Feature-649][admin,web]Add TaskManager table and form in DevOps
[Feature-662][admin,web]Add task version history
[Feature-661][web]Add job checkpoint history
[Feature-666][client]Capture column type conversion exception details in CDCSOURCE
[Feature-668][web]Add task manager info
[Feature-663][admin,web]Add task copy feature
Fix:
[Fix-577][web]Fix datastudio datasource alias is ""
[Fix-574][web]Fix datastudio tabs close_other can not close first tab
[Fix-576][admin]Fix the exception caused by metadata switching
[Fix][gateway]Fix flinkLibPath in K8S configuration is null and cluster-id in test k8s-cluster-config is null
[Fix][common]Fix semicolon exception at the end of FlinkSql
[Fix][core]Fix K8S cluster configuration can't get custom configuration
[Fix-596][admin]Fix NPE on running yarn per-job app
[Fix-603][admin]Fix task information will be cleared when refreshing tasks that have lost connection
[Fix-140][core]Fix 'table.local-time-zone' parameter is invalid
[Fix-607][dlink-alert-email]Fix ClassNotFoundException: javax.mail.Address
[Fix-621][core]Fix can't explain 'show datatbases'
[Fix][core]Fix setParentId method space determination error
[Fix-608][client]Fix can't create a flink table when table name is a sql reserved word in CDCSOURCE
[Fix-629][core]Fix lineage of the same field name cannot be correctly resolved
[Fix][web]Fix cluster configuration page bug
[Fix-638][alert]Fix the email alarm notification cannot customize the nickname
[Fix][connector]Fix dlink-connector-phoenix-1.14 build error
[Fix-670][metadata]Fix Oracle column nullable error
[Fix-674][client]Support MySQL varbinary and binary in CDCSOURCE
Optimization & Improve:
[Optimization-575][web]Optimize tree search results highlight & & selected backgroundColor
[Optimization-605][web]Optimize the alias is empty in several places
[Optimization][client]Optimize explainSqlRecord
[Optimization-627][web]Optimization cluster instance page
[Optimization-633][web]Optimization lineage refresh
[Optimization-641][admin]Optimize job instance api
[Optimization-640][alert,web]Optimization all Alert of sendMsg

Dinky v0.6.4

05 Jun 15:56
cdda5ec
Compare
Choose a tag to compare

Feature:
[Feature-506][client] The CDCSOURCE table parameter supports line feed and column sort by primary key
[Feature-518][client] CDCSOURCE add log
[Feature-525][alert,web] Add @mobile mode of dingTalk Alert
[Feature-254][web] Add StreamGraph JSON Export function
[Feature-534][admin,web] Add task open API page
[Feature-356][web] Add datastudio help page
[Feature-318][core] Add column lineage from db sql
[Feature-545][admin] Add Task Pool to solve frequent database writes
[Feature-552][web] Hide export StreamGraphPlan button when not FlinkSQL task
[Feature-558][web] add delete button of database Manage
[Feature-568][client] Add CDCSOURCE jdbc properties and upgrade flink cdc version

Fix:
[Fix-491][admin,web] Fix refresh the status page with abnormal jitter
[Fix-499][connector] Flink oracle connector can't cast CLOB to String
[Fix-494][web] Fix the savepoint information does not change with the task information
[Fix-503][gateway,client] Compatibility of different versions of the interface ClusterClient
[Fix-509][metabase] Fix MySqlTypeConvert precision and scale is null bug
[Fix-503][client] Remove initFunctions
[Fix-518][client] Fix CDCSOURCE decimal bug
[Fix-520][core] Fix get JobPlanInfo failed
[Fix-522][client] Fix CDCSOURCE OracleCDC number can't be cast to Long
[Fix-527][web] Fix the wechat alarm instance send testMsg error
[Fix-529][web] Fix the task cannot be saved after switching tasks
[Fix-541][web] Fix the problem that the datasource && metadata does not set alias
[Fix-542][web] Fix the name of tab can't be modified when modify task name
[Fix-556][adimin,gateway] Fix flinkLibPath in K8S configuration is null
[Fix-571] [client] Fix CDCSOURCE String can't be cast to Timestamp

Optimization & Improve:
[Optimization-504][doc] Optimization init sql
[Optimization-550][assembly] Optimization package
[Optimization-562][web] Remove preset-ui
[Optimization-564][metadata] Optimization MySqlTypeConvert

Dinky v0.6.3

09 May 15:56
091c24c
Compare
Choose a tag to compare

Feature:
[Feature-422][] CDCSource sync kafka topics
[Feature-429][
] OracleCDCSource sync kafka topics
[Feature-435][client,executor] CDCSource sync doris
[Feature-442][client,executor] CDCSource sync hudi
[Feature-445][client] CDCSource sync add sink table-name rule
[Feature-447][client] CDCSource sync sql
[Feature-451][client] CDCSource sync field type convertion in Flink1.13
[Feature-461][client] CDCSource sync add sink table-name RegExp
[Feature-469][client] Add MysqlCDCSource sync extended configuration
[Feature-477][client] CDCSOURCE add pkList
[Feature-275][client] Add Flink client 1.15
[Feature-488][*] Release v0.6.3

Fix:
[Fix-427][admin] Task status error when task has stoped
[Fix-425][metadata] Fix Oracle Unsupported character set
[Fix-426][metadata] Fix Clickhouse metadata does not display
[Fix-387][web] Metadata switching error
[Fix-442][client,executor] Add Hudi SOURCE_AVRO_SCHEMA
[Fix-454][metadata] Assign to DriverConfig's name
[Fix-424][common] Modify the default value of the sqlSeparator to ';\r\n|;\n'
[Fix][app] Batch execution with yarn-app mode using the useBatchModel parameter
[Fix-457][web] Fix bug of modifying job name
[Fix-472] [client] Mysqlcdc whole database sync Hudi error
[Fix-479][docs] Fix some inaccessible documentation links
[Fix-484] [core] Fix to job plan info was executed twice

Optimization & Improve:
[Improve-434][admin] The release of Stream
[Optimization-439][client] Optimize CDCSource sync doris
[Improvement-456][dlink-doc] Fix start and stop conflicts when deploy more instances in same machine
[Optimization-459][alert-wechat,web] Optimization wechat Webhook sendMsg title
[Improvement-456][dlink-doc] Fix auto.sh LF and disable environment variables