Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Avro sink and make TiCDC compatible with Kafka connect #660

Closed
3 tasks done
amyangfei opened this issue Jun 16, 2020 · 3 comments
Closed
3 tasks done

Support Avro sink and make TiCDC compatible with Kafka connect #660

amyangfei opened this issue Jun 16, 2020 · 3 comments
Assignees
@amyangfei
Labels
difficulty/medium Medium task. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/P1 The issue has P1 priority.
Milestone
v5.0.0-alpha

Comments

@amyangfei
Copy link
Contributor

amyangfei commented Jun 16, 2020 •
edited by uglyengineer
Loading

Feature Request

Is your feature request related to a problem? Please describe:

Apache Kafka provides flexible connectors mechanism, and there exist a lot of third-party connectors implementation which can receive changed logs from upstream in the Avro protocol, send data to Kafka via source connector, and the data in Kafka can be consumed by sink connector and replicated to downstream.

TODO list

  • Support writing Avro encoded Kafka Connector-compatible data when writing to Kafka.
  • Support replicating Avro schema to an open-source Schema Registry, such as Confluent Schema Registry for Kafka
  • Provide a full solution for data replication link, TiDB -> TiCDC -> Kafka -> TiDB, with Avro data from TiCDC to Kafka and using Kafka sink connector to consume data from Kafka to TiDB.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

Value

Value description

This is an important feature in third-party ecosystem

Value score

  • (TBD) / 5

Workload estimation

  • 30 person-day, still need 15 person-day from 7-28

Time

GanttStart: 2020-07-15
GanttDue: 2020-08-31
GanttProgress: 90%
@amyangfei amyangfei added help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. feature request labels Jun 16, 2020
@liuzix liuzix changed the title Support Avro sink and make TiCDC compatible with Kafka connector Support Avro sink and make TiCDC compatible with Kafka connect Jun 19, 2020
@liuzix liuzix mentioned this issue Jun 23, 2020
Merged
@amyangfei amyangfei added this to the v5.0.0 milestone Jun 23, 2020
@amyangfei amyangfei added the priority/P0 The issue has P0 priority. label Jun 23, 2020
@liuzix liuzix mentioned this issue Jul 8, 2020
Merged
@liuzix liuzix mentioned this issue Jul 15, 2020
Merged
@amyangfei amyangfei added the difficulty/medium Medium task. label Jul 29, 2020
liuzix referenced this issue in pingcap/community Jul 29, 2020
@liuzix
Update member-list.md
63ce5a5 
I am eligible for becoming a reviewer given that I fixed this issue pingcap/ticdc#660
@liuzix liuzix mentioned this issue Jul 29, 2020
Merged
kennytm referenced this issue in pingcap/community Jul 30, 2020
@liuzix
Nominate liuzix to be a reviewer in sig-tools (#269)
024875c 
* Update member-list.md

I am eligible for becoming a reviewer given that I fixed this issue pingcap/ticdc#660

* Update member-list.md

Co-authored-by: leoppro <zhaoyilin@pingcap.com>
@abbccdda
Copy link

abbccdda commented Aug 9, 2020

Is this still open?

@scsldb scsldb added priority/P1 The issue has P1 priority. and removed priority/P0 The issue has P0 priority. labels Sep 19, 2020
@scsldb
Copy link

scsldb commented Sep 19, 2020

Reason for reprioritization: development complete, testing in process, more important development tasks at the moment.

@amyangfei
Copy link
Contributor Author

amyangfei commented Nov 4, 2020 •
edited
Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amyangfei amyangfei

Labels
difficulty/medium Medium task. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/P1 The issue has P1 priority.
Projects
None yet
Milestone
v5.0.0-alpha
Development

No branches or pull requests
3 participants
@amyangfei @scsldb @abbccdda