Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove pulsar sink support #7087

Closed
Rustin170506 opened this issue Sep 16, 2022 · 5 comments · Fixed by #6983
Closed

Remove pulsar sink support #7087

Rustin170506 opened this issue Sep 16, 2022 · 5 comments · Fixed by #6983
Labels
area/ticdc Issues or PRs related to TiCDC. component/sink Sink component. type/enhancement The issue or PR belongs to an enhancement.

Comments

@Rustin170506
Copy link
Member

Pulsar is in an unmaintained state, with no unit tests, no integration tests, and no support for new features. So we decided to remove it.

StreamNative officially provides Kafka-on-Pulsar (https://github.com/streamnative/kop) to give Pulsar the ability to support the native Kafka protocol, so the TiCDC Kafka Sink can write data to Pulasr with the power of kop. .

@Rustin170506 Rustin170506 added component/sink Sink component. type/enhancement The issue or PR belongs to an enhancement. area/ticdc Issues or PRs related to TiCDC. labels Sep 16, 2022
ti-chi-bot pushed a commit that referenced this issue Sep 16, 2022
@Jiangrb
Copy link

Jiangrb commented Jan 2, 2023

@hi-rustin

After integrating the kop, the ticdc does not work, and the prompt the error blow :

Error: [CDC:ErrKafkaNewSaramaProducer]new sarama producer: [CDC:ErrKafkaBrokerConfigNotFound]cannot find the `message.max.bytes` from the broker's configuration

I can not found anywhere to config the property of message.max.bytes in kop , so I tried to find the code that describing the config of kafka in kop ,then I found the method 'describeConfigsAsync' in this link kop AdminManager.java
image

It seems that kop does not return the value of the attribute to message.max.bytes, but Ti-CDC required it

@Jiangrb
Copy link

Jiangrb commented Jan 3, 2023

Finally , I modified the code in this AdminManager.java and rebuilded kop, then it works !

that's what I added :

changes

@Kevinzhuo
Copy link

Finally , I modified the code in this AdminManager.java and rebuilded kop, then it works !

that's what I added :

changes

@Jiangrb
I added message.max.bytes, but ticdc still reports this error. Is there any other way to solve it?

@reAsOn2010
Copy link
Contributor

I think Pulsar is actively maintained now? https://github.com/apache/pulsar/actions

@Rustin170506
Copy link
Member Author

I think Pulsar is actively maintained now? https://github.com/apache/pulsar/actions

I meant pulsar sink.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ticdc Issues or PRs related to TiCDC. component/sink Sink component. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants