TiCDC sink initialization should check topic or broker's max message bytes configuration for all kinds of protocols.

[3AceShowHand]

What did you do?

In v4.0.16, if use non open-protocol, such as canal-json, canal, avro to initialize the Kafka sink, changefeed may meet message too large error from the Sarama producer.

Furthermore, the TiCDC cluster was upgraded from v4.0.14 to v4.0.16, we should also consider upgrade compatibility, which means old existing changefeeds should boost and run normal.

What did you expect to see?

changefeed works normally, no matter which kind of protocol.

What did you see instead?

[CDC:ErrKafkaAsyncSendMessage]kafka: Failed to produce message to topic cdc-eps_rt_ods-ods_market_flw_lx_app_log_prod: kafka server: Message was too large, server rejected it to avoid allocation error."]

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

TiCDC version (execute cdc version):

v4.0.16