Inserts failing silently

[jonscyr]

Describe the bug

Missing events in the target table. 3 events at offsets 7407350 - 7407352 are not present in clickhouse table. The connector has attempted to write them, we could see it from the DEBUG logs, but the insert had actually failed. Found insert query in query_log for id "eed49adf-feb9-46a7-abec-e44d9bdc03c2". it failed with MEMORY_LIMIT_EXCEEDED. Shouldn't this have made the sink connector retried or put the events in a DLQ like it was configured to? Instead, I could see this in the connector's logs

|task-0] Response Summary - Written Bytes: [7854], Written Rows: [3] - (QueryId: [eed49adf-feb9-46a7-abec-e44d9bdc03c2]) 

Steps to reproduce

Not sure how to reproduce

Expected behaviour

Failed inserts should have raised exception and caused it to be retried / put in dlq as configured.

Available logs

• sink connector logs for query id eed49adf-feb9-46a7-abec-e44d9bdc03c2: https://gist.github.com/jonscyr/5f843f91f62429b5217a69007108c2c7

• records from clickhouse's system.query_logs for same query id: https://gist.github.com/jonscyr/abff0e26c122dc1b8b3d80a00dcd93e3

Configuration

https://gist.github.com/jonscyr/ef2f400a30a6b63a019d77b8a77f23b4

Environment

We have half-stack setup (CH cloud + strimzi kafka). We've been facing this issue where some batches events gets lost every 1 or two months. We have a script for validator running every day which raises this.
Our kafka-connect are running on LOG_LEVEL=DEBUG.

• Kafka-Connect version: v1.2.3
• Kafka Connect configuration: https://gist.github.com/jonscyr/6566d4f45db6dae63cb0b79a13ff127f
• Kafka version: 3.2
• Kafka environment: Strimzi 0.33.0
• OS: Linux

ClickHouse server

• ClickHouse Server version: 24.6
• CREATE TABLE statements for tables involved: https://gist.github.com/jonscyr/6573e6cb84e37eca7547349af7fa1168
• Sample data for all the table: https://gist.github.com/jonscyr/abbecbd14f56012b6a99bdca514951a6

[jonscyr]

Update: Talking with CH cloud team, they were able to reproduce this. It's because of the send_progress_in_http_headers flag. http response code is 200 even for an out of memory error. @Paultagoras

https://github.com/ClickHouse/clickhouse-kafka-connect/pull/275/files#diff-05b2bb95bf4463df8acbff731837dc199b3c100bd6d498752d82291ea3c8e0dfR177

[Paultagoras]

@jonscyr We've released a "pre-release" version of the connector (https://github.com/ClickHouse/clickhouse-kafka-connect/releases/tag/v1.2.4) that updates the underlying language client to address this, so that folks can see if this resolves their issue (it should, but a wider test is better) - the "official" release should happen next week.

[jonscyr]

thank you @Paultagoras . we've upgraded to 1.2.4. will keep this thread posted.

[Paultagoras]

thank you @Paultagoras . we've upgraded to 1.2.4. will keep this thread posted.

Hi @jonscyr any word on how it's going?