Inconsistent Logging for Tombstone Messages in Elastic Sink Connector

[AbdelkaderR]

kafka-connect-elasticsearch/src/main/java/io/confluent/connect/elasticsearch/ElasticsearchSinkTask.java

Line 107 in 9e07811

reportBadRecord(record, new ConnectException("Cannot write null valued record."));

Hi everyone,
I hope this message finds you well. I am reaching out regarding an issue I encountered while working with the Elastic Sink Connector. Specifically, the problem is related to the handling of tombstone messages in the code.
Upon reviewing the code, I noticed that at line 107, an error message is logged when the connector encounters a tombstone message, irrespective of the configuration set for behavior.on.null.values. According to the current implementation, even if the configuration is set to IGNORE, an error message is still written to the logs. However, I believe the behavior should be consistent with the TombstoneHandler transform, where, if set to Ignore, a DEBUG message is written to the logs instead of an error message.

As per the TombstoneHandler documentation:

• If set to ignore, the tombstone record is ignored silently, with a DEBUG message written to log.
• If set to warn, the tombstone record is ignored silently, with a WARN message written to log.
• If set to fail, the execution fails, a DataException is thrown, and a DEBUG message is written to log.

The current implementation's tendency to log an error message instead of a DEBUG message can potentially impact the connector's performance, especially in scenarios where a large volume of tombstone messages needs to be processed. Writing an error message for each tombstone message introduces unnecessary overhead.

I kindly request a review of this issue and consideration for aligning the logging behavior with the TombstoneHandler transform, ensuring that DEBUG messages are logged when behavior.on.null.values is set to IGNORE.

Thank you for your time and attention to this matter. Please let me know if you require any further information or clarification.

Best regards,
Abdelkader