REQTMOUT errors

[sorbra]

Description

We are using azure-event-hubs-for-kafka with Kafka clients running in Azure Kubernetes Service.

Generally it works as expected, but our clients regularly output Kafka related error messages, as illustrated in kafka error messages.txt.

Apparently these error messages do not cause any problems at the application level, and they do not result in any exceptions being raised.

Has it worked previously?

No. We have been using azure-event-hubs-for-kafka for about a year, and have seen this issue all along.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed or where adequate information has not been provided.

Please provide the relevant information for the following items:

• SDK (include version info): C# confluent-kafka-dotnet version 1.5.3
• If using Apache Kafka Java clients or a framework that uses Apache Kafka Java clients, version: N/A: The issue being raised is related to C#/dotnet core clients.
• Kafka client configuration: kafka config.txt
• Namespace and EventHub/topic name: Many different. Example: bpt-dev-samplens-ehns
• Consumer or producer failure: Error messages contain the text "producer", so I assume it is a producer issue. But, as the errors do not appear at our application code level, I can't be sure.
• Timestamps in UTC <REPLACE with e.g., Nov 7 2018 - 17:15:01 UTC>
• group.id or client.id: group.id = Many different. Example: sample-service
• Logs provided (with debug-level logging enabled if possible, e.g. log4j.rootLogger=DEBUG) or exception call stack
• Standalone repro: Willing to send scenario to repro issue
• Operating system: Azure Kubernetes Service Linux nodes
• Critical issue

If this is a question on basic functionality, please verify the following:

• Port 9093 should not be blocked by firewall ("broker cannot be found" errors)
• Pinging FQDN should return cluster DNS resolution (e.g. $ ping namespace.servicebus.windows.net returns ~ ns-eh2-prod-am3-516.cloudapp.net [13.69.64.0])
• Namespace should be either Standard or Dedicated tier, not Basic (TopicAuthorization errors)

[sorbra]

I seem to have found a solution to this issue in this very thorough article.

The fix was to add the following lines to my Consumer and Producer configurations:

                SocketKeepaliveEnable = true,
                MetadataMaxAgeMs = 180000,

I have tested the solution, and so far the error messages have gone from my logs.