ssl_closed error

[zmstone]

I'm have a similar problem, but now with the producers connections.

Always, the first message that I try to send I got the error: {:connection_down, {:shutdown, :ssl_closed}}

Any tip how to solve it?

Here the config:

  kafka_hosts = parse_kafka_hosts.()
  config :brod,
    clients: [
      kafka_producer: [
        endpoints: kafka_hosts,
        restart_delay_seconds: 10,
        auto_start_producers: true,
        allow_topic_auto_creation: false,
        default_producer_config: [
          retry_backoff_ms: 50,
          max_retries: 3,
          required_acks: -1,
          ack_timeout: 300,
          max_linger_ms: 10,
          max_linger_count: 5
        ]
        client_id: "producer-#{System.fetch_env!("POD_NAME")}",
        endpoints: kafka_hosts,
        ssl: [
          verify: :verify_peer,
          cacertfile: "/etc/ssl/certs/ca-certificates.crt",
          depth: 3,
          customize_hostname_check: [
            match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
          ]
        ],
        sasl: {
          "KAFKA_sasl_mechanisms" |> System.fetch_env!() |> String.downcase() |> String.to_atom(),
          System.fetch_env!("KAFKA_sasl_username"),
          System.fetch_env!("KAFKA_sasl_password")
        }
      ]
    ]

Thanks =D

Originally posted by @robsonpeixoto in #520 (comment)

[zmstone]

Hi @robsonpeixoto
It seems to be a different issue from the original thread, so I created a new issue here.
I'm not quite sure I follow the 'first' part, do you mean the producer can actually produce after a retry ?

[robsonpeixoto]

Thanks @zmstone :)

It is not a brod bug, my bad!

After some debug I figured out what's happening. The Kafka Server always close the connections that area IDLE for more than connections.max.idle.ms, but for some reason the FIN(sent when a connection is closed) package is getting lost and the handle_connection_down is not being called.

Are there a way to work-around it?

Thanks.

[robsonpeixoto]

The confluentinc/librdkafka#3109 (comment) are using an interesting solution for this, that is add connections.max.idle.ms to close the connection before the server/lb do that.

If the brod restart the connection after connections.max.idle.ms could help a lot with this kind scenario.

[zmstone]

Hi @robsonpeixoto
Do I understand it correctly that by closing the connection from client side before the server does so, we can get rid of an error level log ?
Or what else do we gain from it?

[robsonpeixoto]

Closing the connection every connections.max.idle.ms will only avoid a error message. But restarting the connection will increase the probability to keep low latency, because the connection could be ready to be used.

[zmstone]

@robsonpeixoto Thank you.
Latency is a good point, maybe one (or two?) new config deserved ?

[robsonpeixoto]

Maybe add a config restart_idle_connections_ms restarting idle connections could be very useful.

[dwalintukan]

Hello! I am also experiencing this same issue.

Closing the connection every connections.max.idle.ms will only avoid a error message. But restarting the connection will increase the probability to keep low latency, because the connection could be ready to be used.
Did this solve the issue?

Maybe add a config restart_idle_connections_ms restarting idle connections could be very useful.
Was this added and ready to use?