Mint HTTPAdapter failure cycle spams errors

[elliottneilclark]

Using k8s 2.4.0 we've been seeing a lot of errors around mint and HTTPAdapter.

To reproduce on a cluster that has things happening:

• Start some watchs on resources.
• Then, start tailing a pod's log.
• Wait for a little bit.
• Then, you will see some errors start to appear after a few minutes.

I've created a stand-alone repro here:
https://github.com/elliottneilclark/pod_logs

try with:

mix run -e "PodLogs.run(\"default\", \"mypodmakinglogs\")"

That should produce the following errors:

09:58:41.856 [debug] K8s.Client.Mint.HTTPAdapter Terminating HTTPAdapter GenServer #PID<0.224.0>

09:58:41.859 [warning] K8s.Client.Mint.HTTPAdapter Connection closed for reading and writing - stopping this process.

09:58:41.859 [debug] K8s.Client.Mint.HTTPAdapter Terminating HTTPAdapter GenServer #PID<0.229.0>

09:58:41.856 [error] GenServer #PID<0.219.0> terminating
** (stop) :closed
Last message: :healthcheck
State: %K8s.Client.Mint.HTTPAdapter{conn: %Mint.HTTP1{host: "127.0.0.1", port: 40451, request: nil, streaming_request: nil, socket: {:sslsocket, {:gen_tcp, #Port<0.9>, :tls_connection, :undefined}, [#PID<0.223.0>, #PID<0.222.0>]}, transport: Mint.Core.Transport.SSL, mode: :active, scheme_as_string: "https", requests: {[], []}, state: :closed, buffer: "", proxy_headers: [], private: %{}, log: false}, requests: %{}}

09:58:41.856 [error] GenServer #PID<0.224.0> terminating
** (stop) :closed
Last message: :healthcheck
State: %K8s.Client.Mint.HTTPAdapter{conn: %Mint.HTTP1{host: "127.0.0.1", port: 40451, request: nil, streaming_request: nil, socket: {:sslsocket, {:gen_tcp, #Port<0.10>, :tls_connection, :undefined}, [#PID<0.228.0>, #PID<0.227.0>]}, transport: Mint.Core.Transport.SSL, mode: :active, scheme_as_string: "https", requests: {[], []}, state: :closed, buffer: "", proxy_headers: [], private: %{}, log: false}, requests: %{}}

09:58:41.859 [error] GenServer #PID<0.229.0> terminating
** (stop) :closed
Last message: :healthcheck
State: %K8s.Client.Mint.HTTPAdapter{conn: %Mint.HTTP1{host: "127.0.0.1", port: 40451, request: nil, streaming_request: nil, socket: {:sslsocket, {:gen_tcp, #Port<0.11>, :tls_connection, :undefined}, [#PID<0.233.0>, #PID<0.232.0>]}, transport: Mint.Core.Transport.SSL, mode: :active, scheme_as_string: "https", requests: {[], []}, state: :closed, buffer: "", proxy_headers: [], private: %{}, log: false}, requests: %{}}

09:58:41.862 [warning] K8s.Client.Mint.HTTPAdapter Connection closed for reading and writing - stopping this process.

09:58:41.862 [debug] K8s.Client.Mint.HTTPAdapter Terminating HTTPAdapter GenServer #PID<0.234.0>

[mruoss]

Hi @elliottneilclark

Thanks for the repo. That makes my life a lot easier.

For websocket we have to use http/1 so we're using poolboy to maintain a connection pool. Looks like the unused connections are closed and the process needs to be terminated (garbage collected) after a while. I guess I need to use :normal, :shutdown or {:shutdown, term()} when terminating the GenServer, in order to not get these exceptions raised by the Supervisor. Currently I'm using :closed.