Http2 Adapter causing 500 requests

[canuc]

The http2 adapter gets closed after some time, and causes the process to shutdown, if there are messsages in the inbox they get flushed and the phoenix endpoint returns 500 reproducable locally.

The workaround is to disable http2 and always return http1 for our usecase.

[warning] K8s.Client.Mint.HTTPAdapter Connection closed for reading and writing - stopping this process.
[debug] K8s.Client.Mint.HTTPAdapter Terminating HTTPAdapter GenServer #PID<0.2321.0>
[debug] K8s.Client.Mint.ConnectionRegistry DOWN of process #PID<0.2321.0> received.
[error] #PID<0.2350.0> running Phoenix.Endpoint.SyncCodeReloadPlug (connection #PID<0.2082.0>, stream id 9) terminated
Server: localhost:4000 (http)
Request: POST {
** (exit) exited in: GenServer.call(#PID<0.2321.0>, {:recv, #Reference<0.2495543023.3533963267.100249>}, :infinity)
    ** (EXIT) shutdown: :closed
    

[mruoss]

Hi @canuc

This sounds like a bug. But you're not giving me much context to reproduce this. What version of k8s is this? What is the code that leads to this?

Thanks

[razielgn]

I'm also having this issue, using version 2.4.1 (mint 1.5.1). It happens every 3/4 days, sometimes multiple times in a single day.
What I have is a cronjob via Oban that periodically does

K8s.Conn.from_service_account()
|> K8s.Client.run(K8s.Client.list("argoproj.io/v1alpha1", "workflow", namespace: "argo-workflows"))

Here's the stacktrace:

** (exit) exited in: GenServer.call(#PID<0.292039.0>, {:recv, #Reference<0.4064860026.2373451778.104924>}, :infinity)
    ** (EXIT) shutdown: :closed

lib/gen_server.ex in GenServer.call/3 at line 1074
lib/k8s/client/mint_http_provider.ex in anonymous fn/3 in K8s.Client.MintHTTPProvider.stream/5 at line 41
lib/stream.ex in Stream.do_resource/5 at line 1626
lib/stream.ex in Enumerable.Stream.do_each/4 at line 1828
lib/enum.ex in Enum.reduce/3 at line 4387
lib/k8s/client/mint_http_provider.ex in K8s.Client.MintHTTPProvider.request/5 at line 20
lib/app/workers/workflows_refresher.ex in App.Workers.WorkflowsRefresher.perform/1 at line 19

Let me know if I should provide more info, thanks.

[mruoss]

Sorry, I totally forgot about this issue. It reached me while on vacation. I have an idea of what could be the problem, but I need a quiet moment to wrap my head around it.

Note to self: Double check the meaning of the type option of Mint.open?() - docs don't seem obvious right now.

[mruoss]

Hey @canuc and @razielgn

I think #285 actually solves this. Would either of you be willing to test your code with the version of that PR?

[razielgn]

Hello, yes I'm going to deploy this patch today and let it run for a few days. I think by Monday it'll be clear. Thank you!

[mruoss]

@razielgn all quiet so far?

[razielgn]

All quiet so far!

[mruoss]

Nice. Thanks for testing it. I will merge and release.

[razielgn]

Thanks!

[canuc]

This is amazing! thanks so much!!!