Orphaned open requests

[gaynetdinov]

Hi! 👋

I'm using Mint in a proxy gateway phoenix app. The phoenix app accepts a request, sends another HTTP request using Mint and then proxying response back to the client using Plug.Conn.chunk/2 function every time Mint returns {:data, request_ref, data}.

For handling Mint connections I'm (more or less) using the same architecture you've described in the documentation https://hexdocs.pm/mint/architecture.html#wrapping-a-mint-connection-in-a-genserver

The GenServer is started, mint_conn is obtained via Mint.HTTP.connect(:http, host, port). Imagine a sitation when one request have been sent using mint_conn. This request is still open (i.e. I haven't received :done yet for this request, Mint.HTTP.open_request_count(mint_conn) would return 1). Sometimes when I send another request, mint_conn would return :closed error. When I encounter this scenario I can take 2 different approaches:

1. I create a new mint_conn to the same host and port and send the second request through the new mint_conn, because the previous one was closed. If I do so, the first request which was sent using the first mint_conn would never come back, i.e. I'd never receive it in handle_info in my GenServer. So this way I'd loose this request because I'd never receive the response.

2. I create a new mint_conn to the same host and port, take the first request from the GenServer state and re-send it again using the new just created Mint connection. This way I'd not loose this request, but I'd send it twice. Another interesting thing is that I'd eventually receive the first orphaned open requests from the first mint_conn in handle_message and Mint.HTTP.stream(state.mint_conn, message) would return {:error, mint_conn, %Mint.HTTPError{reason: :unexpected_data, data}}.

Could you please help me understand how to properly deal with this sitation? I don't want to loose requests and I also don't want to re-send it (it would not work for non-GET requests anyways). Maybe there is a way to instruct Mint to still receive/wait for the open requests even if I create a new connection to the same host/port?

Thank you! 💜

[whatyouhide]

Hey @gaynetdinov, I'm struggling to understand the problem here. I don't know what you mean when you say things like "create a new mint_conn to the same host and port. Do you mean in the same GenServer that was holding the previous connection? For your situation, do you have a pool of these GenServers?

[ericmj]

You get the :closed error response because the connection was closed, since the connection was closed you may not get any more TCP messages from the socket. If the message will hit handle_info or not is not affected by you opening a new connection. Is it possible that it hits handle_info but Mint.HTTP.stream/2 returns :unknown because you threw away the original connection when you started the new connection? A message is intended for a specific conn struct and only that conn will match the message, it doesn't matter if the new conn was opened against the same host/port pair.

Another interesting thing is that I'd eventually receive the first orphaned open requests from the first mint_conn in handle_message and Mint.HTTP.stream(state.mint_conn, message) would return {:error, mint_conn, %Mint.HTTPError{reason: :unexpected_data, data}}.

This should not happen since the old message has a different socket reference than the new connection, see https://github.com/ericmj/mint/blob/master/lib/mint/http1.ex#L379. What version of mint are you using?

Could you please help me understand how to properly deal with this sitation? I don't want to loose requests and I also don't want to re-send it (it would not work for non-GET requests anyways).

While the specific scenario you are reporting here shouldn't happen you will always need a way to handle this. Requests can fail at any time and you need to decide if you should retry or let the error bubble up.

Maybe there is a way to instruct Mint to still receive/wait for the open requests even if I create a new connection to the same host/port?

This is not handled by Mint. Mint will wait for more messages when you call Mint.HTTP.stream/2. If you open a new connection the other connections won't even know about them since the connection is simply a data structure with no outside communication. If you want to receive Mint to receive messages for an old connection you will have to keep that conn struct around even when you start the new conn. You then have to call Mint.HTTP.stream(old_conn, message), mint will return :unknown if the message is not intended for the that connection, so you can match on :unknown and then pass the message to the new connection instead.

[gaynetdinov]

@whatyouhide

I don't know what you mean when you say things like "create a new mint_conn to the same host and port. Do you mean in the same GenServer that was holding the previous connection?

Yes, when I get {:error, mint_conn, %{reason: :closed}} from Mint.HTTP.request/5 I cannot send any requests through this connection anymore, because it's closed. So I create a new connection, put it into the state of the GenServer and then continue sending requests through this new connection.

For your situation, do you have a pool of these GenServers?

Everything is happening inside one GenServer.

@ericmj

Is it possible that it hits handle_info but Mint.HTTP.stream/2 returns :unknown because you threw away the original connection when you started the new connection?

I don't actually get :unknown from Mint.HTTP.stream/2. My problem is that when the connection is closed and there are still some open requests, the GenServer does not receive responses. I mean I don't receive them in handle_info(message, state). So the problem is that in the situation when a request is sent and then the connection got closed, I'd never get the response of the sent request, i.e. they would not go to handle_info(message, state), so I cannot really do Mint.HTTP.stream(old_conn, message), because I don't have a message.

Take a look at the example in the docs. I've noticed this behaviour because my GenServer takes a lot of memory over time. So I checked its state and the state was full of request_ref. These a refs of the requests which never got response, they never appeared in handle_info(message, state), therefore they never got into process_response({:done, request_ref}, state), so they got accumulated in the state of the GenServer.

What version of mint are you using?

I've tried 0.2.1 and 0.4.0.

[ericmj]

So the problem is that in the situation when a request is sent and then the connection got closed, I'd never get the response of the sent request, i.e. they would not go to handle_info(message, state), so I cannot really do Mint.HTTP.stream(old_conn, message), because I don't have a message.

This is expected. You do not get more data on closed connections, you can only expect messages on open connections.

I've noticed this behaviour because my GenServer takes a lot of memory over time. So I checked its state and the state was full of request_ref.

Regardless of this issue you need a way to time out requests and if you hit a timeout you need to delete all references and restart the connection to clear up the memory.

[whatyouhide]

Also a very important thing: when a connection is closed, all the requests that were open on that connection can be considered closed.

[gaynetdinov]

@ericmj

Sorry I'm confused now.
Do I understand correctly that once connection is closed it does not make sense to wait a response in handle_info which would belong to this closed connection? Which would mean it does not make any sense to keep old closed connection around in a hope that some time I'd get an event in handle_info which would belong to the old closed connection, right?

@whatyouhide

what does it mean that requests are closed? does it mean that it's safe to retry all these requests? Or I should always consider such requests as 'lost' and reply to the client something like 503, retry_later?

[whatyouhide]

Do I understand correctly that once connection is closed it does not make sense to wait a response in handle_info which would belong to this closed connection?

If the connection is closed, you could still get one last message because it might be sitting in your mailbox already when you notice that the connection is closed. I would just throw away the connection and throw away all the messages that return :unknown when passed to Mint.HTTP.stream(new_connection, msg).

what does it mean that requests are closed? does it mean that it's safe to retry all these requests?

This is not a Mint problem in any way. It's a computer science/HTTP problem 😃 Some requests are probably safe to retry, like GETs, some aren't. Ultimately, you cannot know if a request reached the server correctly if the TCP connection is interrupted, since the connection could get interrupted after the request reached and the server replied, so you just didn't get the reply but the request was processed.

[gaynetdinov]

Thank you so much for the detailed answers. You helped me to clarify some open questions I've had. Much appreciated. 💜