What happens when Kong forwards a request to backend but backend does not give any response?

[Sereneee]

Is there a default message that is returned back to frontend?
If so, where is it being thrown from?
Is there a way I can intercept it?
Is there something I can do in the kong plugins lifecycle (presumably in the :response() phase) to detect this connection timeout and return a customized response to frontend?

[kikito]

Hello Sereneee,

It depends on how you define "not giving any response":

• If the backend return non-http response (for example, a binary file without an HTTP header) then Kong will return a 502 status response with the message An invalid response was received from the upstream server
• If the backend does not even accept a connection, then Kong will return immediately with a 503 response The upstream server is currently unavailable
• If the backend accepts the connection but does not produce a response, then kong will eventually kill the connection and return a 504-The upstream server is timing out response

Is there a default message that is returned back to frontend?

The error descriptions returned on each case are defined in https://github.com/Kong/kong/blob/master/kong/error_handlers.lua

If so, where is it being thrown from?

They originate from the nginx proxy module http://nginx.org/en/docs/http/ngx_http_proxy_module.html

Is there a way I can intercept it?
Is there something I can do in the kong plugins lifecycle (presumably in the :response() phase) to detect this connection timeout and return a customized response to frontend?

Yes, you can use the response phase to customize or change completely the returned response. You will need a custom plugin in order to do this. Example:

-- kong/plugins/custom_plugin/handler.lua

function CustomPlugin:response(config)
  if kong.service.response.get_status() == 502 then
    return kong.response.exit(200, "error successfully masked")
  end
end 

[kikito]

Hello @Sereneee, the first option should be trying to make sure that the plugin you have installed is being actually triggered. If you modify it so that it logs:

function ExceptionHandlingHandler:response(conf)
  kong.log.notice("ExceptionHandlingHandler triggered")
  if kong.service.response.get_status() == 502 then
    return kong.response.exit(200, "error successfully masked")
  end
end

And you don't see ExceptionHandlingHandler triggered in your logs, that means that the plugin is not properly assigned to the corresponding Service or Route (or globally).

In addition to this, the message:

*51 connect() failed (111: Connection refused) while connecting to upstream

Worries me. It seems to indicate that the problem might not be that the upstream is returning 502, but that it is not returning anything at all to Kong (refusing the connection). So then it would be Kong itself who returns the 502 message. That one is harcoded. Please make sure that it is possible to connect to the upstream machine from the machine which hosts kong.

[Sereneee]

Hi @kikito , thank you for your response!
Yes, I have managed to make sure the plugin I have installed is actually being triggered.
You are absolutely right about the upstream not returning anything at all to Kong. That is actually what I am trying to get at. Is there a way that Kong can detect this Connection refused in the :response() phase?
Essentially, I would like to do something like this

if kong detects connection refused then
    return kong.response.exit(500, {
      ["code"] = "99999",
      ["title"] = "Connection Refused",
      ["description"] = "No response from upstream",
      ["message"] = "Please try again later"
  })
end

[javierguerragiraldez]

Hi @Sereneee , the :response() phase won't be executed if there's no response from the upstream. A better strategy is to do as @kikito explains, handling the "Connection refused" error. Otherwise, you could have some fallback upstream that only generates "not available" messages, or poll the healthchecker's status from a separate "status report" system.

[Sereneee]

hi @javierguerragiraldez , thanks for your reply!

Sorry I'm a bit lost here, may I ask what do you mean by

...to do as @kikito explains, handling the "Connection refused" error.

I've read through the answer but still don't quite get the picture. If you don't mind, can you explain it with a bit more details please?

[kikito]

Hi @Sereneee, sorry I didn't see this earlier.

If the upstream server does not respond, Kong will respond with a 502 status when it reaches a timeout. There is no way to change this behavior inside Kong itself.

However, that isn't necessarily the end of the road. But what lies ahead is a bit of an unbeaten path; it relies on the fact that Kong can use almost anything as a Service, and that includes Kong itself.

This means that you could, in theory, create two services:

• The first service S1 targets your (Sometimes not responding) upstream server. In your previous example the url of the service would be http://127.0.0.1:9093/v1/503/no/body
• The second service S2 uses S1 as its url. The host would probably be 127.0.0.1. You can make the mapping by using a dedicated port, a path, or a header. It depends a bit on how you made the mapping on the other server.

I think the most straightforward way to it is using different ports for s1 and s2, but it really depends on your infrastructure (perhaps your firewalls don't like adding new ports). You would need to add it to the proxy_listen section in Kong's configuration doc. S1 would not need to expose its port to the outside world, since everyone else would communicate through S2's port (see below).

S2 would need to have its connect_timeout attribute higher than the first service. Otherwise, when the first service times out, the second one will also time out. Once that is configured, you should be able to detect a 502 (which you would be getting from the first Service when there is a timeout) as mentioned above, with a custom plugin.

Then you use the second Service for general access.

• When upstream responds, a request would go like this: Consumer -> S2 -> S1 -> Upstream and the response would be the opposite.
• When upstream is down, a request would do S2 -> S1 -> (timeout). The Responses would be S1 (502) -> S2 (your plugin transforms the response here) -> Consumer.