Skip to content

Commit

Permalink
Openresty update: Fix issues with ssl_cert context
Browse files Browse the repository at this point in the history
The 1.19.3 Openresty version fixed how context is shared between the
ssl_certificate phase and others. Right now, if a dns_client is started
on ssl_cert, at other phases, the UDP socket is closed.

This is a fix at our end, but the fix will happen on Openresty upstream.

On staging env and Lazy config, on ssl_phase it'll retrieve the config
from an HTTP endpoint, and this will create a new dns_client instance
that cannot be used at all. It didn't hit the production env because a
timer retrieves the config.

Openresty config
```
master_process off;
worker_processes  1;
daemon off;
error_log /dev/stdout debug;

events {
  accept_mutex off;
  worker_connections 5000;
}

http {

    server {
        access_log  off;
        listen 8043 ssl http2;
        ssl_certificate certs/test.com.crt;
        ssl_certificate_key certs/test.com.key;

        ssl_certificate_by_lua_block {
          local resolver = require "resty.dns.resolver"
          local r, err = resolver:new{
              nameservers = {"8.8.8.8", {"8.8.4.4", 53} },
              retrans = 5,  -- 5 retransmissions on receive timeout
              timeout = 2000,  -- 2 sec
              no_random = true, -- always start with first nameserver
          }
          ngx.ctx.r = r

          local answers, err, tries = ngx.ctx.r:query("www.google.com", nil, {})
          ngx.log(ngx.ERR, "ANSWERS-->", require("inspect").inspect(answers))
        }

        location / {
          access_by_lua_block {
            ngx.log(ngx.ERR, "ACCESS");
            local answers, err, tries = ngx.ctx.r:query("www.redhat.com", nil, {})
            ngx.log(ngx.ERR, "ANSWER-->", require("inspect").inspect(answers))
            ngx.log(ngx.ERR, "ERR-->", require("inspect").inspect(err))
            ngx.log(ngx.ERR, "TRIES-->", require("inspect").inspect(tries))

          }

          proxy_pass "http://httpbin.org/headers";
        }
    }
}

```

Openresty log:

```
2021/06/22 10:49:44 [error] 258305#258305: *2 [lua] ssl_certificate_by_lua:12: ANSWERS-->{ {
    address = "142.250.185.4",
    class = 1,
    name = "www.google.com",
    section = 1,
    ttl = 25,
    type = 1
  } }, context: ssl_certificate_by_lua*, client: 172.19.0.1, server: 0.0.0.0:8043
2021/06/22 10:49:44 [info] 258305#258305: *1 SSL_do_handshake() failed (SSL: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:SSL alert number 48) while SSL handshaking, client: 172.19.0.1, server: 0.0.0.0:8043
2021/06/22 10:49:48 [error] 258305#258305: *6 [lua] ssl_certificate_by_lua:12: ANSWERS-->{ {
    address = "216.58.209.68",
    class = 1,
    name = "www.google.com",
    section = 1,
    ttl = 220,
    type = 1
  } }, context: ssl_certificate_by_lua*, client: 172.19.0.1, server: 0.0.0.0:8043
2021/06/22 10:49:48 [error] 258305#258305: *5 [lua] access_by_lua(eloy.conf:42):2: ACCESS, client: 172.19.0.1, server: , request: "GET / HTTP/2.0", host: "172.19.0.3:8043"
2021/06/22 10:49:48 [error] 258305#258305: *5 attempt to send data on a closed socket: u:00007FF1BB7C8FD0, c:0000000000000000, client: 172.19.0.1, server: , request: "GET / HTTP/2.0", host: "172.19.0.3:8043"
2021/06/22 10:49:48 [error] 258305#258305: *5 attempt to send data on a closed socket: u:00007FF1BB7C9188, c:0000000000000000, client: 172.19.0.1, server: , request: "GET / HTTP/2.0", host: "172.19.0.3:8043"
2021/06/22 10:49:48 [error] 258305#258305: *5 attempt to send data on a closed socket: u:00007FF1BB7C8FD0, c:0000000000000000, client: 172.19.0.1, server: , request: "GET / HTTP/2.0", host: "172.19.0.3:8043"
2021/06/22 10:49:48 [error] 258305#258305: *5 attempt to send data on a closed socket: u:00007FF1BB7C9188, c:0000000000000000, client: 172.19.0.1, server: , request: "GET / HTTP/2.0", host: "172.19.0.3:8043"
2021/06/22 10:49:48 [error] 258305#258305: *5 attempt to send data on a closed socket: u:00007FF1BB7C8FD0, c:0000000000000000, client: 172.19.0.1, server: , request: "GET / HTTP/2.0", host: "172.19.0.3:8043"
2021/06/22 10:49:48 [error] 258305#258305: *5 [lua] access_by_lua(eloy.conf:42):4: ANSWER-->nil, client: 172.19.0.1, server: , request: "GET / HTTP/2.0", host: "172.19.0.3:8043"
2021/06/22 10:49:48 [error] 258305#258305: *5 [lua] access_by_lua(eloy.conf:42):5: ERR-->"failed to send request to UDP server 8.8.8.8:53: closed", client: 172.19.0.1, server: , request: "GET / HTTP/2.0", host: "172.19.0.3:8043"
2021/06/22 10:49:48 [error] 258305#258305: *5 [lua] access_by_lua(eloy.conf:42):6: TRIES-->{ "failed to send request to UDP server 8.8.8.8:53: closed", "failed to send request to UDP server 8.8.4.4:53: closed", "failed to send request to UDP server 8.8.8.8:53: closed", "failed to send request to UDP server 8.8.4.4:53: closed", "failed to send request to UDP server 8.8.8.8:53: closed" }, client: 172.19.0.1, server: , request: "GET / HTTP/2.0", host: "172.19.0.3:8043"
```

Fix THREESCALE-7230
Signed-off-by: Eloy Coto <eloy.coto@acalustra.com>
  • Loading branch information
eloycoto committed Jun 22, 2021
1 parent c4769a4 commit 0ec3501
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 0 deletions.
7 changes: 7 additions & 0 deletions gateway/src/resty/resolver.lua
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,13 @@ function _M:instance()
if not resolver then
local dns = dns_client:instance(self.nameservers())
resolver = self.new(dns)
end

-- This condition is a bit hacky, but when using UDP cosockets on ssl_cert
-- phase, it'll be closed for other phases, so skip to share on the ssl_cert
-- case.
-- Check THREESCALE-7230 for more info.
if ngx.get_phase() ~= "ssl_cert" then
ctx.resolver = resolver
end

Expand Down
8 changes: 8 additions & 0 deletions gateway/src/resty/resolver/dns_client.lua
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,14 @@ function _M:instance(nameservers)

if not resolver then
resolver = self:new({ nameservers = nameservers })
end

-- This condition is a bit hacky, but when using UDP cosockets on ssl_cert
-- phase, it'll be closed for other phases, so skip to share on the ssl_cert
-- case.
-- Check THREESCALE-7230 for more info.
if ngx.get_phase() ~= "ssl_cert" then
ngx.log(ngx.ERR, "NGX---phase", ngx.get_phase())
ctx.dns = resolver
end

Expand Down

0 comments on commit 0ec3501

Please sign in to comment.