Active health checks cannot set HTTPS target "Healthy" automatically again after "down" simulation #7592

igorgomeslima · 2021-07-15T04:48:01Z

igorgomeslima
Jul 15, 2021

Summary

[Docker] Kong version 2.5.0-centos
When I activate the health check mechanism Kong set targets as “Healthy”, perfectly. But when I try to simulate a “broken target”(shutdown a localhost api, for example) the Kong “checkers” set this target “Unhealthy” and never set this “Healthy” again.

Steps To Reproduce

SERVICES

{
  "host": "my_simple_api_upstream",
  "id": "f0f273cd-df43-48d2-8374-66848cd16f2c",
  "protocol": "https",
  "read_timeout": 60000,
  "tls_verify_depth": null,
  "port": 8000,
  "updated_at": 1626319456,
  "ca_certificates": null,
  "created_at": 1626318853,
  "connect_timeout": 60000,
  "write_timeout": 60000,
  "name": "my_simple_api",
  "retries": 5,
  "path": "/",
  "tls_verify": null,
  "tags": [],
  "client_certificate": null,
  "extras": {
    "createdUser": null,
    "updatedUser": null,
    "id": 3,
    "service_id": "f0f273cd-df43-48d2-8374-66848cd16f2c",
    "kong_node_id": "1",
    "description": null,
    "tags": null,
    "createdAt": "2021-07-15T03:14:13.000Z",
    "updatedAt": "2021-07-15T03:24:16.000Z"
  }
}

ROUTES

{
  "strip_path": true,
  "tags": null,
  "updated_at": 1626319001,
  "destinations": null,
  "headers": null,
  "protocols": [
    "http",
    "https"
  ],
  "methods": null,
  "service": {
    "host": "my_simple_api_upstream",
    "id": "f0f273cd-df43-48d2-8374-66848cd16f2c",
    "protocol": "https",
    "read_timeout": 60000,
    "tls_verify_depth": null,
    "port": 8000,
    "updated_at": 1626319456,
    "ca_certificates": null,
    "created_at": 1626318853,
    "connect_timeout": 60000,
    "write_timeout": 60000,
    "name": "my_simple_api",
    "retries": 5,
    "path": "/",
    "tls_verify": null,
    "tags": [],
    "client_certificate": null,
    "extras": {
      "createdUser": null,
      "updatedUser": null,
      "id": 3,
      "service_id": "f0f273cd-df43-48d2-8374-66848cd16f2c",
      "kong_node_id": "1",
      "description": null,
      "tags": null,
      "createdAt": "2021-07-15T03:14:13.000Z",
      "updatedAt": "2021-07-15T03:24:16.000Z"
    }
  },
  "snis": null,
  "hosts": null,
  "name": "my_simple_api_route",
  "path_handling": "v1",
  "paths": [
    "/mysimpleapi"
  ],
  "preserve_host": false,
  "regex_priority": 0,
  "response_buffering": true,
  "sources": null,
  "id": "1de27a8e-3aee-4513-866a-f3104db41522",
  "https_redirect_status_code": 426,
  "request_buffering": true,
  "created_at": 1626319001
}

UPSTREAMS

{
  "client_certificate": null,
  "created_at": 1626319334,
  "id": "f1af9630-5fda-42ea-ad9a-10f9497e5c95",
  "tags": null,
  "name": "my_simple_api_upstream",
  "algorithm": "round-robin",
  "hash_on_header": null,
  "hash_fallback_header": null,
  "host_header": null,
  "hash_on_cookie": null,
  "healthchecks": {
    "threshold": 0,
    "active": {
      "unhealthy": {
        "http_statuses": [
          429,
          404,
          500,
          501,
          502,
          503,
          504,
          505
        ],
        "tcp_failures": 0,
        "timeouts": 2,
        "http_failures": 2,
        "interval": 7
      },
      "type": "http",
      "http_path": "/",
      "timeout": 1,
      "healthy": {
        "successes": 5,
        "interval": 5,
        "http_statuses": [
          200,
          302
        ]
      },
      "https_sni": null,
      "https_verify_certificate": false,
      "concurrency": 10
    },
    "passive": {
      "unhealthy": {
        "http_failures": 0,
        "http_statuses": [
          429,
          500,
          503
        ],
        "tcp_failures": 0,
        "timeouts": 0
      },
      "healthy": {
        "http_statuses": [
          200,
          201,
          202,
          203,
          204,
          205,
          206,
          207,
          208,
          226,
          300,
          301,
          302,
          303,
          304,
          305,
          306,
          307,
          308
        ],
        "successes": 0
      },
      "type": "http"
    }
  },
  "hash_on_cookie_path": "/",
  "hash_on": "none",
  "hash_fallback": "none",
  "slots": 1000
}

Additional Details & Logs

I'am using the "Active health checks" configuration with two targets running locally on HOST(outside network used by Kong container):

In the Upstream configuration I put the host.docker.internal solved by host file(Windows) so the Kong container can "touch" this targets/services, like:

With this initial configuration on Upstream everything works fine, the Kong algorithm can balance beetween this two targets. But, when I try to enable the "Active health checks" I had a unexpected behavior when simulate a "shutdown" of one these targets.

When I activate the health check mechanism Kong set both target as "Healthy", great. But when I try to simulate a "broken target"(shutdown a localhost api, for example) the Kong "checkers" set this target "Unhealthy" and never set this "Healthy" again.

Kong container logs sample


2021/07/15 04:15:24 [warn] 28#0: *62794 [lua] ring.lua:246: redistributeIndices(): [upstream:a_upstream 1] redistributed indices, size=1000, dropped=0, assigned=0, left unassigned=1000, context: ngx.timer, client: 172.24.0.1, server: 0.0.0.0:8000

2021/07/15 04:15:27 [error] 28#0: *226258 [lua] healthcheck.lua:1096: log(): [healthcheck] (116726d8-af6f-46bb-b779-ce3ed3110c41:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (host.docker.internal:5001)': closed, context: ngx.timer, client: 172.24.0.1, server: 0.0.0.0:8000

2021/07/15 04:15:27 [error] 28#0: *226258 [lua] healthcheck.lua:1096: log(): [healthcheck] (116726d8-af6f-46bb-b779-ce3ed3110c41:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (host.docker.internal:6001)': closed, context: ngx.timer, client: 172.24.0.1, server: 0.0.0.0:8000

2021/07/15 04:15:27 [error] 28#0: *226261 [lua] healthcheck.lua:1096: log(): [healthcheck] (116726d8-af6f-46bb-b779-ce3ed3110c41:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (host.docker.internal:5001)': closed, context: ngx.timer, client: 172.24.0.1, server: 0.0.0.0:8000

2021/07/15 04:15:27 [error] 28#0: *226261 [lua] healthcheck.lua:1096: log(): [healthcheck] (116726d8-af6f-46bb-b779-ce3ed3110c41:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (host.docker.internal:6001)': closed, context: ngx.timer, client: 172.24.0.1, server: 0.0.0.0:8000

If we set the "Healthy" of this targets using the Kong api directly everything works fine, and the balancing can work again:

**Konga Request:**
http://localhost:1337/kong/upstreams/f1af9630-5fda-42ea-ad9a-10f9497e5c95/targets/6efe54ec-084b-4a71-8803-b3e6b4380c4e/healthy

Logs:

2021/07/15 04:22:42 [warn] 23#0: *237542 [lua] healthcheck.lua:1096: log(): [healthcheck] (116726d8-af6f-46bb-b779-ce3ed3110c41:my_simple_api_upstream) healthy forced for host.docker.internal host.docker.internal:6001, client: 172.24.0.2, server: kong_admin, request: "POST /upstreams/f1af9630-5fda-42ea-ad9a-10f9497e5c95/targets/6efe54ec-084b-4a71-8803-b3e6b4380c4e/healthy HTTP/1.1", host: "kong:8001",
172.24.0.2 - - [15/Jul/2021:04:22:42 +0000] "POST /upstreams/f1af9630-5fda-42ea-ad9a-10f9497e5c95/targets/6efe54ec-084b-4a71-8803-b3e6b4380c4e/healthy HTTP/1.1" 204 0 "-" "-",
2021/07/15 04:22:43 [warn] 25#0: *237547 [lua] balancer.lua:313: [healthchecks] failed setting peer status (upstream: 116726d8-af6f-46bb-b779-ce3ed3110c41:my_simple_api_upstream): no peer found by name 'host.docker.internal' and address host.docker.internal:6001, context: ngx.timer,
2021/07/15 04:22:46 [warn] 23#0: *237619 [lua] healthcheck.lua:1096: log(): [healthcheck] (116726d8-af6f-46bb-b779-ce3ed3110c41:my_simple_api_upstream) healthy forced for host.docker.internal host.docker.internal:5001, client: 172.24.0.2, server: kong_admin, request: "POST /upstreams/f1af9630-5fda-42ea-ad9a-10f9497e5c95/targets/18dafd82-fc30-4dde-a28c-9ad5560a0d55/healthy HTTP/1.1", host: "kong:8001",
172.24.0.2 - - [15/Jul/2021:04:22:46 +0000] "POST /upstreams/f1af9630-5fda-42ea-ad9a-10f9497e5c95/targets/18dafd82-fc30-4dde-a28c-9ad5560a0d55/healthy HTTP/1.1" 204 0 "-" "-"

Infra/Config

OS = Windows 10 Pro 21H1, build: 19043.1083
Kong = 2.5.0-centos(tested in 2.3.2-alpine too)[Container]
PostgreSQL = 9.6 [Container]
Docker Engine = v20.10.7

Docker network configuration(used by Kong container)

Why I miss? Any network configuration between HOST and Docker container used by Kong? 😆

kikito · 2021-07-15T09:40:49Z

kikito
Jul 15, 2021
Maintainer

Thank you for reporting this, we'll investigate.

0 replies

locao · 2021-07-15T14:21:38Z

locao
Jul 15, 2021
Maintainer

Hi @igorgomeslima! Are you sure those logs are from Kong 2.5.0? There are a few things that should be different on them.

Anyway, it seems that the behavior is correct because Kong is not getting an answer from the upstream services. I see in the logs you shared that the upstream services are responding to POSTs requests, but do they answer to GETs? Are you able to log in to the Kong container and send a request to those services using curl? For instance:

$ curl -X GET -svk -o/dev/null https://host.docker.internal:5001

0 replies

igorgomeslima · 2021-07-15T16:21:06Z

igorgomeslima
Jul 15, 2021
Author

Hey @locao! You are right, those logs are from Kong 2.3.2-alpine, I tried two versions of Kong(mentioned in the end of issue description) and I put the wrong log version here, my bad.

I recorded a video showing my configuration, if interest, please see ✌.

CURL GET result from Kong container

[root@d916dafbf6f9 ~]# curl -X GET -svk -o/dev/null https://host.docker.internal:5001
About to connect() to host.docker.internal port 5001 (#0)
Trying 192.168.65.2...
Connected to host.docker.internal (192.168.65.2) port 5001 (#0)
Initializing NSS with certpath: sql:/etc/pki/nssdb
skipping SSL peer certificate verification
SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
Server certificate:
subject: CN=localhost
start date: Jun 07 14:27:25 2021 GMT
expire date: Jun 07 14:27:25 2022 GMT
common name: localhost
issuer: CN=localhost
GET / HTTP/1.1
User-Agent: curl/7.29.0
Host: host.docker.internal:5001
Accept: /

HTTP/1.1 404 Not Found
Date: Thu, 15 Jul 2021 16:20:01 GMT
Server: Kestrel
Content-Length: 0

Connection #0 to host host.docker.internal left intact

I've been update Docker network configuration used by Kong container in the issue description.

Some logs from version 2.5.0-centos

Executing task: docker logs --tail 1000 -f d916dafbf6f913c5b4203e613fa2e1b3777a910647036ff1e9e1182b3dfaacb9 <

2021/07/15 15:54:34 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
2021/07/15 15:54:35 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:472: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: connection refused
stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/kong/init.lua:472: in function 'init'
init_by_lua:3: in main chunk
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:472: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: connection refused
stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/kong/init.lua:472: in function 'init'
init_by_lua:3: in main chunk
2021/07/15 15:54:37 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
2021/07/15 15:54:37 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:472: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: connection refused
stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/kong/init.lua:472: in function 'init'
init_by_lua:3: in main chunk
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:472: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: connection refused
stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/kong/init.lua:472: in function 'init'
init_by_lua:3: in main chunk
2021/07/15 15:54:38 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
2021/07/15 15:54:39 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:16: Database needs
bootstrapping or is older than Kong 1.0.

To start a new installation from scratch, run 'kong migrations bootstrap'.

To migrate from a version older than 1.0, migrated to Kong 1.5.0 first.
If you still have 'apis' entities, you can convert them to Routes and Services
using the 'kong migrations migrate-apis' command in Kong 1.5.0.

stack traceback:
[C]: in function 'error'
/usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:16: in function 'check_state'
/usr/local/share/lua/5.1/kong/init.lua:475: in function 'init'
init_by_lua:3: in main chunk
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:16: Database needs bootstrapping or is older than Kong 1.0.

To start a new installation from scratch, run 'kong migrations bootstrap'.

To migrate from a version older than 1.0, migrated to Kong 1.5.0 first.
If you still have 'apis' entities, you can convert them to Routes and Services
using the 'kong migrations migrate-apis' command in Kong 1.5.0.

stack traceback:
[C]: in function 'error'
/usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:16: in function 'check_state'
/usr/local/share/lua/5.1/kong/init.lua:475: in function 'init'
init_by_lua:3: in main chunk
2021/07/15 15:54:40 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
2021/07/15 15:54:40 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:20: New migrations
available; run 'kong migrations up' to proceed
stack traceback:
[C]: in function 'error'
/usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:20: in function 'check_state'
/usr/local/share/lua/5.1/kong/init.lua:475: in function 'init'
init_by_lua:3: in main chunk
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:20: New migrations available; run 'kong migrations up' to proceed
stack traceback:
[C]: in function 'error'
/usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:20: in function 'check_state'
/usr/local/share/lua/5.1/kong/init.lua:475: in function 'init'
init_by_lua:3: in main chunk
2021/07/15 15:54:41 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
2021/07/15 15:54:42 [notice] 1#0: using the "epoll" event method
2021/07/15 15:54:42 [notice] 1#0: openresty/1.19.3.2
2021/07/15 15:54:42 [notice] 1#0: built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
2021/07/15 15:54:42 [notice] 1#0: OS: Linux 5.4.72-microsoft-standard-WSL2
2021/07/15 15:54:42 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2021/07/15 15:54:42 [notice] 1#0: start worker processes
2021/07/15 15:54:42 [notice] 1#0: start worker process 29
2021/07/15 15:54:42 [notice] 1#0: start worker process 30
2021/07/15 15:54:42 [notice] 1#0: start worker process 31
2021/07/15 15:54:42 [notice] 1#0: start worker process 32
2021/07/15 15:54:42 [notice] 1#0: start worker process 33
2021/07/15 15:54:42 [notice] 1#0: start worker process 34
2021/07/15 15:54:42 [notice] 1#0: start worker process 35
2021/07/15 15:54:42 [notice] 1#0: start worker process 36
2021/07/15 15:54:42 [notice] 29#0: 1 [lua] warmup.lua:92: single_dao(): Preloading 'services' into the core_cache..., context: init_worker_by_lua
2021/07/15 15:54:42 [notice] 31#0: 2 [kong] init.lua:290 only worker #0 can manage, context: init_worker_by_lua
2021/07/15 15:54:42 [notice] 33#0: 5 [kong] init.lua:290 only worker #0 can manage, context: init_worker_by_lua
2021/07/15 15:54:42 [notice] 32#0: 4 [kong] init.lua:290 only worker #0 can manage, context: init_worker_by_lua
2021/07/15 15:54:42 [notice] 29#0: 1 [lua] warmup.lua:129: single_dao(): finished preloading 'services' into the core_cache (in 0ms), context: init_worker_by_lua
2021/07/15 15:54:42 [notice] 30#0: 3 [kong] init.lua:290 only worker #0 can manage, context: init_worker_by_lua
2021/07/15 15:54:42 [notice] 35#0: 7 [kong] init.lua:290 only worker #0 can manage, context: init_worker_by_lua
2021/07/15 15:54:42 [notice] 34#0: 6 [kong] init.lua:290 only worker #0 can manage, context: init_worker_by_lua
2021/07/15 15:54:42 [notice] 36#0: 8 [kong] init.lua:290 only worker #0 can manage, context: init_worker_by_lua
2021/07/15 15:54:47 [crit] 33#0: *12 [lua] balancer.lua:826: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2021/07/15 15:54:47 [crit] 30#0: *13 [lua] balancer.lua:826: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2021/07/15 15:54:47 [crit] 32#0: *14 [lua] balancer.lua:826: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2021/07/15 15:54:47 [crit] 29#0: *15 [lua] balancer.lua:826: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2021/07/15 15:54:47 [crit] 35#0: *16 [lua] balancer.lua:826: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2021/07/15 15:54:47 [crit] 36#0: *18 [lua] balancer.lua:826: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2021/07/15 15:54:47 [crit] 34#0: *17 [lua] balancer.lua:826: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
172.18.0.1 - - [15/Jul/2021:15:55:07 +0000] "GET / HTTP/1.1" 404 48 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:55:30 +0000] "GET / HTTP/1.1" 200 10017 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.3 - - [15/Jul/2021:15:56:46 +0000] "GET / HTTP/1.1" 200 10017 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:56:46 +0000] "GET /?connection_id=1 HTTP/1.1" 200 10017 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:56:46 +0000] "GET / HTTP/1.1" 200 10017 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:56:46 +0000] "GET / HTTP/1.1" 200 10017 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:56:46 +0000] "GET /status?size=1000 HTTP/1.1" 200 1410 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:56:48 +0000] "GET /services?size=1000 HTTP/1.1" 200 23 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:56:49 +0000] "GET /services?size=1000 HTTP/1.1" 200 23 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:56:49 +0000] "GET /routes?size=1000 HTTP/1.1" 200 23 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:56:50 +0000] "GET /services?size=1000 HTTP/1.1" 200 23 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:58:02 +0000] "POST /services/ HTTP/1.1" 201 358 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:58:02 +0000] "GET /services?size=1000 HTTP/1.1" 200 381 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:58:04 +0000] "GET /services/32dd1d46-b292-4d62-904c-0d010802b6c2 HTTP/1.1" 200 358 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:58:23 +0000] "PATCH /services/32dd1d46-b292-4d62-904c-0d010802b6c2 HTTP/1.1" 200 383 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:58:26 +0000] "GET /services/32dd1d46-b292-4d62-904c-0d010802b6c2/routes?size=1000 HTTP/1.1" 200 23 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:58:47 +0000] "POST /routes/ HTTP/1.1" 201 492 "-" "-"
172.18.0.3 - - [15/Jul/2021:15:58:47 +0000] "GET /services/32dd1d46-b292-4d62-904c-0d010802b6c2/routes?size=1000 HTTP/1.1" 200 515 "-" "-"
2021/07/15 15:59:07 [error] 36#0: *5371 [lua] balancer.lua:1038: execute(): DNS resolution failed: dns server error: 3 name error. Tried: ["(short)host.internal.docker:(na) - cache-miss","host.internal.docker:33 - cache-miss/scheduled/querying/dns server error: 3 name error","host.internal.docker:1 - cache-miss/scheduled/querying/dns server error: 3 name error","host.internal.docker:5 - cache-miss/scheduled/querying/dns server error: 3 name error"], client: 172.18.0.1, server: kong, request: "GET /mysimpleapi/weatherforecast HTTP/1.1", host: "localhost:8000"
172.18.0.1 - - [15/Jul/2021:15:59:07 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 503 36 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.3 - - [15/Jul/2021:15:59:20 +0000] "PATCH /services/32dd1d46-b292-4d62-904c-0d010802b6c2 HTTP/1.1" 200 383 "-" "-"
172.18.0.1 - - [15/Jul/2021:15:59:21 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:22 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:23 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:23 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:23 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:23 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:23 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:23 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:24 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:24 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:24 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:24 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.3 - - [15/Jul/2021:15:59:38 +0000] "PATCH /services/32dd1d46-b292-4d62-904c-0d010802b6c2 HTTP/1.1" 200 368 "-" "-"
172.18.0.1 - - [15/Jul/2021:15:59:42 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 553 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:44 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 556 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:45 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 555 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:47 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 569 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:48 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 557 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:15:59:49 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 562 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.3 - - [15/Jul/2021:15:59:55 +0000] "GET /upstreams?size=1000 HTTP/1.1" 200 23 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:00:11 +0000] "POST /upstreams HTTP/1.1" 201 913 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:00:11 +0000] "GET /upstreams?size=1000 HTTP/1.1" 200 936 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:00:12 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9 HTTP/1.1" 200 913 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:00:12 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/targets?size=1000 HTTP/1.1" 200 23 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:00:13 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/health?size=1000 HTTP/1.1" 200 72 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:00:28 +0000] "POST /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/targets HTTP/1.1" 201 192 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:00:28 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/targets?size=1000 HTTP/1.1" 200 215 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:00:28 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/health?size=1000 HTTP/1.1" 200 514 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:00:36 +0000] "GET /services?size=1000 HTTP/1.1" 200 391 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:00:37 +0000] "GET /services/32dd1d46-b292-4d62-904c-0d010802b6c2 HTTP/1.1" 200 368 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:00:44 +0000] "PATCH /services/32dd1d46-b292-4d62-904c-0d010802b6c2 HTTP/1.1" 200 370 "-" "-"
172.18.0.1 - - [15/Jul/2021:16:00:46 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 569 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:46 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 561 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:46 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 551 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:47 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 558 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:47 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 563 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:47 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 564 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:47 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 553 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:47 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 564 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:47 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 558 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:47 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:48 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 556 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:48 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 564 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:48 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 556 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:48 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 570 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:48 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 562 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:48 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 554 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:48 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 555 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:00:48 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 563 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.3 - - [15/Jul/2021:16:00:57 +0000] "GET /upstreams?size=1000 HTTP/1.1" 200 936 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:01:43 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9 HTTP/1.1" 200 913 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:01:43 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/targets?size=1000 HTTP/1.1" 200 215 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:01:43 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/health?size=1000 HTTP/1.1" 200 514 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:01:45 +0000] "GET /upstreams?size=1000 HTTP/1.1" 200 936 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:01:59 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9 HTTP/1.1" 200 913 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:01:59 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/targets?size=1000 HTTP/1.1" 200 215 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:01:59 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/health?size=1000 HTTP/1.1" 200 514 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:02:09 +0000] "POST /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/targets HTTP/1.1" 201 192 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:02:09 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/targets?size=1000 HTTP/1.1" 200 408 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:02:09 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/health?size=1000 HTTP/1.1" 200 957 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:02:14 +0000] "GET / HTTP/1.1" 200 10017 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:02:14 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9 HTTP/1.1" 200 913 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:02:14 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/targets?size=1000 HTTP/1.1" 200 408 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:02:14 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/health?size=1000 HTTP/1.1" 200 957 "-" "-"
172.18.0.1 - - [15/Jul/2021:16:02:31 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 554 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:02:32 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 504 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:02:33 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 554 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:02:36 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 508 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:02:38 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 560 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:02:42 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 530 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:02:46 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 564 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:02:47 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 522 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:02:47 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 560 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.3 - - [15/Jul/2021:16:14:04 +0000] "PATCH /upstreams/11792981-2256-4b4e-b772-4512d0a119a9 HTTP/1.1" 200 914 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:14:07 +0000] "GET / HTTP/1.1" 200 10017 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:14:07 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9 HTTP/1.1" 200 914 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:14:07 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/targets?size=1000 HTTP/1.1" 200 408 "-" "-"
172.18.0.3 - - [15/Jul/2021:16:14:08 +0000] "GET /upstreams/11792981-2256-4b4e-b772-4512d0a119a9/health?size=1000 HTTP/1.1" 200 921 "-" "-"
2021/07/15 16:14:09 [error] 36#0: *24563 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:09 [error] 36#0: *24563 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:14 [error] 36#0: *24670 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:14 [error] 36#0: *24670 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
172.18.0.1 - - [15/Jul/2021:16:14:15 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 567 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:14:16 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 518 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.18.0.1 - - [15/Jul/2021:16:14:16 +0000] "GET /mysimpleapi/weatherforecast HTTP/1.1" 200 554 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
2021/07/15 16:14:19 [error] 36#0: *24780 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:19 [error] 36#0: *24780 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:24 [error] 36#0: *24887 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:24 [error] 36#0: *24887 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:29 [error] 36#0: *24994 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:29 [error] 36#0: *24994 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:34 [error] 36#0: *25103 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:34 [error] 36#0: *25103 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:39 [error] 36#0: *25210 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:39 [error] 36#0: *25210 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:44 [error] 36#0: *25326 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:44 [error] 36#0: *25326 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:49 [error] 36#0: *25443 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:49 [error] 36#0: *25443 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:54 [error] 36#0: *25550 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:54 [error] 36#0: *25550 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:59 [error] 36#0: *25657 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:14:59 [error] 36#0: *25657 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:04 [error] 36#0: *25768 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:04 [error] 36#0: *25768 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:09 [error] 36#0: *25875 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:09 [error] 36#0: *25875 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:14 [error] 36#0: *25983 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:14 [error] 36#0: *25983 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:19 [error] 36#0: *26090 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:19 [error] 36#0: *26090 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:24 [error] 36#0: *26198 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:24 [error] 36#0: *26198 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:30 [error] 36#0: *26305 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:30 [error] 36#0: *26305 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:35 [error] 36#0: *26413 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:35 [error] 36#0: *26413 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:40 [error] 36#0: *26520 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:40 [error] 36#0: *26520 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:45 [error] 36#0: *26628 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:45 [error] 36#0: *26628 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:50 [error] 36#0: *26735 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:50 [error] 36#0: *26735 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:55 [error] 36#0: *26842 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:15:55 [error] 36#0: *26842 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:00 [error] 36#0: *26950 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:00 [error] 36#0: *26950 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:05 [error] 36#0: *27057 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:05 [error] 36#0: *27057 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:10 [error] 36#0: *27164 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:10 [error] 36#0: *27164 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:15 [error] 36#0: *27272 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:15 [error] 36#0: *27272 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:20 [error] 36#0: *27382 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:20 [error] 36#0: *27382 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:25 [error] 36#0: *27489 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:25 [error] 36#0: *27489 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:30 [error] 36#0: *27597 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:30 [error] 36#0: *27597 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:35 [error] 36#0: *27704 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:35 [error] 36#0: *27704 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:40 [error] 36#0: *27815 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:40 [error] 36#0: *27815 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:45 [error] 36#0: *27930 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:45 [error] 36#0: *27930 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:50 [error] 36#0: *28043 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:50 [error] 36#0: *28043 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:55 [error] 36#0: *28150 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:16:55 [error] 36#0: *28150 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:00 [error] 36#0: *28257 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:00 [error] 36#0: *28257 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:05 [error] 36#0: *28367 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:05 [error] 36#0: *28367 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:10 [error] 36#0: *28474 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:10 [error] 36#0: *28474 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:15 [error] 36#0: *28588 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:15 [error] 36#0: *28588 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:20 [error] 36#0: *28695 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:20 [error] 36#0: *28695 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:26 [error] 36#0: *28803 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:26 [error] 36#0: *28803 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:31 [error] 36#0: *28910 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:31 [error] 36#0: *28910 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:36 [error] 36#0: *29019 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:36 [error] 36#0: *29019 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:41 [error] 36#0: *29129 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:41 [error] 36#0: *29129 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:46 [error] 36#0: *29237 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:46 [error] 36#0: *29237 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:51 [error] 36#0: *29344 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:51 [error] 36#0: *29344 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:56 [error] 36#0: *29451 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:17:56 [error] 36#0: *29451 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:18:01 [error] 36#0: *29558 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:18:01 [error] 36#0: *29558 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:18:06 [error] 36#0: *29666 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:18:06 [error] 36#0: *29666 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:18:11 [error] 36#0: *29773 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:18:11 [error] 36#0: *29773 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:18:16 [error] 36#0: *29883 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:18:16 [error] 36#0: *29883 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:18:21 [error] 36#0: *29990 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:18:21 [error] 36#0: *29990 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:6001)': closed, context: ngx.timer, client: 172.18.0.3, server: 0.0.0.0:8001
2021/07/15 16:18:26 [error] 36#0: *30098 [lua] healthcheck.lua:1126: log(): [healthcheck] (972542c5-b9f2-42a8-b3eb-ee4278e12cba:my_simple_api_upstream)

0 replies

igorgomeslima · 2021-07-15T19:45:27Z

igorgomeslima
Jul 15, 2021
Author

@locao

UPDATE

When I run curl with the "full path"(host.docker.internal:5001/weatherforecast) of service running on HOST, I got a HTTP/2 200.

The path /weatherforecast route is an GET method, running with .NET 6 sample project.

# curl -X GET -svk -o/dev/null https://host.docker.internal:5001/weatherforecast
*   Trying 192.168.65.2:5001...
* TCP_NODELAY set
* Connected to host.docker.internal (192.168.65.2) port 5001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [94 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [794 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [365 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [102 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=localhost
*  start date: Jun  7 14:27:25 2021 GMT
*  expire date: Jun  7 14:27:25 2022 GMT
*  issuer: CN=localhost
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x5642c0b68660)
} [5 bytes data]
> GET /weatherforecast HTTP/2
> Host: host.docker.internal:5001
> user-agent: curl/7.67.0
> accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200
< content-type: application/json; charset=utf-8
< date: Thu, 15 Jul 2021 19:26:44 GMT
< server: Kestrel
<
{ [544 bytes data]
* Connection #0 to host host.docker.internal left intact

In this situation, how to configure the targets on Upstream? Keeping in mind that if we don't type a path, we won't get a return 200 OK? I need to configure it in my targets Apis to response a 200 code when GET https://host.docker.internal:5001?

0 replies

locao · 2021-07-15T20:44:31Z

locao
Jul 15, 2021
Maintainer

Hey @igorgomeslima, thanks for the logs! In your case, the health checker is telling you that it "failed to receive status line" because you have the wrong protocol configured, and it can't connect to the service you are trying to check. But if you fix that, you would still end with an unhealthy upstream as the 404 status is also considered a failure.

To fix that you must configure the "http_path" to some path you want to test and use the correct protocol in "type". In your case I'd suggest:

"healthchecks": {
    ...
    "active": {
      "unhealthy": {
       ...
      },
      "type": "https",
      "http_path": "/weatherforecast",
       ...
      "healthy": {
        ...
      },
    },
    "passive": {
    ...
    }
  },

(I redacted everything that is not relevant to this issue)

We don't support Konga, but it seems by your video that those options are available in the Upstream Details tab.

I'm closing this issue as it is not a bug. Please feel free to reopen it if the problem persists or create a new Discussion if you would like some help from the community to configure your environment.

0 replies

igorgomeslima · 2021-07-16T02:39:38Z

igorgomeslima
Jul 16, 2021
Author

Hey @locao thank you so much for your help on this. Yours comments give me a lot of insights.

But, some things sounds strange for me when I try to configure an Https target.

Please see, I set the configuration mentioned before:

"healthchecks": {
...
"active": {
"unhealthy": {
...
},
"type": "https",
"http_path": "/health",
...
"healthy": {
...
},
},
"passive": {
...
}
},

Konga

And I've been configured a "health check" endpoint in my targets:

HTTPS target
https://host.docker.internal:5001/health (inside container)

HTTP target
http://host.docker.internal:7000/health (inside container)

With this configuration, after simulate "down" services, the "Kong health checkers" can only set status of http targets.

See:

"Down" both targets:

Both targets http & https UNHEALTHY!

2021/07/16 01:48:15 [error] 32#0: *158408 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:48:20 [error] 32#0: *158518 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:48:25 [error] 32#0: *158628 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:48:31 [warn] 32#0: *158734 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) unhealthy TIMEOUT increment (1/1) for 'host.docker.internal(192.168.65.2:7000)', context: ngx.timer,
2021/07/16 01:48:31 [warn] 32#0: *158754 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:31 [warn] 30#0: *158757 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:32 [warn] 35#0: *158760 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:32 [warn] 36#0: *158763 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:32 [warn] 34#0: *158768 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:32 [warn] 37#0: *158771 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:32 [warn] 31#0: *158774 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:32 [warn] 33#0: *158777 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer

"Up" both targets:

Only http target HEALTHY again!

2021/07/15 23:09:43 [error] 32#0: *15173 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/15 23:09:53 [error] 32#0: *15393 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:48:25 [error] 32#0: *158628 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:48:31 [warn] 32#0: *158754 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:31 [warn] 32#0: *158734 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) unhealthy TIMEOUT increment (1/1) for 'host.docker.internal(192.168.65.2:7000)', context: ngx.timer,
2021/07/16 01:48:15 [error] 32#0: *158408 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:48:20 [error] 32#0: *158518 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:48:31 [warn] 30#0: *158757 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:32 [warn] 35#0: *158760 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:32 [warn] 36#0: *158763 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:32 [warn] 34#0: *158768 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:32 [warn] 37#0: *158771 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:32 [warn] 31#0: *158774 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:48:32 [warn] 33#0: *158777 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to UNHEALTHY, context: ngx.timer,
2021/07/16 01:52:58 [error] 32#0: *164479 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:53:03 [warn] 32#0: *164589 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) healthy SUCCESS increment (2/5) for 'host.docker.internal(192.168.65.2:7000)', context: ngx.timer,
2021/07/16 01:52:58 [warn] 32#0: *164479 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) healthy SUCCESS increment (1/5) for 'host.docker.internal(192.168.65.2:7000)', context: ngx.timer,
2021/07/16 01:53:23 [error] 32#0: *165035 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:53:19 [warn] 36#0: *164959 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:53:19 [warn] 34#0: *164962 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:53:19 [warn] 30#0: *164941 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:53:18 [warn] 31#0: *164931 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:53:18 [warn] 33#0: *164928 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:53:18 [warn] 37#0: *164925 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:53:18 [warn] 35#0: *164922 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:53:18 [error] 32#0: *164916 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:53:18 [warn] 32#0: *164919 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:53:13 [warn] 32#0: *164808 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) healthy SUCCESS increment (4/5) for 'host.docker.internal(192.168.65.2:7000)', context: ngx.timer,
2021/07/16 01:53:13 [error] 32#0: *164808 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:53:18 [warn] 32#0: *164916 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) healthy SUCCESS increment (5/5) for 'host.docker.internal(192.168.65.2:7000)', context: ngx.timer,
2021/07/16 01:53:08 [warn] 32#0: *164697 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) healthy SUCCESS increment (3/5) for 'host.docker.internal(192.168.65.2:7000)', context: ngx.timer,
2021/07/16 01:53:08 [error] 32#0: *164697 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:53:03 [error] 32#0: *164589 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:53:38 [error] 32#0: *165360 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:53:33 [error] 32#0: *165251 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:53:28 [error] 32#0: *165143 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:56:25 [error] 32#0: *169002 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:56:25 [warn] 32#0: *169002 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) healthy SUCCESS increment (1/5) for 'host.docker.internal(192.168.65.2:7000)', context: ngx.timer,
2021/07/16 01:56:30 [warn] 32#0: *169114 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) healthy SUCCESS increment (2/5) for 'host.docker.internal(192.168.65.2:7000)', context: ngx.timer,
2021/07/16 01:56:30 [error] 32#0: *169114 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:56:35 [warn] 32#0: *169222 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) healthy SUCCESS increment (3/5) for 'host.docker.internal(192.168.65.2:7000)', context: ngx.timer,
2021/07/16 01:56:35 [error] 32#0: *169222 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:56:40 [warn] 32#0: *169332 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) healthy SUCCESS increment (4/5) for 'host.docker.internal(192.168.65.2:7000)', context: ngx.timer,
2021/07/16 01:56:40 [error] 32#0: *169332 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:56:45 [warn] 32#0: *169446 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) healthy SUCCESS increment (5/5) for 'host.docker.internal(192.168.65.2:7000)', context: ngx.timer,
2021/07/16 01:56:45 [warn] 32#0: *169449 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:56:45 [error] 32#0: *169446 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:56:45 [warn] 34#0: *169452 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:56:45 [warn] 35#0: *169458 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:56:45 [warn] 36#0: *169455 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:56:46 [warn] 37#0: *169461 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:56:46 [warn] 33#0: *169464 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:56:46 [warn] 31#0: *169467 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:56:46 [warn] 30#0: *169470 [lua] balancer.lua:258: callback(): [healthchecks] balancer f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream reported health status changed to HEALTHY, context: ngx.timer,
2021/07/16 01:56:50 [error] 32#0: *169563 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:56:55 [error] 32#0: *169671 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:57:00 [error] 32#0: *169779 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,
2021/07/16 01:57:05 [error] 32#0: *169890 [lua] healthcheck.lua:1126: log(): [healthcheck] (f5c39920-a0e7-4b01-b765-a29dd90a742b:my_simple_api_upstream) failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer

Some curl tests inside Kong container

With HTTP:// on URI

Without HTTP:// on URI

With HTTPS:// on URI

Without HTTPS:// on URI

Well, on this last curl test we can see an Empty reply from server and I think Kong checkers call with this pattern(according logs), and I think Kong receive "Empty reply from server" and this "status" is not able to set "HEALTHY" for this specific target.

Log sample -> failed to receive status line from 'host.docker.internal (192.168.65.2:5001)': closed, context: ngx.timer,

192.168.65.2 = host.docker.internal

Only to check:

With HTTPS:// protocol = HTTP/1.1 200 OK

With HTTPS:// protocol = Empty reply from server

😱

Make sense? I need something like "force Kong checkers to use HTTPS protocol on calls"? I miss something?

Please @locao, do you think this is more adequate on discussions? I cannot re-open this issue with my privileges.

Thanks! 🐱‍👤

0 replies

locao · 2021-07-16T14:50:55Z

locao
Jul 16, 2021
Maintainer

Hi @igorgomeslima! Is this weather service open source? Can you share it so I can run some tests here?

If this is an issue with Kong I can reopen this ticket, if it's a configuration problem I will move it to the Discussions.

Thank you for your help!

0 replies

igorgomeslima · 2021-07-16T18:49:31Z

igorgomeslima
Jul 16, 2021
Author

Hey @locao, the weather service is the default Web Api sample project used by .NET scaffolding CLI dotnet new webapi or Visual Studio project/solution wizard.

If you kindly can, I've made a repo with my sample applications: Weather Forecast Apis - HTTP & HTTPS.

~~1. I'll later try to put it in a docker-compose file to simplify this setup(but it's a little bit more complicated when involving HTTPS certs on containers 😂, anyway...).~~

I've put some info in README.md file, but, if it not help, feel free to tell me here. 👀

Thank you for your support!

-

UPDATE @locao

Now the project has a docker-compose.yml, if relevant, I've put more info on README.md.

0 replies

igorgomeslima · 2021-07-21T15:46:45Z

igorgomeslima
Jul 21, 2021
Author

Hi @locao, any news?

0 replies

locao · 2021-07-22T20:07:23Z

locao
Jul 22, 2021
Maintainer

Hi @igorgomeslima! Thanks for the detailed test environment. I was able to reproduce your issue and the solution here.

The service provided indeed blocks any HTTP request:

$ curl -sv http://localhost:5001/health
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 5001 (#0)
> GET /health HTTP/1.1
> Host: localhost:5001
> User-Agent: curl/7.64.1
> Accept: */*
>
* Empty reply from server
* Connection #0 to host localhost left intact
* Closing connection 0

Using HTTPS without a certificate also does not work:

$ curl -sv https://localhost:5001/health
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 5001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0

Using curl internal certificate is good enough to get a response:

$ curl -svk https://localhost:5001/health
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 5001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=localhost
*  start date: Jun  7 14:27:25 2021 GMT
*  expire date: Jun  7 14:27:25 2022 GMT
*  issuer: CN=localhost
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd7a2009600)
> GET /health HTTP/2
> Host: localhost:5001
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< cache-control: no-store, no-cache
< date: Thu, 22 Jul 2021 19:50:49 GMT
< pragma: no-cache
< content-type: text/plain
< expires: Thu, 01 Jan 1970 00:00:00 GMT
< server: Kestrel
<
* Connection #0 to host localhost left intact
Healthy* Closing connection 0

With these restraints, a valid upstream configuration would be:

upstreams:
- name: weather-https-upstream
  algorithm: round-robin
  healthchecks:
    threshold: 0
    active:
      unhealthy:
        http_statuses:
        - 429
        - 404
        - 500
        - 501
        - 502
        - 503
        - 504
        - 505
        tcp_failures: 3
        timeouts: 3
        http_failures: 3
        interval: 1
      type: https
      http_path: /health
      timeout: 1
      healthy:
        successes: 1
        interval: 1
        http_statuses:
        - 200
        - 302
      https_verify_certificate: false
      concurrency: 10
  hash_on_cookie_path: /
  hash_on: none
  hash_fallback: none
  slots: 10000
targets:
- weight: 100
  target: localhost:5001
  upstream: weather-https-upstream

When the service is up, I get these log entries in Kong:

2021/07/22 16:46:32 [debug] 15492#0: *473 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) Checking localhost 127.0.0.1:5001 (currently healthy)
2021/07/22 16:46:32 [debug] 15492#0: *473 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) Reporting 'localhost (127.0.0.1:5001)' (got HTTP 200)

If I stop the service, I can see these entries:

2021/07/22 16:46:43 [error] 15492#0: *528 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) failed SSL handshake with 'localhost (127.0.0.1:5001)': handshake failed, context: ngx.timer
2021/07/22 16:46:43 [warn] 15492#0: *528 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) unhealthy TCP increment (1/3) for 'localhost(127.0.0.1:5001)', context: ngx.timer

... and then:

2021/07/22 16:46:45 [debug] 15492#0: *538 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) Checking localhost 127.0.0.1:5001 (currently mostly_healthy)
2021/07/22 16:46:45 [warn] 15492#0: *538 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) unhealthy TCP increment (3/3) for 'localhost(127.0.0.1:5001)', context: ngx.timer
2021/07/22 16:46:45 [debug] 15492#0: *538 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream], event=unhealthy, pid=15492, data=table: 0x32c3ea30
2021/07/22 16:46:45 [debug] 15492#0: *538 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) event: target status 'localhost(127.0.0.1:5001)' from 'true' to 'false'
2021/07/22 16:46:45 [warn] 15492#0: *540 [lua] healthcheckers.lua:92: callback(): [healthchecks] balancer 0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream reported health status changed to UNHEALTHY, context: ngx.timer

Now an unhealthy target is being checked:

2021/07/22 16:46:46 [debug] 15492#0: *544 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) checking unhealthy targets: #1
2021/07/22 16:46:46 [debug] 15492#0: *544 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) Checking localhost 127.0.0.1:5001 (currently unhealthy)

When the service is back:

2021/07/22 16:46:56 [debug] 15492#0: *594 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) Checking localhost 127.0.0.1:5001 (currently unhealthy)
2021/07/22 16:46:56 [debug] 15492#0: *594 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) Reporting 'localhost (127.0.0.1:5001)' (got HTTP 200)
2021/07/22 16:46:56 [warn] 15492#0: *594 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) healthy SUCCESS increment (1/1) for 'localhost(127.0.0.1:5001)', context: ngx.timer
2021/07/22 16:46:56 [debug] 15492#0: *594 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream], event=healthy, pid=15492, data=table: 0x337c10e0
2021/07/22 16:46:56 [debug] 15492#0: *594 [lua] healthcheck.lua:1126: log(): [healthcheck] (0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream) event: target status 'localhost(127.0.0.1:5001)' from 'false' to 'true'
2021/07/22 16:46:56 [warn] 15492#0: *596 [lua] healthcheckers.lua:92: callback(): [healthchecks] balancer 0dc6f45b-8f8d-40d2-a504-473544ee190b:weather-https-upstream reported health status changed to HEALTHY, context: ngx.timer

Kong is behaving as expected for this case, but the active health check only works with this service if the type is https. I don't use Konga, so I'm not sure how this option is set. Please, check your upstream configuration using:

$ curl http://localhost:8001/upstreams | jq

1 reply

igorgomeslima Jul 23, 2021
Author

Hi @locao, thank you for your support!

The certificate in this sample is a dev-localhost certificate, maybe for this we cannot validate the issuer.

I followed your settings on Upstream and get your result 😉.