Service with type ExternalName was not working properly due to unresolved address

[reynaldiwijaya]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG

NGINX Ingress controller version:

0.20.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.8", GitCommit:"7eab6a49736cc7b01869a15f9f05dc5b49efb9fc", GitTreeState:"clean", BuildDate:"2018-09-14T15:54:20Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• AWS
• Linux ip-172-20-54-40 4.9.0-7-amd64 Basic structure  #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x86_64 GNU/Linux

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

What happened:

Service type externalName is not resolved, resulting it to be wrapped as a ipv6 address instead of ipv4

What you expected to happen:

Service type external name will then forward the request to intended address

How to reproduce it (as minimally and precisely as possible):

Ingress definition

 apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
   annotations:
     kubernetes.io/ingress.class: nginx
     kubernetes.io/tls-acme: "true"
     nginx.ingress.kubernetes.io/ssl-redirect: "true"
     nginx.ingress.kubernetes.io/vts-filter-key: '"$uri $server_name"'
   creationTimestamp: 2018-10-26T09:14:34Z
   generation: 2
   name: khaki-service
   namespace: dev
   resourceVersion: "113417090"
   selfLink: /apis/extensions/v1beta1/namespaces/dev/ingresses/khaki-service
   uid: 8e420595-d8ff-11e8-b26f-06ebc499f550
 spec:
   backend:
     serviceName: test-khaki-service
     servicePort: 80
   tls:
   - hosts:
     - khaki.dev.xxx.co
     secretName: khaki-xxx-tls
 status:
   loadBalancer:
     ingress:
     - ip: 13.251.xx.xxx
     - ip: 54.179.xxx.xxx

Service definition

 apiVersion: v1
 kind: Service
 metadata:
   creationTimestamp: 2018-10-29T07:07:10Z
   name: test-khaki-service
   namespace: dev
   resourceVersion: "113416910"
   selfLink: /api/v1/namespaces/dev/services/test-khaki-service
   uid: 40cce488-db49-11e8-b26f-06ebc499f550
 spec:
   externalName: khaki.xxx.co.s3-website-ap-southeast-1.amazonaws.com
   ports:
   - name: http
     port: 80
     protocol: TCP
     targetPort: 80
   sessionAffinity: None
   type: ExternalName
 status:
   loadBalancer: {}

Anything else we need to know:

Got this error from the log
2018/10/29 10:02:36 [error] 5801#5801: *96438 [lua] balancer.lua:174: balance(): error while setting current upstream peer [khaki.xxx.co.s3-website-ap-southeast-1.amazonaws.com]:80: invalid IPv6 address while connecting to upstream, client: 218.255.91.182, server: _, request: "GET /favicon.ico HTTP/1.1", host: "khaki.xxx.co", referrer: "http://khaki.xxx.co/"

I debugged to the lua code, and I found on balancer.lua that when the balancer is doing sync_backend(), the variable backend.service.spec is empty (which was supposed to include type and will then trigger a function to resolve the dns on external Name endpoint). Printing backend.service will show that spec is referring to a table address.

2018/10/29 10:09:15 [warn] 5940#5940: *97949 [lua] balancer.lua:93: sync_backend(): spec,table: 0x401cfcb0, context: ngx.timer

But printing the content would result in nothing

This behavior may also be found on ClusterIP service that we have. It is strange since only some of these service don't have it.

[aledbf]

@ElvinEfendi ping ^^

[reynaldiwijaya]

Ah did not notice it happened since the path is empty. Thanks a lot @aledbf @ElvinEfendi