request help: configure backend service with external url.

[elvis-cai]

Issue description

Hi, trying to configure apisix ingress controller with multiple external url traffic splitting or failover, not sure if current approach is correct, if there's other workaround, please correct me, thanks.
Here's two features I am looking at:

• if bar.com is down, apisix should point to foo.com as failover
• if bar.com and foo.com are both active, traffic will be splitting to these two urls.

kind: ApisixRoute
metadata:
  name: httpserver-route
spec:
  http:
  - name: rule1
    match:
      hosts:
      - local.httpbin.org
      paths:
      - /*
    backends:
      - serviceName: apple-service
        servicePort: 80
        weight: 100
        resolveGranularity: service
      - serviceName: banana-service
        servicePort: 80
        weight: 50
        resolveGranularity: service
---
kind: Service
apiVersion: v1
metadata:
  name: banana-service
spec:
  type: ExternalName
  externalName: foo.com
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
---
kind: Service
apiVersion: v1
metadata:
  name: apple-service
spec:
  type: ExternalName
  externalName: bar.com
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80

After applying this file, I get Warning ResourceSyncAborted 1s (x5 over 3s) ApisixIngress ApisixIngress synced failed, with error: conflict headless service and backend resolve granularity, seems backend service could not be type: ExternalName

Environment

• your apisix-ingress-controller version (output of apisix-ingress-controller version --long);
  deploy with this doc https://apisix.apache.org/docs/ingress-controller/deployments/gke/
  version: apache/apisix-ingress-controller:1.0.0

• your Kubernetes cluster version (output of kubectl version);
  Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.10-gke.1600", GitCommit:"7b8e568a7fb4c9d199c2ba29a5f7d76f6b4341c2", GitTreeState:"clean", BuildDate:"2021-05-07T09:18:53Z", GoVersion:"go1.15.10b5", Compiler:"gc", Platform:"linux/amd64"}

• if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a).
  N/A

[gxthrj]

cc @tokers @tao12345666333 if you have time. Do we have to do special processing on ExternalName service?

[gxthrj]

We still need to discuss the security of using ExternalName Service.

[tokers]

@elvis-cai It seems that we don't handle the external service correctly, as it's also "headless".

apisix-ingress-controller/pkg/kube/translation/util.go

Line 42 in 1c17b41

if backend.ResolveGranularity == "service" && svc.Spec.ClusterIP == "" {

Fix is simple, just adding another condition that the service type is not ExternalName.

[tokers]

@elvis-cai It seems that we don't handle the external service correctly, as it's also "headless".

apisix-ingress-controller/pkg/kube/translation/util.go

Line 42 in 1c17b41

if backend.ResolveGranularity == "service" && svc.Spec.ClusterIP == "" {

Fix is simple, just adding another condition that the service type is not ExternalName.

Also, PR's welcome! :)

[elvis-cai]

cool, thanks @tokers for pointing out the root cause of the issue 👍 , by looking at kong ingress controller, also could find a condition check for externalname https://github.com/Kong/kubernetes-ingress-controller/blob/main/internal/parser/parser.go#L377.

[tokers]

cool, thanks @tokers for pointing out the root cause of the issue 👍 , by looking at kong ingress controller, also could find a condition check for externalname https://github.com/Kong/kubernetes-ingress-controller/blob/main/internal/parser/parser.go#L377.

Got it, would you like to submit a PR to optimize it? Thanks!

[elvis-cai]

kk, could have a look at this issue.

Another thing maybe not related to ingress controller, I did another test using apisix admin api, looks like rewrite host header support was added in May(apache/apisix#4208), so I could access each node defined in upstream.nodes with its header separately.
When testing the active health check, seems it is reading upstream.checks.active.host, if I miss this field, the active health check won't work as expected, so the idea is by default apisix gateway will load balancing each node, in this case is apple and banana, if one of the node is unheathy, apisix will go to the healthy one only.

Here's the colde I test:

{
	"uri": "/*",
	"host": "httpbin.org",
        "upstream": {
	"pass_host": "node",
        "nodes": {
		"my.apple.com:80": 1,
		"my.banana.com:80": 1
        },
        "type": "roundrobin",
        "retries": 2,
        "checks": {
            "active": {
                "timeout": 1,
		"http_path": "/",
                "healthy": {
                    "interval": 2,
                    "successes": 1
                },
                "unhealthy": {
                    "interval": 1,
                    "http_failures": 2
                },
                "req_headers": ["User-Agent: curl/7.29.0"]
            },
            "passive": {
                "healthy": {
                    "http_statuses": [200, 201],
                    "successes": 3
                },
                "unhealthy": {
                    "http_statuses": [500,503],
                    "http_failures": 1,
                    "tcp_failures": 3
                }
            }
        }
    }
}'

when hitting apisix gateway with header httpbin.org, it can go to apple and banana randomly, when scale down apple instance, could get 503 issue, here's the testing code:

for i in $(seq 1 1000); do curl  -H "Host: httpbin.org" ${APISIX_GATEWAY_URL} ; done 
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.17.7</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.17.7</center>
</body>
</html>
banana
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.17.7</center>
</body>
</html>
banana
banana
banana

here's the output when both apple and banana are runing:

for i in $(seq 1 1000); do curl  -H "Host: httpbin.org" ${APISIX_GATEWAY_URL}  ; done 
apple
banana
apple
apple
apple
banana
banana
apple
apple
apple
banana
banana
apple
banana

[tzssangglass]

relate to: apache/apisix#4906, w can discuss it there.

[suninuni]

cool, thanks @tokers for pointing out the root cause of the issue +1 , by looking at kong ingress controller, also could find a condition check for externalname https://github.com/Kong/kubernetes-ingress-controller/blob/main/internal/parser/parser.go#L377.

Got it, would you like to submit a PR to optimize it? Thanks!

Any updates?

Also for other solutions, have you considered supporting the configuration of apisix service in ApisixRoute?

[elvis-cai]

Since apache/apisix#4906 issue is fixed, trying to test add ExternalName support, and add passHost option in type ApisixUpstreamConfig struct, it works as expected with one k8s svc(type: ExternalName). 👍

Here comes to another question when have two nodes in the same apisix upstream, in k8s scenario, it is appending all endpoints to the list of node as code https://github.com/apache/apisix-ingress-controller/blob/master/pkg/kube/translation/translator.go#L238.

However in my case, I would like to have two external urls falling in the same node list, more detailed example could be found in this gist https://gist.github.com/elvis-cai/368ea541d3b9b7c66a8757e5017286b9#file-gistfile1-txt-L21, comments are the yaml file to apply, the output will be two apisix upstreams created, not sure what is the best way to have banana and apple url in the same node list.

[tokers]

Since apache/apisix#4906 issue is fixed, trying to test add ExternalName support, and add passHost option in type ApisixUpstreamConfig struct, it works as expected with one k8s svc(type: ExternalName). 👍

Here comes to another question when have two nodes in the same apisix upstream, in k8s scenario, it is appending all endpoints to the list of node as code https://github.com/apache/apisix-ingress-controller/blob/master/pkg/kube/translation/translator.go#L238.

However in my case, I would like to have two external urls falling in the same node list, more detailed example could be found in this gist https://gist.github.com/elvis-cai/368ea541d3b9b7c66a8757e5017286b9#file-gistfile1-txt-L21, comments are the yaml file to apply, the output will be two apisix upstreams created, not sure what is the best way to have banana and apple url in the same node list.

It would be tough to do this as both the backend are outside of the Kubernetes cluster, unless we can write the backend address directly in the ApisixRoute resource.

[tao12345666333]

I've recently spent a lot of time understanding your actual needs and writing a detailed proposal. I'll post on the mailing list and GitHub later, and we'll discuss how to implement it.

IIRC, this feature is not included in all current Ingress controller implementations. This will be a huge feature.

[github-actions]

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

[tao12345666333]

For specific progress and detailed technical solutions, we can use #927 to track. I will close this one. Thanks

I plan to be in two weeks.