bug: Ingress controller not creating upstream on ApisixUpstream resource creation

[luismiguelsaez-steercrm]

Current Behavior

I am installing APISIX the apisix-ingress-controller Helm chart version 0.14.0' as a dependency of the main [APISIX Helm chart](https://github.com/apache/apisix-helm-chart/blob/apisix-2.9.0/charts/apisix/Chart.yaml) 2.9.0`.

After installing it, I am able to create routes by adding ApisixRoute resources, but the ApisixUpstream resources are not created in the same way, seeing "not found" errors in the ingress controller pod.

As far as I know, both resources are created from the ingress controller, but only the routes are working.

Here are the values in the main APISIX Helm chart, that are passed to the ingress controller sub-chart:

ingress-controller:
  config:
    apisix:
      adminAPIVersion: v3
      existingSecret: apisix-admin-credentials
      existingSecretAdminKeyKey: admin
      serviceNamespace: apisix
    apisixResourceSyncInterval: 30s
    ingressPublishService: apisix/apisix-admin
    ingressStatusAddress: []
    kubernetes:
      ingressClass: apisix
      namespaceSelector:
        - kubernetes.io/metadata.name=identity-dev2
        - kubernetes.io/metadata.name=infra-1-dev2
      resyncInterval: 30s
    logLevel: debug
    logOutput: stderr
  enabled: true
  gateway:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /healthz
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: traffic-port
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: http
      service.beta.kubernetes.io/aws-load-balancer-name: dev2-apisix-ingress
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
      service.beta.kubernetes.io/aws-load-balancer-type: external
    externalTrafficPolicy: Local
    tls:
      enabled: true
    type: LoadBalancer
  replicaCount: 1
metrics:
  serviceMonitor:
    enabled: true
rbac:
  create: true
replicaCount: 2
service:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: **REDACTED**
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: traffic-port
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-name: dev2-apisix
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: **REDACTED**
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: 443
    service.beta.kubernetes.io/aws-load-balancer-type: external
  externalTrafficPolicy: Local
  type: LoadBalancer
serviceAccount:
  create: true

Expected Behavior

After adding an ApisixUpstream resource to the cluster in one of the watched namespaces, the ingress controller creates the corresponding APISIX upstream by sending the request to the APISIX admin API.

Error Logs

2024-10-02T20:54:04+08:00 debug apisix/apisix_upstream.go:515 ApisixUpstream add event arrived {"object": {"metadata":{"name":"rooster-test","namespace":"identity-dev2","uid":"6bcc67e6-7bb7-48a1-bb26-cd80713613ef","resourceVersion":"797561152","generation":1,"creationTimestamp":"2024-10-02T12:54:04Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{"apiVersion":"apisix.apache.org/v2","kind":"ApisixUpstream","metadata":{"annotations":{},"name":"rooster-test","namespace":"identity-dev2"},"spec":{"discovery":{"serviceName":"rooster-ui-dev2","type":"kubernetes"},"loadbalancer":{"type":"roundrobin"}}}\n"},"managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"apisix.apache.org/v2","time":"2024-10-02T12:54:04Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:discovery":{".":{},"f:serviceName":{},"f:type":{}},"f:loadbalancer":{".":{},"f:type":{}}}}}]},"spec":{"loadbalancer":{"type":"roundrobin"},"discovery":{"serviceName":"rooster-ui-dev2","type":"kubernetes"}},"status":{}}}
2024-10-02T20:54:04+08:00 debug apisix/upstream.go:43 try to look up upstream {"name": "identity-dev2_rooster-test", "url": "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams", "cluster": "default"}
2024-10-02T20:54:04+08:00 debug apisix/upstream.go:59 failed to find upstream in cache, will try to lookup from APISIX {"name": "identity-dev2_rooster-test", "error": "not found"}
2024-10-02T20:54:04+08:00 error apisix/apisix_upstream.go:333 failed to get upstream identity-dev2_rooster-test: not found
2024-10-02T20:54:04+08:00 warn apisix/apisix_upstream.go:489 sync ApisixUpstream failed, will retry {"object": {"Type":1,"Object":{"Key":"identity-dev2/rooster-test","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "not found"}

Steps to Reproduce

1. Install APISIX ingress controller from the main APISIX Helm chart: https://github.com/apache/apisix-helm-chart/blob/apisix-2.9.0/charts/apisix/Chart.yaml
2. Add an ApisixRoute that gets created without problems
3. Try to create a route by adding an ApisixUpstream resource to the cluster in one of the watched namespaces
4. The corresponding APISIX route is not created and the ingress's logs show "not found" errors

Environment

• APISIX Ingress controller version (run apisix-ingress-controller version --long): 1.8.0
• Kubernetes cluster version (run kubectl version): v.1.29.7
• OS version if running APISIX Ingress controller in a bare-metal environment (run uname -a): N/A

[Revolyssup]

FYI, the upstream and route resource both are created by ApisixRoute CR. ApisixUpstream CR configures the already exisiting upstream created via ApisixRoute.

[luismiguelsaez-steercrm]

Does that mean that I cannot just create an ApisixUpstream, configure it to use kubernetes discovery to match the required Service and specify that upstream in the ApisixRoute at .spec.http.upstreams?

If so, I misunderstood how it works but also is a great limitation in the flexibility to configure the upstreams.