Transport Server from Rabbitmq on Openshift, connection refused

[andrescolodrero]

Hi,
I recently deployed the ingress operator in Openshift and im trying to access an instance of Rabbitmq out the cluster,
For testing, HTTP ingress work correct, but im trying to setuip tcp access.

But i cant make transportServer to work with rabbitmq.
To reproduce:

1. Install Rabbitmq on rabbitmq-system namespace
2. Install a INgress controller with the operator on rabbitmq-system
3. Expose rabbitmq services "rabbitmq-sandbox2" on port 5672

kind: Service
apiVersion: v1
metadata:
  annotations:
    rabbitmq.com/topology-allowed-namespaces: rabbitmq-system
  resourceVersion: '74006948'
  name: rabbitmq-sandbox2
  namespace: rabbitmq-system
  ownerReferences:
    - apiVersion: rabbitmq.com/v1beta1
      kind: RabbitmqCluster
      name: rabbitmq-sandbox2
      uid: d74ea3d0-4f06-40a2-9157-4bc7fc9352a5
      controller: true
      blockOwnerDeletion: true
  labels:
    app.kubernetes.io/component: rabbitmq
    app.kubernetes.io/name: rabbitmq-sandbox2
    app.kubernetes.io/part-of: rabbitmq
spec:
  ports:
    - name: amqp
      protocol: TCP
      appProtocol: amqp
      port: 5672
      targetPort: 5672
    - name: management
      protocol: TCP
      appProtocol: http
      port: 15672
      targetPort: 15672
    - name: stream
      protocol: TCP
      appProtocol: rabbitmq.com/stream
      port: 5552
      targetPort: 5552
    - name: prometheus
      protocol: TCP
      appProtocol: prometheus.io/metrics
      port: 15692
      targetPort: 15692
  type: ClusterIP

  selector:
    app.kubernetes.io/name: rabbitmq-sandbox2

4. Create global configuration and create listener opn por "30552"

`apiVersion: k8s.nginx.org/v1alpha1
kind: GlobalConfiguration
metadata:
name: nginx-configuration
namespace: rabbitmq-system
spec:
listeners:

• name: rabbitmq-tcp
  port: 30552
  protocol: TCP`

6. Create a transport server
   `apiVersion: k8s.nginx.org/v1alpha1
   kind: TransportServer
   metadata:
   name: rabbitmq-listener
   namespace: rabbitmq-system
   spec:
   host: rabbitmq.apps.ocptest.domain.com
   listener:
   name: rabbitmq-tcp
   protocol: TCP
   upstreams:

• name: rabbitmq-sandbox2
  service: rabbitmq-sandbox2
  port: 5672
  action:
  pass: rabbitmq-sandbox2`

For testing:
andres@MIS3749:~$ curl rabbit.apps.ocptest.domain.com:30552 -vvv

• Trying 10.21.64.102:30552...
• TCP_NODELAY set
• connect to 10.xx.xx.xx2 port 30552 failed: Connection refused
• Failed to connect to rabbit.apps.ocptest.domain.com port 30552: Connection refused
• Closing connection 0

[brianehlert]

Your ports are not matching. You did sort out that you
You added additional ports to the service definition for rabbitmq - they all appear to be TCP ports, so I expect them to all be:

1. represented by a Listener in GlobalConfiguration
2. represented by a TransportServer
3. modify the ingress controller service to expose the port

5672, 15672, 5552, 15692 (maybe not this one if it is Prometheus) - these all need both a Listener and a TransportServer to represent them.

The Listener under GlobalConfiguration tells NGINX to listen for connections on this port. The TransportServer then defines how to handle the traffic from that listener. Modifying the Service of the ingress controller says which additional external ports to expose.

[andrescolodrero]

Hi,
i simpllify the service to:

kind: Service
apiVersion: v1
metadata:
  name: my-service
  namespace: rabbitmq-system
spec:
  ports:
    - name: rabbitmq
      protocol: TCP
      port: 5672
      targetPort: 5672
  selector:
    app.kubernetes.io/name: rabbitmq-sandbox2

and for simplicifty too, match the ports:

apiVersion: k8s.nginx.org/v1alpha1
kind: GlobalConfiguration 
metadata:
  name: nginx-configuration
  namespace: rabbitmq-system
spec: 
  listeners:
  - name: rabbitmq-tcp
    port: 5672
    protocol: TCP

and reconfigure the listener:

apiVersion: k8s.nginx.org/v1alpha1
kind: TransportServer
metadata:
  name: rabbitmq-listener
  namespace: rabbitmq-system
spec:
  host: rabbitmq.apps.ocptest.domain.com
  listener:
    name: rabbitmq-tcp
    protocol: TCP
upstreams:
- name: my-service
  service: my-service
  port: 5672
action:
  pass: my-service

still having same issue.
is there a way to check if the listener is running? or im missing something?

[andrescolodrero]

i can not see how the transporServer create a new listener or if the listener is running.
i forget to add the the configuration of the ingress service, where i added 5672 service.

Namespace:                rabbitmq-system
Labels:                   app.kubernetes.io/instance=nginxingress-sample
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=nginx-ingress
                          app.kubernetes.io/version=3.2.0
                          helm.sh/chart=nginx-ingress-0.18.0
Annotations:              meta.helm.sh/release-name: nginxingress-sample
                          meta.helm.sh/release-namespace: rabbitmq-system
Selector:                 app.kubernetes.io/instance=nginxingress-sample,app.kubernetes.io/name=nginx-ingress
Type:                     LoadBalancer
IP:                       172.30.154.152
Port:                     amqp  5672/TCP
TargetPort:               5672/TCP
NodePort:                 amqp  31195/TCP
Endpoints:                <none>
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  32440/TCP
Endpoints:                <none>
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  32525/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     30838

it is supossed now that nginx will be listening on 5672?

From a test pod:
~ $ curl 172.30.144.210:5672
curl: (1) Received HTTP/0.9 when not allowed

From out Kubernetes:
curl rabbitmq.apps.ocptest.domain.com:5672
curl: (7) Failed to connect to rabbitmq.apps.ocptest.domain.com port 5672: Connection refused

Testing ingress:
curl curl rabbitmq.apps.ocptest.arionbanki.is:80
(works)

[brianehlert]

the TransportServer resource is how this project defines TCP/UDP (aka stream) traffic handling.
Adding a TransportServer requires the GlobalConfiguration resource to be modified to define a listener for that port.
expecting the ingress controller listen on an arbitrary port other than 80 and 443 requires the ingress controller Service definition to be modified to expose that arbitrary port outside the K8s cluster.

Loadbalancer defines how traffic is routed to the K8s cluster(s) from the client.
Or it is used for exposing nodeport services. Which your ingress controller is one.

There is an example here of how to expose a DNS service that runs in K8s to external clients (the DNS used by clients outside of K8s): https://github.com/nginxinc/kubernetes-ingress/tree/main/examples/custom-resources/basic-tcp-udp

This project does not support exposing TCP/UDP traffic using the Ingress or VirtualServer resources.

That said. It is possible to expose your rabbitmq service using nodeport and loadbalancer and not through the ingress controller. It is your choice.
Generally when you choose to expose a service using an ingress controller, that backend service is of type ClusterIP (not exposed externally any other way).

[jasonwilliams14]

@andrescolodrero can you provide the output of:

kubectl get ts -A

If that comes back ok, can you send the output of:

kubectl exec -it -n nginx-ingress <name_of_pod> -- nginx -T

You can scrub the .conf but will give us an idea of what is being generated.

Thanks