Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only Terminate mode is supported. Route TLSRoute not supported. #6922

Open
1 task done
mlalam opened this issue Jan 6, 2025 · 13 comments
Open
1 task done

Only Terminate mode is supported. Route TLSRoute not supported. #6922

mlalam opened this issue Jan 6, 2025 · 13 comments
Labels
bug Something isn't working

Comments

@mlalam
Copy link

mlalam commented Jan 6, 2025

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Deployed following manifest:

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: kong
  namespace: kong-system
spec:
  gatewayClassName: kong
  listeners:
  - name: wildcard-tls
    protocol: TLS
    port: 443
    hostname: "*.example.com"
    tls:
      mode: Passthrough
      certificateRefs:
      - group: ''
        kind: Secret
        name: default-ingress-cert      
    allowedRoutes:
      kinds:
      - group: gateway.networking.k8s.io
        kind: TLSRoute    
      namespaces:
        from: Selector
        selector:
          matchLabels:
            shared-gateway-access: "true"

Expected Behavior

TLS listener with passthrough mode must be created successfully.

Steps To Reproduce

Deploy the above manifest using KIC 3.4.1 on kubernetes 1.29

Kong Ingress Controller version

3.4.1

Kubernetes version

1.29

Anything else?

Gateway created with following listener exception.
FYI - I'm using AWS ALB Controller component for provisioning NLB.

Status:
Addresses:
Type: Hostname
Value: xxxxxx
Conditions:
Last Transition Time: 2025-01-06T22:50:30Z
Message: Listener 0 is not accepted.
Observed Generation: 14
Reason: ListenersNotValid
Status: False
Type: Accepted
Last Transition Time: 2025-01-06T22:50:30Z
Message: There are other conditions that are not yet ready
Observed Generation: 14
Reason: Pending
Status: False
Type: Programmed
Last Transition Time: 2025-01-06T19:13:03Z
Message:
Observed Generation: 8
Reason: Ready
Status: True
Type: DataPlaneReady
Last Transition Time: 2025-01-06T19:13:03Z
Message:
Observed Generation: 8
Reason: Ready
Status: True
Type: ControlPlaneReady
Last Transition Time: 2025-01-06T19:13:03Z
Message:
Observed Generation: 8
Reason: Ready
Status: True
Type: GatewayService
Listeners:
Attached Routes: 0
Conditions:
Last Transition Time: 2025-01-06T22:50:30Z
Message:
Observed Generation: 14
Reason: NoConflicts
Status: False
Type: Conflicted
Last Transition Time: 2025-01-06T22:50:30Z
Message:
Observed Generation: 14
Reason: UnsupportedProtocol
Status: False
Type: Accepted
Last Transition Time: 2025-01-06T22:50:30Z
Message:
Observed Generation: 14
Reason: Pending
Status: False
Type: Programmed
Last Transition Time: 2025-01-06T22:50:30Z
Message: Only Terminate mode is supported. Route TLSRoute not supported.
Observed Generation: 14
Reason: InvalidRouteKinds
Status: False
Type: ResolvedRefs
Name: wildcard-tls
Supported Kinds:
Events:

@mlalam mlalam added the bug Something isn't working label Jan 6, 2025
@mlalam
Copy link
Author

mlalam commented Jan 6, 2025

Refer to #6912 details as well.

❯ k api-resources | grep -E "kong|gateway"

ingressclassparameterses configuration.konghq.com/v1alpha1 true IngressClassParameters
kongclusterplugins kcp configuration.konghq.com/v1 false KongClusterPlugin
kongconsumergroups kcg configuration.konghq.com/v1beta1 true KongConsumerGroup
kongconsumers kc configuration.konghq.com/v1 true KongConsumer
kongcustomentities kce configuration.konghq.com/v1alpha1 true KongCustomEntity
kongingresses ki configuration.konghq.com/v1 true KongIngress
konglicenses kl configuration.konghq.com/v1alpha1 false KongLicense
kongplugins kp configuration.konghq.com/v1 true KongPlugin
kongupstreampolicies kup configuration.konghq.com/v1beta1 true KongUpstreamPolicy
kongvaults kv configuration.konghq.com/v1alpha1 false KongVault
tcpingresses configuration.konghq.com/v1beta1 true TCPIngress
udpingresses configuration.konghq.com/v1beta1 true UDPIngress
controlplanes kocp gateway-operator.konghq.com/v1beta1 true ControlPlane
dataplanemetricsextensions gateway-operator.konghq.com/v1alpha1 true DataPlaneMetricsExtension
dataplanes kodp gateway-operator.konghq.com/v1beta1 true DataPlane
gatewayconfigurations kogc gateway-operator.konghq.com/v1beta1 true GatewayConfiguration
kongplugininstallations kpi gateway-operator.konghq.com/v1alpha1 true KongPluginInstallation
konnectextensions gateway-operator.konghq.com/v1alpha1 true KonnectExtension
backendlbpolicies blbpolicy gateway.networking.k8s.io/v1alpha2 true BackendLBPolicy
backendtlspolicies btlspolicy gateway.networking.k8s.io/v1alpha3 true BackendTLSPolicy
gatewayclasses gc gateway.networking.k8s.io/v1 false GatewayClass
gateways gtw gateway.networking.k8s.io/v1 true Gateway
grpcroutes gateway.networking.k8s.io/v1 true GRPCRoute
httproutes gateway.networking.k8s.io/v1 true HTTPRoute
referencegrants refgrant gateway.networking.k8s.io/v1beta1 true ReferenceGrant
tcproutes gateway.networking.k8s.io/v1alpha2 true TCPRoute
tlsroutes gateway.networking.k8s.io/v1alpha2 true TLSRoute
udproutes gateway.networking.k8s.io/v1alpha2 true UDPRoute

Also, added "CONTROLLER_FEATURE_GATES" env variable with "GatewayAlpha=true" as its value for gateway proxy.

Using gateway operator 1.4 with "experimental" CRDs from https://github.com/kubernetes-sigs/gateway-api/releases/tag/v1.2.1

@mlalam
Copy link
Author

mlalam commented Jan 6, 2025

@mheap, any suggestion?

I am following instructions based on this documentation - https://docs.konghq.com/kubernetes-ingress-controller/latest/guides/services/tls/#tls-passthrough

@pmalek , including you to get your insights based on Kong/gateway-operator#112 . I see that issue has been closed with "Closing as not planned for now" comments.

Does this mean TLS Route is not supported yet in kong (Kong/gateway-operator#64 )?

@mlalam
Copy link
Author

mlalam commented Jan 8, 2025

Anyone faced similar exception/error? Hoping to get some help on TLS pass-through implementation with KIC/Kong Gtw using KGO.

@mlalam
Copy link
Author

mlalam commented Jan 9, 2025

@pmalek , any suggestion/recommendations? thanks.

@pmalek
Copy link
Member

pmalek commented Jan 9, 2025

@mlalam TLSListener is not supported yet as per https://github.com/Kong/gateway-operator/blob/20f084194be07bb17d149e0b1b68e603e05a9ccd/controller/gateway/controller_reconciler_utils.go#L524-L534 and the fact that #64 is still open.

You should observe something like in your Gateway's status:

conditions:
  - lastTransitionTime: "2025-01-09T16:57:40Z"
    message: Listener 0 is not accepted.
    observedGeneration: 1
    reason: ListenersNotValid
    status: "False"
    type: Accepted
  - lastTransitionTime: "2025-01-09T16:57:40Z"
    message: There are other conditions that are not yet ready
    observedGeneration: 1
    reason: Pending
    status: "False"
    type: Programmed
listeners:
  - attachedRoutes: 0
    conditions:
      - lastTransitionTime: "2025-01-09T16:57:40Z"
        message: ""
        observedGeneration: 1
        reason: NoConflicts
        status: "False"
        type: Conflicted
      - lastTransitionTime: "2025-01-09T16:57:40Z"
        message: ""
        observedGeneration: 1
        reason: UnsupportedProtocol
        status: "False"
        type: Accepted
      - lastTransitionTime: "2025-01-09T16:57:40Z"
        message: ""
        observedGeneration: 1
        reason: Pending
        status: "False"
        type: Programmed
      - lastTransitionTime: "2025-01-09T16:57:40Z"
        message: Only Terminate mode is supported. Referenced secret kong-system/default-ingress-cert does not exist. Route TLSRoute not supported.
        observedGeneration: 1
        reason: InvalidRouteKinds
        status: "False"
        type: ResolvedRefs
    name: wildcard-tls
    supportedKinds: []

note the UnsupportedProtocol and ResolvedRefs listener conditions.

Hope that helps.

@mlalam
Copy link
Author

mlalam commented Jan 13, 2025

thanks @pmalek .
I see the open ticket #64 hasn't been touched for 8 years. Is it in the roadmap for KGO in near future?

@pmalek
Copy link
Member

pmalek commented Jan 14, 2025

thanks @pmalek . I see the open ticket #64 hasn't been touched for 8 years. Is it in the roadmap for KGO in near future?

Cannot speak for this PR but it's pretty old and looks unrelated to this issue here (it's about route's protocols not Gateway's listeners).

@mlalam
Copy link
Author

mlalam commented Jan 14, 2025

hmm. I will provide my requirement/situation, hope you can help me with some direction.

  1. The upstream service does certificate based authentication.
  2. If I set "konghq.com/client-cert" on the service annotation, it works to expectation and I can see that cert coming through for validation in the application SSL log.
  3. If I remove the cert and send it through browser (yes, tls_verify is turned on with CA certs added through service annotations), I don't see the cert coming through to the app side.

I can see the cert available in kong though. How do I forward the cert from kong gateway to the upstream service?

I tried this post-function plugin recommendation, but still it is not working.

https://stackoverflow.com/questions/68665466/kong-gateway-how-to-retrieve-client-cert-and-set-to-header

Any help/assist is much appreciated. I can't use mTls plugin because I am using OSS version.

@mlalam
Copy link
Author

mlalam commented Jan 16, 2025

@pmalek , got any suggestions?

@mlalam
Copy link
Author

mlalam commented Jan 21, 2025

hello, anyone out there to help me out on my requirement (TLS Pass-through)?

@pmalek
Copy link
Member

pmalek commented Jan 22, 2025

Hi @mlalam

Sorry for the delayed reponse.

I only had time to check 1 thing which is whether the example manifest for TLSRoute passthrough using Gateway API would make sense for this scenario but I can see that KIC still marks the TLSRoute as unsupported even with GatewayAlpha feature flag:

2025-01-22T16:52:42Z    debug    controllers.TLSRoute    Processing tlsroute    {"GatewayV1Alpha2TLSRoute": {"name":"tlsecho","namespace":"default"}, "v": 1, "namespace": "default", "name": "tlsecho"}
2025-01-22T16:52:42Z    debug    controllers.TLSRoute    Checking deletion timestamp    {"GatewayV1Alpha2TLSRoute": {"name":"tlsecho","namespace":"default"}, "v": 1, "namespace": "default", "name": "tlsecho"}
2025-01-22T16:52:42Z    debug    controllers.TLSRoute    Retrieving GatewayClass and Gateway for route    {"GatewayV1Alpha2TLSRoute": {"name":"tlsecho","namespace":"default"}, "v": 1, "namespace": "default", "name": "tlsecho"}
2025-01-22T16:52:42Z    debug    controllers.TLSRoute    Listener does not support this route    {"GatewayV1Alpha2TLSRoute": {"name":"tlsecho","namespace":"default"}, "parentRef.gateway": "default/kong", "listener": "http", "v": 1, "reason": "unsupported route kind"}
2025-01-22T16:52:42Z    debug    controllers.TLSRoute    Listener is not ready    {"GatewayV1Alpha2TLSRoute": {"name":"tlsecho","namespace":"default"}, "parentRef.gateway": "default/kong", "listener": "tls", "v": 1, "reason": "listener not programmed yet"}
2025-01-22T16:52:42Z    debug    controllers.TLSRoute    Checking if the tlsroute's gateways are ready    {"GatewayV1Alpha2TLSRoute": {"name":"tlsecho","namespace":"default"}, "v": 1, "namespace": "default", "name": "tlsecho"}
2025-01-22T16:52:42Z    debug    controllers.TLSRoute    Ensuring status contains Gateway associations    {"GatewayV1Alpha2TLSRoute": {"name":"tlsecho","namespace":"default"}, "v": 1, "namespace": "default", "name": "tlsecho"}
2025-01-22T16:52:42Z    debug    controllers.TLSRoute    TLSRoute not configured, requeueing    {"GatewayV1Alpha2TLSRoute": {"name":"tlsecho","namespace":"default"}, "v": 1, "namespace": "default", "name": "tlsecho"}

When I find some time I'll try to get more insights into this issue.

@mlalam
Copy link
Author

mlalam commented Jan 24, 2025

Hi @pmalek, very surprised why gateway operator doesn't support this, when the underlying components KIC, kong gateway (nginx behind it) supports this. Currently, we are stuck and unable to upgrade to kong gateway due to this missing feature. I will be waiting for any update on this. thanks for your support on this so far!

@mlalam
Copy link
Author

mlalam commented Feb 3, 2025

Hello @pmalek, got any update? Should I proceed with unmanaged gateway setup and ignore gateway operator to make it work?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants