TCP Stream does not work after migration

[adcoutinho]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

Kong Enterprise 3.2.2.1

Current Behavior

Hello guys, recently I need to migrate from Kong Gateway OSS 2.8.1 deployed as container in AWS ECS to Kong Enterprise 3.2.2.1 Hybrid Mode on AWS EC2 with Autoscaling Groups, because ECS has a limiting number of ports per container.
After setup the solution all HTTP traffic works as expected, but none of my TCP Streams works anymore.
Right now all HTTP traffic flows through my new environment but TCP stills in the old one.

Steps To Reproduce

Configs
Service:

Route:

Anything else?

error.log:

[bungle]

@adcoutinho, can you try setting router_flavor to traditional:
https://github.com/Kong/kong/blob/master/kong.conf.default#L1666

or with ENV variable: KONG_ROUTER_FLAVOR=traditional kong restart, and report back at us, if it helps.

[adcoutinho]

Thanks @bungle router_flavor = traditional works, but as I could check it will be removed in the next major release as informed in https://github.com/Kong/kong/blob/master/kong.conf.default#L1666.

So I change my config to use Upstreams and it works as expected, whithout router_flavor change:
Ex.:

services:
- connect_timeout: 60000
  enabled: true
  host: finopsdb
  name: teste-tcp
  port: 5432
  protocol: tcp
  retries: 5
  routes:
  - destinations:
    - ip: 0.0.0.0/0
      port: 7000
    name: teste-tcp
    preserve_host: false
    protocols:
    - tcp
    strip_path: false
  tags:
  - teste
  write_timeout: 60000
upstreams:
- algorithm: consistent-hashing
  hash_on: ip
  healthchecks:
    active:
      concurrency: 10
      healthy:
        http_statuses:
        - 200
        - 302
        interval: 0
        successes: 0
      http_path: /
      https_verify_certificate: false
      timeout: 1
      type: tcp
      unhealthy:
        interval: 0
        tcp_failures: 0
        timeouts: 0
    threshold: 0
  name: finopsdb
  slots: 10000
  targets:
  - target: XXX:5432
    weight: 100

[adcoutinho]

Back on this, after I recreate Kong Gateway Nodes with more Stream ports I saw errors again:

2023/05/09 22:16:46 [error] 4018#0: *58545 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.37, server: 0.0.0.0:7005
2023/05/09 22:16:47 [error] 4018#0: *58546 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.30, server: 0.0.0.0:7003
2023/05/09 22:16:47 [error] 4018#0: *58547 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.37, server: 0.0.0.0:7001
2023/05/09 22:16:47 [error] 4018#0: *58548 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.37, server: 0.0.0.0:7000
2023/05/09 22:16:47 [error] 4018#0: *58549 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.30, server: 0.0.0.0:7005
2023/05/09 22:16:47 [error] 4018#0: *58550 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.37, server: 0.0.0.0:7005
2023/05/09 22:16:47 [error] 4018#0: *58551 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.37, server: 0.0.0.0:7000
2023/05/09 22:16:47 [error] 4018#0: *58552 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.37, server: 0.0.0.0:7000
2023/05/09 22:16:47 [error] 4018#0: *58553 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.30, server: 0.0.0.0:7003
2023/05/09 22:16:48 [error] 4018#0: *58555 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.37, server: 0.0.0.0:7004
2023/05/09 22:16:48 [error] 4018#0: *58556 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.37, server: 0.0.0.0:7000
2023/05/09 22:16:48 [error] 4018#0: *58557 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.37, server: 0.0.0.0:7003
2023/05/09 22:16:48 [error] 4018#0: *58559 stream [lua] handler.lua:1084: before(): no Route found with those values while prereading client data, client: 10.253.8.37, server: 0.0.0.0:7000

If I delete and recreate some TCP Routes those errors almost stop, but when I reboot or recreate the vm they came back for all Stream Listen ports.
It seems the only way to go by now is setting KONG_ROUTER_FLAVOR=traditional :(

[adcoutinho]

Unfortunately I stumbled with the same problem treated here #10514 as I try to work with six Stream Ports.

[locao]

@adcoutinho #10514 was not released yet. The good news is Kong 3.3.0 is around the corner and will include this fix.

[hanshuebner]

Is this problem solved by #10514 ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[adcoutinho]

Sorry, I forgot to post my feedback.

Yes #10514 solved the problem!

[adcoutinho]

Thanks to everyone involved !!!