This repository was archived by the owner on Jan 24, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 140
[FEATURE] Support multiple listeners by using advertisedListeners #669
Labels
type/feature
Indicates new functionality
Comments
@BewareMyPower Did you mean to assign me to this? |
@willwinberg Sorry it's my mistake, I just typed "w" and chose the false item |
This was referenced Sep 9, 2021
BewareMyPower
pushed a commit
that referenced
this issue
Sep 26, 2021
Fixes #669 Fixes #574 To support multiple listeners with multiple protocols, we need to set follow configration, and deprecated `kafkaAdvertisedListeners` and `kafkaListenerName`. > kafkaListeners=internal://0.0.0.0:9092,internal_ssl://0.0.0.0:9093,external://0.0.0.0:19002,external_ssl:0.0.0.0:19003 > kafkaProtocalMap=internal:PLAINTEXT,internal_ssl:SSL,external:PLAINTEXT,external_ssl:SSL > advertisedListeners={pulsar's listeners},internal:pulsar//192.168.1.10:9092,internal_ssl:pulsar//192.168.1.10:9093,external:pulsar//172.16.1.10:19002,external_ssl:pulsar://172.16.1.10:19003 1. Define the listenerNames and port of each listenerName in `kafkaListeners` 2. Define the listenerNames and protocol of each listenerName in `kafkaProtocalMap` 3. Add listenerNames and advertised address of each listenerName into `advertisedListeners` Kafka client should connect to the port of the listenerName according to KIP-103. Co-authored-by: wangjialing <wangjialing@cmss.chinamobile.com>
BewareMyPower
pushed a commit
that referenced
this issue
Sep 30, 2021
Fixes #669 Fixes #574 To support multiple listeners with multiple protocols, we need to set follow configration, and deprecated `kafkaAdvertisedListeners` and `kafkaListenerName`. > kafkaListeners=internal://0.0.0.0:9092,internal_ssl://0.0.0.0:9093,external://0.0.0.0:19002,external_ssl:0.0.0.0:19003 > kafkaProtocalMap=internal:PLAINTEXT,internal_ssl:SSL,external:PLAINTEXT,external_ssl:SSL > advertisedListeners={pulsar's listeners},internal:pulsar//192.168.1.10:9092,internal_ssl:pulsar//192.168.1.10:9093,external:pulsar//172.16.1.10:19002,external_ssl:pulsar://172.16.1.10:19003 1. Define the listenerNames and port of each listenerName in `kafkaListeners` 2. Define the listenerNames and protocol of each listenerName in `kafkaProtocalMap` 3. Add listenerNames and advertised address of each listenerName into `advertisedListeners` Kafka client should connect to the port of the listenerName according to KIP-103. Co-authored-by: wangjialing <wangjialing@cmss.chinamobile.com>
A follow-up to this effort: #818 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Is your feature request related to a problem? Please describe.
Here provide a design for #574, could also be a solution for #633
Suppose we deploy kop with pulsar on a host which have two NIC, such as a cloud ECS. 192.168.1.10 is the address for private network inside the cloud and 172.16.1.10 is the address for public network.
We start pulsar endpoints at 6650, 6651 on both NIC for pulsar, and start endpoints at 9092, 9093, 19002, 19003 for kafka.
We would config like this for pulsar:
We would config like this for kafka:
Kop current use
kafkaAdvertisedListeners
to config advertised address for kafka, but only support one address of one protocol.Describe alternatives you've considered
Deprecated the setting
kafkaAdvertisedListeners
.remove the function
KafkaRequestHandler.getProtocolDataToAdvertise()
and config kop like this:
We should create four endpoints in this case mapped port with listenerName:
PRIVATE:9092,PRIVATE_SSL:9093,PUBLIC:19002,PUBLIC_SSL:19003
When we receive a request from client, we get the listener name mapped with the endpoint which client connected to, use the listener name to do topic lookup and get the advertised address configed in
advertisedListeners
, get the protocal mapped with the listener name configed inkafkaProtocalMap
, then we could generate the lookup response to client.The text was updated successfully, but these errors were encountered: