Skip to content
This repository was archived by the owner on Jan 24, 2024. It is now read-only.

[FEATURE] Support multiple listeners by using advertisedListeners #669

Closed
wangjialing218 opened this issue Aug 20, 2021 · 3 comments · Fixed by #742
Closed

[FEATURE] Support multiple listeners by using advertisedListeners #669

wangjialing218 opened this issue Aug 20, 2021 · 3 comments · Fixed by #742
Assignees
Labels
type/feature Indicates new functionality

Comments

@wangjialing218
Copy link
Contributor

wangjialing218 commented Aug 20, 2021

Is your feature request related to a problem? Please describe.
Here provide a design for #574, could also be a solution for #633

Suppose we deploy kop with pulsar on a host which have two NIC, such as a cloud ECS. 192.168.1.10 is the address for private network inside the cloud and 172.16.1.10 is the address for public network.
We start pulsar endpoints at 6650, 6651 on both NIC for pulsar, and start endpoints at 9092, 9093, 19002, 19003 for kafka.

We would config like this for pulsar:

internalListenerName=pulsar_private
advertisedListeners=pulsar_private:pulsar://192.168.1.10:6650,pulsar_private_ssl:pulsar+ssl://192.168.1.10:6651,pulsar_public:pulsar://172.16.1.10:6650,pulsar_pulic_ssl:pulsar+ssl://172.16.1.10:6651

We would config like this for kafka:

listeners=PRIVATE://0.0.0.0:9092,PRIVATE_SSL://0.0.0.0:9093,PUBLIC://0.0.0.0:19002,PUBLIC_SSL:0.0.0.0:19003
advertised.listeners=PRIVATE://192.168.1.10:9092,PRIVATE_SSL://192.168.1.10:9093,PUBLIC://172.16.1.10:19002,PUBLIC_SSL:172.16.1.10:19003
listener,security.protocal.map=PRIVATE:PLAINTEXT,PRIVATE_SSL:SSL,PUBLIC:PLAINTEXT,PUBLIC_SSL:SSL

Kop current use kafkaAdvertisedListeners to config advertised address for kafka, but only support one address of one protocol.

Describe alternatives you've considered
Deprecated the setting kafkaAdvertisedListeners.
remove the function KafkaRequestHandler.getProtocolDataToAdvertise()
and config kop like this:

kafkaListeners=PRIVATE://0.0.0.0:9092,PRIVATE_SSL://0.0.0.0:9093,PUBLIC://0.0.0.0:19002,PUBLIC_SSL:0.0.0.0:19003
kafkaProtocalMap=PRIVATE:PLAINTEXT,PRIVATE_SSL:SSL,PUBLIC:PLAINTEXT,PUBLIC_SSL:SSL
advertisedListeners={pulsar's listeners},PRIVATE:pulsar//192.168.1.10:9092,PRIVATE_SSL:pulsar//192.168.1.10:9093,PUBLIC:pulsar//172.16.1.10:19002,PUBLIC_SSL:pulsar:172.16.1.10:19003

We should create four endpoints in this case mapped port with listenerName: PRIVATE:9092,PRIVATE_SSL:9093,PUBLIC:19002,PUBLIC_SSL:19003

When we receive a request from client, we get the listener name mapped with the endpoint which client connected to, use the listener name to do topic lookup and get the advertised address configed in advertisedListeners, get the protocal mapped with the listener name configed in kafkaProtocalMap, then we could generate the lookup response to client.

@wangjialing218 wangjialing218 added the type/feature Indicates new functionality label Aug 20, 2021
BewareMyPower pushed a commit that referenced this issue Aug 26, 2021
…Name (#670)

This is the first step for #669, only support one listener which configed in `kafkaListenerName`, and only support PLAINTEXT

Co-authored-by: wangjialing <wangjialing@cmss.chinamobile.com>
BewareMyPower pushed a commit that referenced this issue Aug 26, 2021
…Name (#670)

This is the first step for #669, only support one listener which configed in `kafkaListenerName`, and only support PLAINTEXT

Co-authored-by: wangjialing <wangjialing@cmss.chinamobile.com>
@willwinberg
Copy link

@BewareMyPower Did you mean to assign me to this?

@BewareMyPower
Copy link
Collaborator

@willwinberg Sorry it's my mistake, I just typed "w" and chose the false item

BewareMyPower pushed a commit that referenced this issue Sep 26, 2021

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
Fixes #669
Fixes #574

To support multiple listeners with multiple protocols, we need to set follow configration, and deprecated `kafkaAdvertisedListeners` and `kafkaListenerName`.

> kafkaListeners=internal://0.0.0.0:9092,internal_ssl://0.0.0.0:9093,external://0.0.0.0:19002,external_ssl:0.0.0.0:19003
> kafkaProtocalMap=internal:PLAINTEXT,internal_ssl:SSL,external:PLAINTEXT,external_ssl:SSL
> advertisedListeners={pulsar's listeners},internal:pulsar//192.168.1.10:9092,internal_ssl:pulsar//192.168.1.10:9093,external:pulsar//172.16.1.10:19002,external_ssl:pulsar://172.16.1.10:19003

1. Define the listenerNames and port of each listenerName in `kafkaListeners`
2. Define the listenerNames and protocol of each listenerName in `kafkaProtocalMap`
3. Add listenerNames and advertised address of each listenerName into `advertisedListeners`

Kafka client should connect to the port of the listenerName according to KIP-103.

Co-authored-by: wangjialing <wangjialing@cmss.chinamobile.com>
BewareMyPower pushed a commit that referenced this issue Sep 30, 2021
Fixes #669
Fixes #574

To support multiple listeners with multiple protocols, we need to set follow configration, and deprecated `kafkaAdvertisedListeners` and `kafkaListenerName`.

> kafkaListeners=internal://0.0.0.0:9092,internal_ssl://0.0.0.0:9093,external://0.0.0.0:19002,external_ssl:0.0.0.0:19003
> kafkaProtocalMap=internal:PLAINTEXT,internal_ssl:SSL,external:PLAINTEXT,external_ssl:SSL
> advertisedListeners={pulsar's listeners},internal:pulsar//192.168.1.10:9092,internal_ssl:pulsar//192.168.1.10:9093,external:pulsar//172.16.1.10:19002,external_ssl:pulsar://172.16.1.10:19003

1. Define the listenerNames and port of each listenerName in `kafkaListeners`
2. Define the listenerNames and protocol of each listenerName in `kafkaProtocalMap`
3. Add listenerNames and advertised address of each listenerName into `advertisedListeners`

Kafka client should connect to the port of the listenerName according to KIP-103.

Co-authored-by: wangjialing <wangjialing@cmss.chinamobile.com>
@EronWright
Copy link
Contributor

A follow-up to this effort: #818

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type/feature Indicates new functionality
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants