[incubator/kafka] Support option to expose Kafka via NodePort

[ghost]

README.md excerpt:

Connecting to Kafka from outside Kubernetes

Review and optioanlly override to enable the example text concerned with external access in values.yaml.

Once configured, you should be able to reach Kafka via NodePorts, one per replica. In kops where private, topology is enabled, this feature publishes an internal round-robin DNS record using the following naming scheme. The external access feature of this chart was tested with kops on AWS using flannel networking. If you wish to enable external access to Kafka running in kops, your security groups will likely need to be adjusted to allow non-Kubernetes nodes (e.g. bastion) to access the Kafka external listener port range.

{{ .Release.Name }}.{{ .Values.external.domain }}

Port numbers for external access used at container and NodePort are unique to each container in the StatefulSet. Using the default external.firstListenerPort number with a replicas value of 3, the following container and NodePorts will be opened for external access: 31090, 31091, 31092. All of these ports should be reachable from any host to NodePorts are exposed because Kubernetes routes each NodePort from entry node to pod/container listening on the same port (e.g. 31091).

The external.servicePort at each external access service (one such service per pod) is a relay toward
the a containerPort with a number matching its respective NodePort. The range of NodePorts is set, but should not actually listen, on all Kafka pods in the StatefulSet. As any given pod will listen only one such port at a time, setting the range at every Kafka pod is a reasonably safe configuration.

[ghost]

/assign @mattfarina

[ghost]

/assign @unguiculus

[unguiculus]

cc @benjigoldberg @faraazkhan @h0tbird

[benjigoldberg]

@josdotso This is a huge step for this chart! Thanks for doing this work. I can think of a number of people who will be thrilled that external access is now an option.

One question about image tags.

The init container/pod selector game is a bit amusing, but really Kubernetes doesnt give many other options. I attempted this same functionality with nginx-ingress/headless service/TCP exposure but never had much luck, I think mostly due to kafka configurations which you appear to have nailed in your config. Lovely to see this configuration though -- it involves far fewer moving parts than other attempts that I have seen!

[benjigoldberg]

Is there a better image tag by any chance?

[eicnix]

Shouldn't the image also be configureable? Otherwise people that cannot download from docker hub are unable to use this feature.

[ghost]

Agreed on both points. I'll fix this ASAP. Thanks!!!

[ghost]

Note to self: typo ^

[ghost]

@benjigoldberg , @eicnix : Please take another look. I believe I've addressed all concerns above.

[jdumars]

/ok-to-test

[benjigoldberg]

LGTM

[unguiculus]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: josdotso, unguiculus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [unguiculus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[piter42zx]

Hi,

First of all, thanks a lot for this!

I installed the chart locally in a minikube with the external access enabled and configured "advertised.listeners": |-
EXTERNAL://192.168.99.100:$((31090 + ${KAFKA_BROKER_ID}))
where 192.168.99.100 is the ip of the minikube.

My objective is to access the Kafka broker from outside the K8s cluster.

I used port-forward on the zookeeper pod and successfully created the topic test1 (as explained in the chart notes).

However, when I try to create a message using the following command to start an interactive message producer session I get the error shown below:
kafka-console-producer --broker-list 192.168.99.100:31090 --topic test1

WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

Am I missing some configuration?

Thanks in advance!

[benjigoldberg]

@josdotso I havent used the external config yet personally, any advice you can offer to @piter42zx?

[ghost]

@piter42zx : Thanks for the bug report!

See: #4400

cc: @benjigoldberg