[incubator/kafka] Fix initContainer failure which did not error

[ghost]

What this PR does / why we need it:

Over in #3754, I submitted a PR to allow optional external access to Kafka. During review, I changed from one kubectl image to another -- and apparently did not adequately vet the change.

A commenter @piter42zx at #3754 describes a failure mode as seen at minikube: #3754 (comment)

Special notes for your reviewer:

This PR fixes the issue identified above. It uses a kubectl image which is not based on docker "scratch" -- and which brings with it sh so that we can be more paranoid about our init container's behavior. I've verified using the following override file that Kafka is exposed as expected by the commenter referenced above.

To best validate that the external endpoint should be available, run:

kubectl get endpoints -l app=kafka

replicas: 1

external:
  domain: 192.168.99.100
  enabled: true
  firstListenerPort: 31090
  servicePort: 19092

configurationOverrides:
  advertised.listeners: EXTERNAL://192.168.99.100:$((31090 + ${KAFKA_BROKER_ID}))
  auto.create.topics.enable: "true"
  listener.security.protocol.map: PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
  unclean.leader.election.enable: "false"

zookeeper:
  servers: 1

[ghost]

/assign @benjigoldberg

[benjigoldberg]

@josdotso can you go into more detail about why we need to sh vs just use the scratch image? I don't follow the problem totally.

[ghost]

@benjigoldberg

Using the old from scratch image did not label the pods at all, exited 0, and produced no init container log at all.

With sh in the mix, we can set -euxc which tells sh to error upon any non-zero command, error on any vars it cannot interpolate, enable shell verbosity enough to tell us what shell command ran. stdout gives useful output anyway -- but only on success. Even if the scratch approach worked in most cases, it would never tell us what shell command started kubectl. Therefore, we must use a shell here to properly debug the interpolation of the variables.

When label is not present, the services created for external access are set to use dead endpoints. This happens because selector doesn't match unlabeled pods.

Here's an example log output after this PR:

$ kubectl logs kafka-kafka-0 -c init-ext
+ kubectl label pods kafka-kafka-0 --namespace default pod=kafka-kafka-0
pod "kafka-kafka-0" labeled

[benjigoldberg]

Thanks for the explanation 👍 makes sense to me

[benjigoldberg]

/ok-to-test

[benjigoldberg]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: benjigoldberg, josdotso

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• incubator/kafka/OWNERS [benjigoldberg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ghost]

Thanks @benjigoldberg and sorry for the mishap!

[benjigoldberg]

@josdotso np, it happens. Thanks for patching this!