Allow configuration of hosts on worker nodes using --add-host flag

[ciroque]

What would you like to be added:

In the Docker provision module there is logic that assembles the arguments for running Nodes. While there are many arguments that can be specified, one that is missing is the --add-host flag.

The addition of this flag would be very beneficial to alleviate an issue we have happened across, described below.

The general idea would be to add a new block (extra_hosts ?) to the configuration that defines a single flag instance to be passed through to the Docker run command, thus this:

extra_hosts {
    alias: host.docker.internal
    name: 172.17.0.1
}

would translate to: --add-host=host.docker.internal:172.17.0.1

Of course naming is hard, so those names are just placeholders pending discussion.

I am willing to do the work, scheduling and timing T.B.D. if the community believes this is a worthwhile effort.

Why is this needed:

Docker Networking and the magic host

When running Docker on Mac and Windows there exists a magic host entry (host.docker.internal) that can be used to reference other containers running in Docker.
This host is not available, by default, when running Docker in Linux. It is possible to add this host when running an image with the --add-host=host.docker.internal:host-gateway flag, e.g.:

### Note: the --add-host flag is required only when running Docker on Linux...

docker run -rm -dit --add-host:host.docker.internal:host-gateway

This is a pretty straight-forward way to ensure the DNS entry is available in Linux.

While there are a number of available options for configuring system nodes using Kind; the option to add hosts does not exist.

This means there is no straight-forward way to add the magic Docker DNS entry to the Kind Worker Nodes when running on Linux.

Options

Three options were considered for solving this issue:

1. Manually add the DNS entry into the worker nodes /etc/hosts file;
2. Submit a PR for Kind that adds support for the --add-hosts flag;
3. Use CoreDNS hosts entries to inject the magic host;

Spike the DNS entries

This was a super hacky way of solving the issue by running a command on each worker node that added an entry to the hosts file for host.docker.internal.

docker exec -it mara-worker bash -c "echo 172.17.0.1 host.docker.internal >> /etc/hosts"
docker exec -it mara-worker2 bash -c "echo 172.17.0.1 host.docker.internal >> /etc/hosts"

Note: The 172.17.0.1 magic IP address is the default IP for the Docker bridge network.

Submit a PR to Kind

This is actually the most desirable solution. It is clean, it improves the quality of an existing project that is widely used and respected, and it avoids any hackery to the Kubernetes cluster.

Update CoreDNS hosts

This involves editing a ConfigMap in the Cluster used by CoreDNS for configuration. CoreDNS allows configuration of plugins in the ConfigMap, one of which is the hosts plugin.

This involves a manual step, and manual input which can be error-prone; however, the editing mechanism in k8s includes a validation
step that disallows invalid ConfigMaps from being applied. So there is a safety net in place.

The general idea is:

1. Use kubectl edit to edit the CoreDNS ConfigMap;
2. Add the magic host entry to the Corefile section;
3. Save the ConfigMap (validation happens automatically);
4. Restart the CoreDNS Pods;

Workaround

We are presently using the third option, updating the CoreDNS Corefile with the record, but as noted, being able to do this in Kind would be ideal.

[BenTheElder]

extra_hosts {
alias: host.docker.internal
name: 172.17.0.1
}

I think we should just correctly handle this case instead of adding another config surface

kind/images/base/files/usr/local/bin/entrypoint

Line 452 in 3610f60

docker_host_ip="$( (head -n1 <(timeout 5 getent ahostsv4 'host.docker.internal') | cut -d' ' -f1) || true)"

[ciroque]

extra_hosts {
alias: host.docker.internal
name: 172.17.0.1
}

I think we should just correctly handle this case instead of adding another config surface

kind/images/base/files/usr/local/bin/entrypoint

Line 452 in 3610f60

docker_host_ip="$( (head -n1 <(timeout 5 getent ahostsv4 'host.docker.internal') | cut -d' ' -f1) || true)"

Okay, cool, I will take a closer look next week and come back with a proposal around updating that, thanks!

[aaomidi]

I think we should just correctly handle this case instead of adding another config surface

I think another config surface makes a bit of sense here since this is technically a configurable surface on Docker (?).

It's hard to anticipate every need going into the future IMO, and this definitely isn't something that should be on by default I think either.

[BenTheElder]

I think another config surface makes a bit of sense here since this is technically a configurable surface on Docker (?).

From a maintainability point of view we do NOT support all possible docker options. The goal is to provide Kubernetes clusters, and every knob is another possible broken interaction that we need to test and support.

It's hard to anticipate every need going into the future IMO, and this definitely isn't something that should be on by default I think either.

It's not supportable to cover every possible usage anyhow.
I don't see why this couldn't be on by default, docker containers already have special hostnames like this.

[aaomidi]

I don't see why this couldn't be on by default, docker containers already have special hostnames like this.

Security considerations tbh. This allows a container to talk to a service that's on the host, and that by default seems like it can open a can of worms.

I was wrong, see below.

[BenTheElder]

Security considerations tbh. This allows a container to talk to a service that's on the host, and that by default seems like it can open a can of worms.

This is possible with or without the host name. The hostname just makes it slightly less complicated to discover the IP.

There are docker installs (mac, Windows) that already have the host name available by default.

One thing not discussed so far: podman and #3429 (nerctl)

[aaomidi]

This is possible with or without the host name. The hostname just makes it slightly less complicated to discover the IP.

Yep, sorry I forgot to come back to this issue later. You're right this has no impact on security considerations from that perspective.