No network interface created on Host that links to Guest

[AlmirKadric]

Currently there is no virtual network interface created on the Host (MacOSX) which links to the interface inside the Guest (VM). This creates a lot of limitation in terms of IP routing.

As it stands the Guest can route outwards and even ping the Host but the other way is not true. This also adds additional limitations to docker which are not desirable.

What I am trying to achieve is to create a route in the Host which hops over the Guest into one of its containers. However since there is no IP mapped to the Guest inside the Host there is no way to do this.

VirtualBox solves this by create a virtual interface with an IP address as such:

vboxnet5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
    ether 0a:00:27:00:00:05
    inet 172.16.41.1 netmask 0xffffff00 broadcast 172.16.41.255

Also in the past this was solved with docker-machine using the following:

sudo route -n add 172.17.0.0/16 $(docker-machine ip <MACHINE NAME>)

Furthermore, from what I have seen so far it seems that bhyve and xhyve do support creating tap interfaces on the Host which should map to the Guest which would solve this whole issue.
https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html
https://github.com/mist64/xhyve#networking
http://tuntaposx.sourceforge.net/

[AlmirKadric]

Edit: Seems this would be possible if an option is added to the docker configuration to allow the setting of the tap interface.

https://forums.docker.com/t/support-tap-interface-for-direct-container-access-incl-multi-host/17835/3

[rn]

HyperKit allows you to create network interface (see hyperkit.1). For example:

sudo ./build/com.docker.hyperkit -A -m 512M -s 0:0,hostbridge c -s 2:0,virtio-net -l com1,stdio -f kexec,vmlinuz64,initrd.img,test/initrd.gz,earlyprintk=serial console=ttyS

will create a bridge interface on the host and a virtio network interface inside the VM. If dhcp is configured inside the VM it will get an IP address in the same range as the host bridge. For example in my case:

eth0      Link encap:Ethernet  HWaddr 1E:9D:A1:45:CF:32
          inet addr:192.168.64.24  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1896 (1.8 KiB)  TX bytes:1760 (1.7 KiB)

and I can ping that IP address from the host.

[ijc]

I don't think this is a hyperkit issue since as @rneugeba says hyperkit already supports this sort of thing.

I think this is actually a Docker 4 Mac feature request and so should be made via the D4M feedback channels (please consult the docs for the right place to go). However, if I understand the request correctly, the answer is that this functionality is exposed via the usual -p option to docker run (and its compose and swarm equivalents) to publish the container ports on the host. If I've misunderstood the use case then please clarify when making the request to D4M.

[AlmirKadric]

Thanks for that, I found a entry inside the forums with exactly what I'm after
https://forums.docker.com/t/support-tap-interface-for-direct-container-access-incl-multi-host/17835

[inancgumus]

So, what's the solution, why this issue has been closed without a solution?

[AlmirKadric]

@inancgumus The issue was closed since it wasn't an issue in hyperkit. Hyperkit does support the network interfaces required. The issue is that docker for mac is using it wrong, thus the additional external links. To clarify though, docker for mac still hasn't to my knowledge fixed this. I've given up hope to be honest =_=

P.S. lots of info in this thread, including some hacky solutions: https://forums.docker.com/t/support-tap-interface-for-direct-container-access-incl-multi-host/17835/17

[rn]

Docker for Mac is not "using it wrong". It is using VPNKit to provide network connectivity, which does not create a host interface (and does not require to be run as root) and provides additional functionality like better VPN support and exposing container ports to localhost etc.

@inancgumus there are two comments above explaining why this was closed: #45 (comment) and #45 (comment)

[inancgumus]

@AlmirKadric @rneugeba Hmm, sad. Is someone here from Docker? I've read almost all of the solutions and the topics but no luck yet. Probably, I'm going to install an ubuntu inside os x through a virtual machine (which vaporizes the Docker's direct benefits).

It's been years and I still see this problems. I wonder where Docker spends its money. It forces us to use non-osx OSes. Of course, these claims can easily be defendable by Docker team, however, it doesn't make our developers' life easier, "as promised".

:(

[rn]

I work for Docker. As mentioned above, the OP was requesting a feature which is already is present, hence this issue was closed. What you seem to be looking for is a feature in Docker for Mac (which uses hyperkit) and should open an issue on https://github.com/docker/for-mac/issues

[inancgumus]

@rneugeba I don't need to, thx for the advice, because there are many issues about this problem already lingering with little hope, calling help from Docker team:

• network_mode: "host" probably not working as expected docker/for-mac#1031
• Access the Containers from Host like on Linux docker/for-mac#155
• net=host mode bug under mac, container with default bridge mode can not access host opening port  docker/for-mac#68
• ping inside container does not ping actual host ip but seems to ping localhost docker/for-mac#57

[AlmirKadric]

@rneugeba sorry for saying it was being used wrong, cant help but be snarky about it considering the timeline. To clarify it, docker for mac isn't using it wrong, its just isn't consistent across different platforms. On docker for windows and docker for linux you can route over the host virtual machine to get access to the guest containers. However in docker for mac this isn't the case due to the tap interface not being created and no means are provided to allow you to create it easily (there are some hackish attempts out there to get some form of routing working, some are listed inside the thread I mentioned).

But yes I agree, this isn't a problem with hyperkit and the issue belongs in the docker for mac issue tracker.

@inancgumus yes it is quite sad, I have given up hope on docker for mac fixing the issue any time soon. They seem swamped with other work, and this features/bug just doesn't seem to be priority. From what I have seen (or perhaps a gut feeling), there seems to be some conflict as to whether or not Docker should allow routing into the guest containers. It seems that on windows and linux it is more a side effect of the tools used and on mac providing this consistency would be going against some design choice that was made along the way. I do hope this is resolved soon as it is a very much needed feature for so many people, but I wouldn't hold my breath. Probably better to find a work around in the mean time.

[djs55]

For Docker feature requests, take a look at the roadmap repo: https://github.com/docker/roadmap/issues?q=is%3Aissue+is%3Aopen+network

[enzofrnt]

Hi,
Any possibility of fix ?

[gregnr]

If a tun (L3) interface works for your needs, you can use docker-mac-net-connect which creates a lightweight WireGuard tunnel under the hood between macOS and the Docker Linux VM (just a single Go binary on macOS side since WireGuard is built into the Linux kernel).

[enzofrnt]

If a tun (L3) interface works for your needs, you can use docker-mac-net-connect which creates a lightweight WireGuard tunnel under the hood between macOS and the Docker Linux VM (just a single Go binary on macOS side since WireGuard is built into the Linux kernel).

Wouldn’t that VPN be a bit slow?

[gregnr]

That's the first I've heard of it - have you tried it? VPNs will always have extra overhead, but in the case of WireGuard it's very small - I doubt it's anything you would notice.

The current state of Docker for mac requires a tunnel to accomplish L2/L3 networking (without hacks). WireGuard was chosen for that project because the protocol is built into the Linux kernel - closer to the metal / faster than other VPNs today. Also doesn't require a background process on the Linux VM since it is handled via the kernel module. Topology wise it's just a single hop going between macOS host and Linux VM (via VPNKit).

If you're having performance issues please submit an issue, I would love to investigate.

[enzofrnt]

@gregnr Alright, I'll give it a try.
However, I have a few questions. When you mention "hack", what exactly are you referring to? Is it possible to establish a direct connection between the LinuxKit interface and the Mac interface?
Also, can I use IPvLAN over this WireGuard setup to expose IPs on my local network?

Thanks you.

[gregnr]

These are the hacks I was referring to:
https://forums.docker.com/t/support-tap-interface-for-direct-container-access-incl-multi-host/17835

HyperKit fully supports bridge interfaces, but Docker for mac doesn't expose it. So the method above manually starts HyperKit with the modified config.

ipvlan in L2 mode would not work, since WireGuard operates over L3 (meaning DHCP, mDNS, etc won't work). ipvlan L3 should work but hasn't been tested. What is your use case for ipvlan?

[enzofrnt]

@gregnr
If my understanding is right.
With this hack, I'm able to bind the interface from macOS witch I can find here :

To the interface that are present in LinuxKit here :
Also, as I understand this command, give me access to the Linux kernel of Docker :
nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock

Right ?

Some HomeKit container doesn't like to be used with same IP and port forwarding. So I prefer ipvlan and that also easier to as one IP by service for me.

[enzofrnt]

And, by taking a look at it :
https://forums.docker.com/t/support-tap-interface-for-direct-container-access-incl-multi-host/17835

I'm not sure, I will be able to do it my self…