Revamp port forwarding to support UDP #2411

balajiv113 · 2024-06-10T08:29:37Z

Fixes #366
Discussion - #2402

Revamps port forwarding to use existing GRPC communication.

Advantages

No SSH daemon (See Notes below)
No Subprocess for SSH
Faster (See performance below)
Ease of support for newer protocol (We do support UDP as well with this implementation)

Todo

Performance
GRPC TCP - ~3.80 Gbits/sec
GRPC TCP Reverse - ~4.77 Gbits/sec
SSH TCP - ~3.38 Gbits/sec
SSH TCP Reverse - ~3.08 Gbits/sec

Notes

We will support only TCP and UDP via this approach. Unix and unixgram support socket to be added later as a separate PR
We support forwarding from host (Listen) -> guest (Dial). We don't support reverse forwarding host (Dial) -> guest (Listen) as of this PR. But can be extended and support it in a separate PR

alexandear · 2024-06-10T15:05:49Z

pkg/guestagent/api/guestservice.proto

+  string protocol = 1;
+  string ip = 2;
+  int32 port = 3;


Perhaps this is a breaking change. According to https://protobuf.dev/programming-guides/proto3/#assigning "This number cannot be changed once your message type is in use"

Better:

Suggested change

string protocol = 1;

string ip = 2;

int32 port = 3;

string ip = 1;

int32 port = 2;

string protocol = 3;

Just curious, from the doc its talking about the order number assigned to each field.

protocol, ip, port as well looks fine right ?? Do we have a say that int32 (port) cannot be at last ??

balajiv113 · 2024-06-13T05:19:07Z

Converting to draft as it needs some more changes related to handling of ports on host side

balajiv113 · 2024-06-27T09:03:25Z

The failing tests run fine locally. Looks like some UDP port is enabled in these github runner and that is causing issue

Edit: Found the issue, it was related to mutex. I missed to unlock mutex properly on error cases

balajiv113 · 2024-06-27T13:18:27Z

@lima-vm/maintainers
Do we need to back this feature on with a config in lima yaml ??

Just thinking because we are looking at this as a drop-in replacement. If we went with config we need to mark this experimental and later bring it to stable / default. Also it will pollute the yaml as well

One another idea is to back this with a ENV / start option

AkihiroSuda · 2024-06-27T13:22:05Z

ENV

👍

balajiv113 · 2024-06-27T15:09:56Z

Added a env and updated docs as well for this feature.

@lima-vm/maintainers
We are good to review this PR.

I have done decent amount of testing in macOS and a very basic test in linux. For performance tried with iperf3 for both tcp and udp.
Need some help to test in windows (Not sure if we even support port forwarding here, i think wsl takes care of it)

.github/workflows/test.yml

pkg/guestagent/api/guestservice.proto

pkg/hostagent/hostagent.go

website/content/en/docs/Config/Port/_index.md

AkihiroSuda · 2024-07-23T05:18:52Z

@lima-vm/maintainers Can we merge this?

moenodedev · 2024-08-08T19:26:06Z

@jandubois @afbjorklund Do you need more testers?

jandubois · 2024-08-09T00:02:36Z

I'm sorry, I won't have time to test/review this until September. 😞

suhailskhan · 2024-08-11T08:34:16Z

@balajiv113 I have been testing UDP port forwarding with some Docker Compose applications, and I may have found an issue.

After I first run docker compose up, any UDP ports that I choose to expose are reachable, as expected. However, I find that after I restart the application, those ports are not reachable anymore. This does not happen with TCP port forwarding. TCP ports that I expose in the container(s) are reachable even after restarting. The only way I was able to make the UDP ports for my Compose apps reachable again was to restart the Lima VM. However, the problem reappears when I restart the Compose app.

To test, I built Lima from your most recent commit in this PR at the time of writing this comment. I created a VM using the Docker template with the following command: limactl create --vm-type vz template://docker --name=default. Below is the docker-compose.yml for a simple Docker Compose application that demonstrates the issue:

services:
  iperf3-server:
    image: networkstatic/iperf3
    command: -s
    ports:
      - "5201:5201/udp"
      - "5201:5201/tcp"

To verify whether or not the iperf3-server container is listening on TCP port 5201 and UDP port 5201, I used sudo lsof -i TCP:5201 and sudo lsof -i UDP:5201, respectively. If it is listening on a port, the command should have output. If there is no output, then that means the service is not listening on the port. When I run the Compose app for the first time, I get output from both commands. After restarting the app, I only get output from the lsof command for the TCP port.

Although I was primarily using Docker Compose in my tests, it seems to affect Docker containers in general. If I start a container with a certain UDP port forwarded, then stop it, then that UDP port is not able to be exposed in subsequently created containers, until the Lima VM is restarted.

Would be curious to see if you or anyone else can reproduce this.

I am testing this on macOS Sonoma 14.5 on an Intel Mac. I am also able to test on a Sonoma machine with Apple silicon.

balajiv113 · 2024-08-12T06:18:11Z

@suhailskhan Thanks for trying it out.

Yes it is indeed a issue. It happens due to conn cache on guest side. On closing of listener its not cleared properly
Will fix and let you know

balajiv113 · 2024-08-12T11:22:34Z

@suhailskhan Fixed it. It should work now

suhailskhan · 2024-08-12T22:01:53Z

@balajiv113 Pulled, rebuilt, and tested. Works indeed!

AkihiroSuda · 2024-08-18T14:05:28Z

website/content/en/docs/config/Port/_index.md

+> **Warning**
+> This mode is experimental
+
+| ⚡ Requirement | Lima >= 0.23 |


AkihiroSuda · 2024-08-18T14:05:38Z

website/content/en/docs/config/Port/_index.md

+- Doesn't support UDP based port forwarding
+- Spans child process on host for running SSH master.
+
+### Using GRPC (Default)


Suggested change

### Using GRPC (Default)

### Using GRPC (Default since Lima v1.0)

AkihiroSuda · 2024-08-18T14:06:15Z

website/content/en/docs/config/Port/_index.md

+
+### Using SSH
+
+SSH based port forwarding is the default and current model that is supported in lima.


Suggested change

SSH based port forwarding is the default and current model that is supported in lima.

SSH based port forwarding is the default and current model that is supported in Lima prior to v1.0

AkihiroSuda · 2024-08-30T05:41:11Z

Lint is failing

Signed-off-by: Balaji Vijayakumar <kuttibalaji.v6@gmail.com>

balajiv113 · 2024-09-02T06:26:08Z

@AkihiroSuda Done. All Green

AkihiroSuda

Thanks

The ServicePort.Protocol is always uppercase, e.g. "TCP", but the api.IPPort.protocol is always lowercase, i.e. "tcp". Since UDP support was added in lima-vm#2411 the hostagent filters on the protocol values. Signed-off-by: Jan Dubois <jan.dubois@suse.com>

⚠️ **CAUTION: this is a major update, indicating a breaking change!** ⚠️ This MR contains the following updates: | Package | Update | Change | |---|---|---| | [lima-vm/lima](https://github.com/lima-vm/lima) | major | `v0.23.2` -> `v1.0.2` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>lima-vm/lima (lima-vm/lima)</summary> ### [`v1.0.2`](https://github.com/lima-vm/lima/releases/tag/v1.0.2) [Compare Source](lima-vm/lima@v1.0.1...v1.0.2) #### Changes - DNS: - Fixed the host resolver regression in v1.0.0 [#2939](lima-vm/lima#2939) ([#2964](lima-vm/lima#2964)) - `limactl create`: - Fixed races during parallel downloads ([#2903](lima-vm/lima#2903), thanks to [@nirs](https://github.com/nirs)) - Optimized qcow2-to-raw conversion for vz mode ([#2933](lima-vm/lima#2933), thanks to [@nirs](https://github.com/nirs)) - `limactl start-at-login`: - Fixed the support for Linux hosts (systemd) ([#2943](lima-vm/lima#2943), thanks to [@kachick](https://github.com/kachick)) - nerdctl: - Updated to [v2.0.1](https://github.com/containerd/nerdctl/releases/tag/v2.0.1) ([#2966](lima-vm/lima#2966)) - Templates: - Updated to the latest revisions ([#2936](lima-vm/lima#2936) [#2953](lima-vm/lima#2953), thanks to [@tcooper](https://github.com/tcooper)) - Web site: - Added an example of running Lima on GitHub Actions to run commands on non-Ubuntu ([#2954](lima-vm/lima#2954)): https://lima-vm.io/docs/examples/gha/ - Project: - Invite Nir Soffer ([@nirs](https://github.com/nirs)) as a Reviewer ([#2916](lima-vm/lima#2916), thanks to [@jandubois](https://github.com/jandubois)) Full changes: https://github.com/lima-vm/lima/milestone/51?closed=1 Thanks to [@SpiffyEight77](https://github.com/SpiffyEight77) [@alexandear](https://github.com/alexandear) [@jandubois](https://github.com/jandubois) [@kachick](https://github.com/kachick) [@nirs](https://github.com/nirs) [@norio-nomura](https://github.com/norio-nomura) [@tamird](https://github.com/tamird) [@tcooper](https://github.com/tcooper) #### Usage ```console [macOS]$ limactl create [macOS]$ limactl start ... INFO[0029] READY. Run `lima` to open the shell. [macOS]$ lima uname Linux ``` *** The binaries were built automatically on GitHub Actions. The build log is available for 90 days: https://github.com/lima-vm/lima/actions/runs/12134682585 The sha256sum of the SHA256SUMS file itself is `02ef78494c498ca4180915ba78d5e2fc471ed401f63dfb2b5864c3711f3c0fb2` . *** Release manager: [@AkihiroSuda](https://github.com/AkihiroSuda) ### [`v1.0.1`](https://github.com/lima-vm/lima/releases/tag/v1.0.1) [Compare Source](lima-vm/lima@v1.0.0...v1.0.1) Reverted the default port forwarder from gRPC to SSH for the stability reason ([#2864](lima-vm/lima#2864)). This reversion fixes several regressions related to `docker run -p` in Lima v1.0.0 ([#2859](lima-vm/lima#2859)). Although the gRPC forwarder is faster and has an advanced feature (UDP support), it turned out to be still immature. Set `LIMA_SSH_PORT_FORWARDER=false` to opt-in to the gRPC forwarder. See <https://lima-vm.io/docs/config/port/>. Full changes: https://github.com/lima-vm/lima/milestone/50?closed=1 Thanks to [@alexandear](https://github.com/alexandear) [@jandubois](https://github.com/jandubois) [@norio-nomura](https://github.com/norio-nomura) #### Usage ```console [macOS]$ limactl create [macOS]$ limactl start ... INFO[0029] READY. Run `lima` to open the shell. [macOS]$ lima uname Linux ``` *** The binaries were built automatically on GitHub Actions. The build log is available for 90 days: https://github.com/lima-vm/lima/actions/runs/11735352652 The sha256sum of the SHA256SUMS file itself is `f5c12d003e25dc46291803a8acae9e9d325a45eca0c1f9f40bd6852ec8ed9be1` . *** Release manager: [@AkihiroSuda](https://github.com/AkihiroSuda) ### [`v1.0.0`](https://github.com/lima-vm/lima/releases/tag/v1.0.0) [Compare Source](lima-vm/lima@v0.23.2...v1.0.0) With the support from 110+ contributors in 3+ years, the Lima project has finally reached v1.0. 🎉 This release introduces several breaking changes, such as switching the default machine driver from QEMU to VZ for better filesystem performance. The `limactl` CLI is designed to print hints when the user hits those breaking changes. e.g., `limactl create template://experimental/vz` now fails with a hint that suggests using `limactl create --vm-type=vz template://default` instead. 🔴 = Major breaking changes 🟡 = Minor breaking changes - VZ: - Graduate VZ machine driver from experimental ([#2758](lima-vm/lima#2758)) - 🔴 Use VZ by default for new instances on macOS >= 13.5 ([#1951](lima-vm/lima#1951)) - Support nested virtualization on M3 ([#2530](lima-vm/lima#2530), thanks to [@abiosoft](https://github.com/abiosoft)) - Optimize qcow2-to-raw image conversion (lima-vm/go-qcow2reader@v0.1.2...v0.4.0 , thanks to [@nirs](https://github.com/nirs)) - Support specifying a custom kernel ([#2562](lima-vm/lima#2562), thanks to [@norio-nomura](https://github.com/norio-nomura)) - QEMU: - Graduate 9p mount driver from experimental ([#2758](lima-vm/lima#2758)) - 🔴 Use 9p by default for most templates ([#1953](lima-vm/lima#1953), [#2822](lima-vm/lima#2822)) - riscv64: switch from u-boot to EDK2 ([#2592](lima-vm/lima#2592)) - Network: - Graduate user-v2 network driver from experimental ([#2758](lima-vm/lima#2758)) - Support UDP port forwarding ([#2411](lima-vm/lima#2411), thanks to [@balajiv113](https://github.com/balajiv113)) - 🔴 Strictly require `socket_vmnet` binary to be owned by root ([#2734](lima-vm/lima#2734)) - SSH: - 🟡 Disable `ssh.loadDotSSHPubKeys` by default ([#2706](lima-vm/lima#2706)) - YAML: - Support generating jsonschema ([#2306](lima-vm/lima#2306), thanks to [@afbjorklund](https://github.com/afbjorklund)) - Support specifying `param` for provisioning scripts ([#2570](lima-vm/lima#2570), thanks to [@jandubois](https://github.com/jandubois)) - Support specifying `minimumLimaVersion` and `vmOpts.qemu.minimumVersion` ([#2659](lima-vm/lima#2659), thanks to [@jandubois](https://github.com/jandubois)) - Support template expansion in mounts ([#2588](lima-vm/lima#2588), thanks to [@norio-nomura](https://github.com/norio-nomura)) - `limactl` CLI: - Add `limactl tunnel` command so as to allow the host to join the guest network ([#2710](lima-vm/lima#2710)) - Add `--log-format=json` ([#2584](lima-vm/lima#2584), thanks to [@nirs](https://github.com/nirs)) - `limactl prune`: Add `--keep-referred` ([#2569](lima-vm/lima#2569), thanks to [@norio-nomura](https://github.com/norio-nomura)) - nerdctl: - Updated to [v2.0.0](https://github.com/containerd/nerdctl/releases/tag/v2.0.0) ([#2178](lima-vm/lima#2178)) - rootless: allocate 1G subuids from 524288 (0x80000) for new users ([#2725](lima-vm/lima#2725)) - Templates: - 🔴 `experimental/vz`: Merged into the `default` template ([#2730](lima-vm/lima#2730), [#2736](lima-vm/lima#2736)) - 🟡 `experimental/{riscv64, armv7l}`: Merged into the `default` template ([#2730](lima-vm/lima#2730), [#2736](lima-vm/lima#2736)) - 🔴 `vmnet`: Removed in favor of `limactl create --network=lima:shared template://default` ([#2736](lima-vm/lima#2736)) - 🟡 `experimental/net-user-v2`: Removed in favor of `limactl create --network=lima:user-v2 template://default` ([#2736](lima-vm/lima#2736)) - 🔴 `experimental/9p`: Removed in favor of `limactl create --mount-type=9p template://default` ([#2736](lima-vm/lima#2736)) - 🟡 `experimental/virtiofs-linux`: Removed in favor of `limactl create --mount-type=virtiofs template://default` ([#2736](lima-vm/lima#2736)) - 🔴 `alpine`: Renamed to `alpine-iso` ([#2704](lima-vm/lima#2704)) - 🔴 `alpine-image`: Renamed to `alpine` ([#2704](lima-vm/lima#2704)) - `archlinux`: Demoted from Tier 1 to Tier 2 ([#2717](lima-vm/lima#2717), [#2823](lima-vm/lima#2823)) - `default`, `ubuntu`, ...: Updated to Ubuntu 24.10. The older versions are available as `ubuntu-20.04`, `ubuntu-22.04`, and `ubuntu-24.04` ([#2755](lima-vm/lima#2755), [#2795](lima-vm/lima#2795)) - `fedora`: Updated to Fedora 41 ([#2821](lima-vm/lima#2821), [#2822](lima-vm/lima#2822), thanks to [@subpop](https://github.com/subpop)) - `opensuse`: Renamed to `opensuse-leap`. Still aliased as `opensuse` ([#2612](lima-vm/lima#2612), thanks to [@afbjorklund](https://github.com/afbjorklund)) - `experimental/opensuse-tumbleweed`: Support aarch64 ([#2613](lima-vm/lima#2613), thanks to [@afbjorklund](https://github.com/afbjorklund)) - `hack/update-template.sh` is added for automating updates ([#1347](lima-vm/lima#1347), thanks to [@norio-nomura](https://github.com/norio-nomura)) - Project: - Invite Norio Nomura ([@norio-nomura](https://github.com/norio-nomura)) as a Reviewer ([#2567](lima-vm/lima#2567)) Full changes: https://github.com/lima-vm/lima/milestone/47?closed=1 Thanks to [@AdamKorcz](https://github.com/AdamKorcz) [@Mr-Sunglasses](https://github.com/Mr-Sunglasses) [@SmartManoj](https://github.com/SmartManoj) [@YorikSar](https://github.com/YorikSar) [@abiosoft](https://github.com/abiosoft) [@afbjorklund](https://github.com/afbjorklund) [@alexandear](https://github.com/alexandear) [@balajiv113](https://github.com/balajiv113) [@hasan4791](https://github.com/hasan4791) [@jandubois](https://github.com/jandubois) [@nirs](https://github.com/nirs) [@norio-nomura](https://github.com/norio-nomura) [@pvdvreede](https://github.com/pvdvreede) [@subpop](https://github.com/subpop) [@tsukasaI](https://github.com/tsukasaI) #### Usage ```console [macOS]$ limactl create [macOS]$ limactl start ... INFO[0029] READY. Run `lima` to open the shell. [macOS]$ lima uname Linux ``` *** The binaries were built automatically on GitHub Actions. The build log is available for 90 days: https://github.com/lima-vm/lima/actions/runs/11695321667 The sha256sum of the SHA256SUMS file itself is `4bd200a163111fe78c6f3e6de405113d416053802fe1507597f9a42f89a98c90` . *** Release manager: [@AkihiroSuda](https://github.com/AkihiroSuda) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

balajiv113 force-pushed the revamp-portfwd branch 3 times, most recently from 5286d54 to 2871242 Compare June 10, 2024 13:56

alexandear reviewed Jun 10, 2024

View reviewed changes

balajiv113 force-pushed the revamp-portfwd branch from 2871242 to fdb01e8 Compare June 11, 2024 04:54

balajiv113 marked this pull request as draft June 13, 2024 05:18

balajiv113 force-pushed the revamp-portfwd branch 7 times, most recently from 7185139 to 50ed124 Compare June 26, 2024 14:22

balajiv113 force-pushed the revamp-portfwd branch 3 times, most recently from 707cc75 to 1e49f9c Compare June 27, 2024 13:01

balajiv113 marked this pull request as ready for review June 27, 2024 13:18

balajiv113 force-pushed the revamp-portfwd branch from 1e49f9c to 8e761f3 Compare June 27, 2024 15:06

AkihiroSuda added this to the v0.23.0 (tentative) milestone Jun 30, 2024