`limactl start` fails to bind to SSH port (ssh: connect to host 127.0.0.1 port 60022: Connection refused)

[agatti]

On my machine/configuration SSH seems to not be able to bind to port 60022 and therefore things fail to start. I've checked whether I already have anything else listening on that port on the localhost interface but the port is indeed available.

The image to boot is the default one, and I'm running macOS 11.3.1. QEMU comes from MacPorts and entitlements are correctly installed. Debug log follows:

agatti@tardigrade ~ % limactl --debug start
DEBU[0000] interpreting argument "default" as an instance name "default"
INFO[0000] Using the existing instance "default"
INFO[0000] Starting QEMU
DEBU[0000] qCmd.Args: [/opt/local/bin/qemu-system-x86_64 -cpu Haswell-v4 -machine q35,accel=hvf -smp 4,sockets=1,cores=4,threads=1 -m 4096 -drive if=pflash,format=raw,readonly,file=/opt/local/share/qemu/edk2-x86_64-code.fd -drive file=/Users/agatti/.lima/default/diffdisk,if=virtio -boot c -cdrom /Users/agatti/.lima/default/cidata.iso -net nic,model=virtio -net user,net=192.168.5.0/24,hostfwd=tcp:127.0.0.1:60022-:22 -device virtio-rng-pci -device virtio-vga -device virtio-keyboard-pci -device virtio-mouse-pci -parallel none -name lima-default -pidfile /Users/agatti/.lima/default/qemu-pid]
INFO[0000] SSH: 127.0.0.1:60022
INFO[0000] Waiting for the essential requirement 1 of 3: "ssh"
DEBU[0000] executing script "ssh"
DEBU[0000] executing ssh for script "ssh": /usr/bin/ssh [ssh -o ControlMaster=auto -o ControlPath=/Users/agatti/.lima/default/ssh.sock -o ControlPersist=5m -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -p 60022 127.0.0.1 -- /bin/bash]
DEBU[0000] stdout="", stderr="ssh: connect to host 127.0.0.1 port 60022: Connection refused\r\n", err=failed to execute script "ssh": stdout="", stderr="ssh: connect to host 127.0.0.1 port 60022: Connection refused\r\n": exit status 255

[then keeps looping on waiting for ssh]

Is there anything else I should check to get this to at least start?

[AkihiroSuda]

Do you seen any error on the QEMU VGA screen or on the serial console (Ctrl-Alt-2)?

[vidner]

I have the same issue, i tried to manually run the ssh command with verbose option and its says the ControlPath does not exist.

ssh -vvv -o ControlMaster=auto -o ControlPath=/Users/vidner/.lima/default/ssh.sock -o ControlPersist=5m -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -p 60022 127.0.0.1 -- /bin/bash
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/vidner/.ssh/config
debug1: /Users/vidner/.ssh/config line 21: Skipping Host block because of negated match for 127.0.0.1
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: /etc/ssh/ssh_config line 52: Applying options for *
debug2: resolve_canonicalize: hostname 127.0.0.1 is address
debug1: auto-mux: Trying existing master
debug1: Control socket "/Users/vidner/.lima/default/ssh.sock" does not exist
debug2: ssh_connect_direct
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 60022.
debug1: connect to address 127.0.0.1 port 60022: Connection refused
ssh: connect to host 127.0.0.1 port 60022: Connection refused

[agatti]

@AkihiroSuda unfortunately I don't see any errors in the QEMU window or in the serial console...

[AkihiroSuda]

MacPorts version of QEMU might be compiled without support for libslirp (unconfirmed): https://github.com/macports/macports-ports/blob/master/emulators/qemu/Portfile

Could you try homebrew package of QEMU?

[agatti]

Tried installing homebrew's version, nothing really changed. Short of rebuilding QEMU myself, is there anything else I can try?

[idrissneumann]

Same errors here (installed with homebrew):

ineumann ~ $ limactl --debug start
DEBU[0000] interpreting argument "default" as an instance name "default" 
INFO[0000] Using the existing instance "default"        
INFO[0000] Downloading "https://github.com/containerd/nerdctl/releases/download/v0.9.0/nerdctl-full-0.9.0-linux-arm64.tar.gz" 
DEBU[0000] file "/var/folders/gj/052mqkk12r55d2xv7lymqf640000gn/T/lima-download-nerdctl975222294/nerdctl-full-0.9.0-linux-arm64.tar.gz" is cached as "/Users/ineumann/Library/Caches/lima/download/by-url-sha256/08927199d45dbbfa40ed207c68567a6913b7b347c62c033a5c501220bd8958b2/data" 
INFO[0000] Using cache "/Users/ineumann/Library/Caches/lima/download/by-url-sha256/08927199d45dbbfa40ed207c68567a6913b7b347c62c033a5c501220bd8958b2/data" 
INFO[0000] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/ineumann/.lima/default/serial.log") 
DEBU[0000] [hostagent] qCmd.Args: [/opt/homebrew/bin/qemu-system-aarch64 -cpu cortex-a72 -machine virt,accel=hvf,highmem=off -smp 4,sockets=1,cores=4,threads=1 -m 4096 -drive if=pflash,format=raw,readonly,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/ineumann/.lima/default/diffdisk,if=virtio -cdrom /Users/ineumann/.lima/default/cidata.iso -net nic,model=virtio -net user,net=192.168.5.0/24,hostfwd=tcp:127.0.0.1:60022-:22 -device virtio-rng-pci -display none -vga none -device ramfb -device usb-ehci -device usb-kbd -device usb-mouse -parallel none -chardev socket,id=char-serial,path=/Users/ineumann/.lima/default/serial.sock,server,nowait,logfile=/Users/ineumann/.lima/default/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/ineumann/.lima/default/qmp.sock,server,nowait -qmp chardev:char-qmp -name lima-default -pidfile /Users/ineumann/.lima/default/qemu.pid] 
DEBU[0000] received an event                             event="{2021-06-23 16:37:15.771105 +0100 CET {false false false [] 60022}}"
INFO[0000] SSH Local Port: 60022                        
INFO[0000] [hostagent] Waiting for the essential requirement 1 of 4: "ssh" 
DEBU[0000] [hostagent] executing script "ssh"           
DEBU[0000] [hostagent] stdout="", stderr="ssh: connect to host 127.0.0.1 port 60022: Connection refused\r\n", err=failed to execute script "ssh": stdout="", stderr="ssh: connect to host 127.0.0.1 port 60022: Connection refused\r\n": exit status 255 
INFO[0000] [hostagent] QEMU has exited                  
DEBU[0000] received an event                             event="{2021-06-23 16:37:15.820858 +0100 CET {false false true [] 0}}"
FATA[0000] exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/ineumann/.lima/default/ha.stderr.log") 
ineumann ~ $ cat /Users/ineumann/.lima/default/ha.stderr.log
{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/Users/ineumann/.lima/default/serial.log\")","time":"2021-06-23T16:37:15+01:00"}
{"level":"debug","msg":"qCmd.Args: [/opt/homebrew/bin/qemu-system-aarch64 -cpu cortex-a72 -machine virt,accel=hvf,highmem=off -smp 4,sockets=1,cores=4,threads=1 -m 4096 -drive if=pflash,format=raw,readonly,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/ineumann/.lima/default/diffdisk,if=virtio -cdrom /Users/ineumann/.lima/default/cidata.iso -net nic,model=virtio -net user,net=192.168.5.0/24,hostfwd=tcp:127.0.0.1:60022-:22 -device virtio-rng-pci -display none -vga none -device ramfb -device usb-ehci -device usb-kbd -device usb-mouse -parallel none -chardev socket,id=char-serial,path=/Users/ineumann/.lima/default/serial.sock,server,nowait,logfile=/Users/ineumann/.lima/default/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/ineumann/.lima/default/qmp.sock,server,nowait -qmp chardev:char-qmp -name lima-default -pidfile /Users/ineumann/.lima/default/qemu.pid]","time":"2021-06-23T16:37:15+01:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: \"ssh\"","time":"2021-06-23T16:37:15+01:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2021-06-23T16:37:15+01:00"}
{"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 60022: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 60022: Connection refused\\r\\n\": exit status 255","time":"2021-06-23T16:37:15+01:00"}
{"error":"exit status 1","level":"info","msg":"QEMU has exited","time":"2021-06-23T16:37:15+01:00"}
exit status 1

And the homebrew dependancies:

(QEMU 6.0.0 and libslirp 4.6.1)

And precision about the mac:

(MBP 13 Ship M1)

[AkihiroSuda]

Qemu still requires a patch for ARM Mac

https://gist.github.com/nrjdalal/e70249bb5d2e9d844cc203fd11f74c55

[agatti]

@AkihiroSuda in my case, I'm using an Intel-based macbook so it can't be that?

[AkihiroSuda]

@agatti please show *.log files under ~/.lima/default

[idrissneumann]

@AkihiroSuda Thanks it works well with the patch of QEMU.

Seems pretty promising! Thanks again.

[unixorn]

I'm seeing this same issue. I'm on an Intel MacBook Pro, macOS 11.4 (20F71).

❯ limactl --debug start
DEBU[0000] interpreting argument "default" as an instance name "default"
? Creating an instance "default" Proceed with the default configuration
INFO[0001] Downloading "https://github.com/containerd/nerdctl/releases/download/v0.9.0/nerdctl-full-0.9.0-linux-amd64.tar.gz"
DEBU[0001] file "/var/folders/dw/55kvh5_57x340vm7251kgvmm0000gn/T/lima-download-nerdctl004299890/nerdctl-full-0.9.0-linux-amd64.tar.gz" is cached as "/Users/jpb/Library/Caches/lima/download/by-url-sha256/4c048fab91ba88527ad58a6b4dc4c3bfa35e828b8147565456b76d362ec2e6cb/data"
INFO[0001] Using cache "/Users/jpb/Library/Caches/lima/download/by-url-sha256/4c048fab91ba88527ad58a6b4dc4c3bfa35e828b8147565456b76d362ec2e6cb/data"
INFO[0002] Attempting to download the image from "~/Downloads/hirsute-server-cloudimg-amd64.img"
INFO[0002] Attempting to download the image from "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-amd64.img"
DEBU[0002] file "/Users/jpb/.lima/default/basedisk" is cached as "/Users/jpb/Library/Caches/lima/download/by-url-sha256/e1fed960ebd29619676c7ab7535bc83f7fb2ad71739edb6fde4e17bce0b61a47/data"
INFO[0003] Using cache "/Users/jpb/Library/Caches/lima/download/by-url-sha256/e1fed960ebd29619676c7ab7535bc83f7fb2ad71739edb6fde4e17bce0b61a47/data"
INFO[0003] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/jpb/.lima/default/serial.log")
DEBU[0003] [hostagent] qCmd.Args: [/usr/local/bin/qemu-system-x86_64 -cpu Haswell-v4 -machine q35,accel=hvf -smp 4,sockets=1,cores=4,threads=1 -m 4096 -drive if=pflash,format=raw,readonly,file=/usr/local/share/qemu/edk2-x86_64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/jpb/.lima/default/diffdisk,if=virtio -cdrom /Users/jpb/.lima/default/cidata.iso -net nic,model=virtio -net user,net=192.168.5.0/24,hostfwd=tcp:127.0.0.1:60022-:22 -device virtio-rng-pci -display none -device virtio-vga -device virtio-keyboard-pci -device virtio-mouse-pci -parallel none -chardev socket,id=char-serial,path=/Users/jpb/.lima/default/serial.sock,server,nowait,logfile=/Users/jpb/.lima/default/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/jpb/.lima/default/qmp.sock,server,nowait -qmp chardev:char-qmp -name lima-default -pidfile /Users/jpb/.lima/default/qemu.pid]
DEBU[0003] received an event                             event="{2021-06-23 22:26:47.702068 -0600 MDT {false false false [] 60022}}"
INFO[0003] SSH Local Port: 60022
INFO[0003] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
DEBU[0003] [hostagent] executing script "ssh"
DEBU[0003] [hostagent] stdout="", stderr="ssh: connect to host 127.0.0.1 port 60022: Connection refused\r\n", err=failed to execute script "ssh": stdout="", stderr="ssh: connect to host 127.0.0.1 port 60022: Connection refused\r\n": exit status 255
INFO[0013] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
DEBU[0013] [hostagent] executing script "ssh"
DEBU[0019] [hostagent] stdout="", stderr="kex_exchange_identification: read: Connection reset by peer\r\n", err=failed to execute script "ssh": stdout="", stderr="kex_exchange_identification: read: Connection reset by peer\r\n": exit status 255
INFO[0029] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
DEBU[0029] [hostagent] executing script "ssh"
DEBU[0029] [hostagent] stdout="", stderr="jpb@127.0.0.1: Permission denied (publickey).\r\n", err=failed to execute script "ssh": stdout="", stderr="jpb@127.0.0.1: Permission denied (publickey).\r\n": exit status 255
INFO[0039] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"

qemu info:

❯ brew info qemu
qemu: stable 6.0.0 (bottled), HEAD
Emulator for x86 and PowerPC
https://www.qemu.org/
/usr/local/Cellar/qemu/6.0.0 (161 files, 569.8MB) *
  Poured from bottle on 2021-06-23 at 20:24:00
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/qemu.rb
License: GPL-2.0-only
==> Dependencies
Build: libtool ✔, meson ✔, ninja ✔, pkg-config ✔
Required: glib ✔, gnutls ✔, jpeg ✔, libpng ✔, libslirp ✔, libssh ✔, libusb ✔, lzo ✔, ncurses ✔, nettle ✔, pixman ✔, snappy ✔, vde ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 16,161 (30 days), 40,020 (90 days), 126,729 (365 days)
install-on-request: 16,057 (30 days), 39,861 (90 days), 125,610 (365 days)
build-error: 0 (30 days)
❯

[AkihiroSuda]

@unixorn #17 (comment)

[elixir67]

@agatti and @unixorn I have the same issue with intel based macbook pro, macos 11.5.2, did you find any solution? Thanks in advance!

I finally find it need sign the binary manually similar as https://github.com/lima-vm/lima#qemu-crashes-with-hv_error, while I didn't see the HV_ERROR in the logs.
Then it turns to another issue, similar as https://github.com/lima-vm/lima#stuck-on-waiting-for-the-essential-requirement-1-of-x-ssh, while my libslirp is already v4.6.1. Any idea on that?