Multipass can't get instances' IPs on macOS Sequoia

[naynayu]

Describe the bug
I have been trying to create instances for a few weeks but it always times out. I even uninstalled and reinstalled, same issue. I have attached logs, hopefully, someone is able to make sense of it.

To Reproduce
How, and what happened?

1. multipass launch
2. Process starts and never completes

Expected behavior
What did you expect to happen?

I expect that if launch a VM it should start and launch a VM both on the terminal and open a shell on the GUI

Logs
Please provide logs from the daemon, see accessing logs on where to find them on your platform.

The logs attached show what happens when I try to spin up an instance on both CLI and GUI at the same time
Multipass Logs 4_9.txt

Additional info

• OS: macOS Sequoia Beta 15.0 (24A5331b)
• multipass version
  multipass 1.14.0+mac
  multipassd 1.14.0+mac
• multipass info
  Name: adapted-bunny
  State: Unknown
  Snapshots: 0
  IPv4: --
  Release: --
  Image hash: f4b47dd0a608 (Ubuntu 20.04 LTS)
  CPU(s): --
  Load: --
  Disk usage: --
  Memory usage: --
  Mounts: --

Name: sporting-salmon
State: Unknown
Snapshots: 0
IPv4: --
Release: --
Image hash: 5ecac6447be6 (Ubuntu 24.04 LTS)
CPU(s): --
Load: --
Disk usage: --
Memory usage: --
Mounts: --

Name: wired-vole
State: Unknown
Snapshots: 0
IPv4: --
Release: --
Image hash: 5ecac6447be6 (Ubuntu 24.04 LTS)
CPU(s): --
Load: --
Disk usage: --
Memory usage: --
Mounts: --

• multipass get local.driver
  qemu
  Additional context
  Add any other context about the problem here.

[Andrey-DevOps]

The same issue

[ricab]

Hi @naynayu, do you have your firewall turned on? Does the situation improve if you disable it?

BTW, you appear to have attached GUI logs. We'd need the Multipass daemon's logs instead.

[naynayu]

Hi @ricab Thanks for your response.

No. Firewall isn't turned on. I cant seem to find the multipass daemon logs even after logging in as root.

I was able to filter the multipass logs from the Console. Will that do?

[ricab]

Hi @naynayu, does ls /Library/Logs/Multipass/multipassd.log not find anything? But yes, Multipass logs from the console should be the same thing.

Be sure to have a look at our troubleshooting doc. We are currently working to update it, but it has some important info as it is.

[naynayu]

@ricab So ls /Library/Logs/Multipass/multipassd.log does not produce anything. But when I did cat /Library/Logs/Multipass/multipassd.log, it worked. Here are the logs:

Multipassd.txt

[naynayu]

Adding firewall outputs as well:

[naynayu]

@ricab Hi, I have tried all the solutions on the troubleshooting doc, no luck. 😭

[ricab]

Hi @nayneenarh, I just had a look at the logs you sent and I see two cases of instances starting up and having their network interfaces up, but Multipass not being able to SSH in. Most likely, the instance isn't getting an IP or Multipass can't find the right one. We are trying to update our troubleshooting but in the meantime please have a look at this recipe. You should be in either case 2 or 3. Let us know how you fare with that.

[IronCore864]

It was working in previous versions but stopped working on macOS Sequoia Beta (and on the official release today, too). Probably something related to the new OS version.

[zamb0]

Hey @nayneenarh, same issue 🥲

[ricab]

Hi all, it looks like we might have a problem in Sequoia, which we are investigating. Sorry for your troubles.

[ricab]

FYI, we have found the issue. Now we need to find a solution.

[naynayu]

FYI, we have found the issue. Now we need to find a solution.

Oh great! Looking forward to the solution. If it's something we can also help with, please feel free to let us know.

[vsarunas]

Now that the RC has been released; it appears that the VM is actually up; and IP address is assigned, just Multipass is not able to get that. For example shutdown causes logs like:

Could not broadcast shutdown message in VM: ssh connection failed: 'Timeout connecting to 192.168.67.6'

While new VMs starting have:

[trace] [immune-flathead] Could not determine IP address within 1000ms

The VM is actually up and running as shown by arp output; a little script to parse the IPs:

#!/bin/bash
multipass list | grep -e "Unknown" -e "Starting" | awk '{print $1}' | while read -r vm; do
    mac=$(ps -ef | grep "qemu-system" | grep "$vm" | grep -o 'mac=[0-9a-fA-F:]*' | awk -F= '{print $2}')
    [ -z "$mac" ] && mac="unknown"
    # convert :00: to :0: or :0a: to :a: several times
    for i in {1..6}; do
        mac=$(echo "$mac" | sed -E 's/:0([0-9a-fA-F]):/:\1:/g; s/:00:/:0:/g')
    done
    
    ip=$(arp -an | grep "$mac" | awk -F'[()]' '{print $2}')  
    printf "VM: %-25s  IP: %-15s MAC Address: %-17s\n" "$vm" "${ip:-unknown}" "$mac"
done

$ multipass list
Name                    State             IPv4             Image
accepting-planthopper   Unknown           --               Ubuntu 24.04 LTS
immune-flathead         Unknown           --               Ubuntu 24.04 LTS
linux-dev               Unknown           --               Ubuntu 22.04 LTS

$ bash multipass_ip.sh 
VM: accepting-planthopper      IP: 192.168.67.9    MAC Address: 52:54:0:1e:18:4e 
VM: immune-flathead            IP: 192.168.67.8    MAC Address: 52:54:0:d:4a:22  
VM: linux-dev                  IP: 192.168.67.7    MAC Address: 52:54:0:14:e0:76 

I'm able to create a new VM and login to it using root's key:

ssh -i /var/root/Library/Application\ Support/multipassd/ssh-keys/id_rsa ubuntu@192.168.67.9

[vsarunas]

It appears that the format of /var/db/dhcpd_leases has changed or is corrupted on new version; previously it had 1,mac_address, now something is very long and doesn't respond to IP instance actually gets assigned:

more /var/db/dhcpd_leases
{
        name=accepting-planthopper
        ip_address=192.168.67.4
        hw_address=ff,f1:f5:dd:7f:0:2:0:0:ab:11:32:ca:56:11:8a:74:67:42
        identifier=ff,f1:f5:dd:7f:0:2:0:0:ab:11:32:ca:56:11:8a:74:67:42
        lease=0x66e8a656
}
{
        name=immune-flathead
        ip_address=192.168.67.3
        hw_address=ff,f1:f5:dd:7f:0:2:0:0:ab:11:b4:f8:94:34:d8:4b:79:9c
        identifier=ff,f1:f5:dd:7f:0:2:0:0:ab:11:b4:f8:94:34:d8:4b:79:9c
        lease=0x66e8a642
}

[georgeliao]

@vsarunas
Thanks for sharing your findings. Meanwhile, we also found the exact same thing and we are working on a more robust solution that can retrieve the ip address from the /var/db/dhcpd_leases file.

[asemarian]

I'm having the same problem today after upgrading to macOS Sequoia last night.

[geozeke]

I'll add, me too (upgraded to Sequoia yesterday), but this information may help: If I launch an instance using a bridged network (--bridged) it still hangs on "Starting..." and eventually times out. However it pops up, by name, on my network and gets a valid IP address according to my router portal.

[johalun]

I'll add, me too (upgraded to Sequoia yesterday), but this information may help: If I launch an instance using a bridged network (--bridged) it still hangs on "Starting..." and eventually times out. However it pops up, by name, on my network and gets a valid IP address according to my router portal.

Confirmed this works for a new VM using the launch command, but how would do get this to work for an existing VM?

Edit: Got it working by running multipass set local.$VM.bridged=true. Then, turns out my work VM's disk got corrupted by Multipass not working correctly (I guess from forced shutdowns?). After a boot into the serial console to fix the drive with fsck, I can now access my VM again using this method.

[geozeke]

Thanks. This is curious. Even though my bridged network attempt still results in the VM's IP popping up on my network, the VM itself never starts. Remains stuck on Starting..., then times out.

It looks like the team is zeroing in on a fix. I'll wait for bit. 😉

[freef4ll]

@johalun, see this comment for finding IPs of existing VMs.

[johalun]

@freef4ll Thanks, but the problem was never finding the IP. I did that using nmap to scan for open ssh ports on the VM subnet. The problem was that the disk got corrupted by Multipass so the VM was stuck in initramfs, requiring manual fsck.

[kacperk]

It seems that multipass is missing system permissions to connect to local networks (I have a pop-up to allow permissions to connect to local network for many apps since upgrade to Sequoia, but not from multipass). I can connect to vm via ssh, I can ping it from terminal, but multipass itself can't access it.

[ricab]

Hi @ralgozino, I don't think this is related to new changes. Would you mind opening a new issue and filling in the form with your info? Thanks.

[ralgozino]

FWIW, after deleting with --purge the primary instance and recreating it, the home folder got mounted again.

[rachadaccoumeh]

Hi, I have the same problem here the new test package worked for me after deleting the whole multipass with all previous instances but each instance only works normally for me the first time after creating the instance otherwise I'm not able to connect to it from terminal or GUI some time I'm getting started status but unable to connect to it some time I got unknown status ( i noticed that I'm able to connect to it via ssh) but I'm unable to access the web server via same IP used for ssh.

[levkropp]

Hi @rachadaccoumeh ,

Since this issue is closed, could you please open a new issue and fill out the issue template? This will help us get all the necessary information to troubleshoot effectively. Thank you!

[vsarunas]

@ricab - The source code also dropped for 15.0 - apple-oss-distributions/bootp@9ccd9f1#diff-eacf6b6816280114b896232f1d1460a593e617ebe319a6fafceb3a3574552203R62-R64 ; from a quick look don't see what could have changed the file format.

[ricab]

Interesting @vsarunas, thanks for the news! We've had one report last week of someone who was on 15 but did not see the change in dhcpd_leases, so I wonder if there is some other condition necessary to trigger the change/bug.

In any case, reading from that file was sometimes unreliable (old IPs in the mix and such). So we'll try the ARP approach now and see how it fares for people.

As an update: the release packages have been mostly ready for a few days now, but we are waiting for them to get signed internally. Unfortunately, there are ongoing issues with the hardware, so it is taking longer than usual. You can find (unsigned) RCs with a number of other bug fixes here.

[vsarunas]

In case there are issues to install the RC1 and verification warnings:

Apple could not verify “multipass-1.14.1-rc1+mac.14+gf2381bfe9.mac-Darwin.pkg” is free of malware that may harm your Mac or compromise your privacy.

Need to do:

xattr -d com.apple.quarantine  ~/Downloads/multipass-1.14.1-rc1+mac.14+gf2381bfe9.mac-Darwin.pkg

[niluan-zwl]

The same issue 。Today after upgrading to macOS Sequoia 15.01 all instances in multipass cannot be started and cannot obtain an IP address.
The test package worked for me, but some instances have started, while others are still throwing errors.

[2024-10-09T16:51:54.806] [debug] [corplink] QMP: {"QMP": {"version": {"qemu": {"micro": 1, "minor": 2, "major": 8}, "package": ""}, "capabilities": ["oob"]}}

[2024-10-09T16:51:56.393] [info] [general] mac_address value is : 52:54:00:f7:63:df
[2024-10-09T16:51:56.393] [warning] [general] before arp_process->execute()
[2024-10-09T16:51:56.395] [debug] [arp] [4305] started: arp -an
[2024-10-09T16:51:56.398] [info] [general] arp_exit_state.completed_successfully() value is : true
[2024-10-09T16:51:56.398] [info] [general] arp_ouput_stream.toStdString() value is : ? (10.91.128.1) at 0:0:0:0:0:1 on en5 ifscope [ethernet]

[2024-10-09T16:51:57.267] [warning] [corplink] qemu-system-aarch64: error while loading state for instance 0x0 of device 'cpu'

[2024-10-09T16:51:57.267] [warning] [qemu-system-aarch64]
[2024-10-09T16:51:57.268] [warning] [corplink] qemu-system-aarch64: Error -1 while loading VM state

[jakub-zawislak]

I have two instances in Multipass. I installed the test package of Multipass a few days ago. After the installation, only one instance was working. Today, the previously working instance instance also crashed.

In both instances, I have logs like these:

[debug] [primary] process working dir ''
[info] [primary] process program 'qemu-system-aarch64'
[info] [primary] process arguments '-machine, virt,gic-version=3, -accel, hvf, -drive, file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on, -cpu, host, -nic, vmnet-shared,model=virtio-net-pci,mac=52:54:00:6b:21:83, -device, virtio-scsi-pci,id=scsi0, -drive, file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda, -device, scsi-hd,drive=hda,bus=scsi0.0, -smp, 4, -m, 7168M, -qmp, stdio, -chardev, null,id=char0, -serial, chardev:char0, -nographic, -cdrom, /var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/cloud-init-config.iso'
[debug] [qemu-system-aarch64] [647] started: qemu-system-aarch64 -machine virt,gic-version=3 -nographic -dump-vmstate /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/multipassd.OfALEJ
[info] [primary] process state changed to Starting
[info] [primary] process state changed to Running
[debug] [qemu-system-aarch64] [649] started: qemu-system-aarch64 -machine virt,gic-version=3 -accel hvf -drive file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -cpu host -nic vmnet-shared,model=virtio-net-pci,mac=52:54:00:6b:21:83 -device virtio-scsi-pci,id=scsi0 -drive file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 4 -m 7168M -qmp stdio -chardev null,id=char0 -serial chardev:char0 -nographic -cdrom /var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/cloud-init-config.iso
[info] [primary] process started
[debug] [primary] Waiting for SSH to be up
[debug] [arp] [650] started: arp -an
[debug] [primary] QMP: {"QMP": {"version": {"qemu": {"micro": 1, "minor": 2, "major": 8}, "package": ""}, "capabilities": ["oob"]}}

[debug] [primary] QMP: {"return": {}}

[debug] [arp] [872] started: arp -an
[debug] [arp] [893] started: arp -an
[debug] [arp] [1035] started: arp -an
[debug] [arp] [1045] started: arp -an

I already rebooted macOS a few times, I was also stopping instances with --force or killing the qemu process.
I have no issues with creating new instances.

[stuartskelton]

Hi

I just wanted to add the test package worked on my newly updated and fresh 15.0.

Regards

Stuart

[michaeldam]

1.14.1 RC1 works on M1 running on fresh MacOS 15.0.1

Obviously needed the xattr to be able to install: #3661 (comment)

[sekiskylink]

Hi @ricab,
Thanks for the tips here. I am using a Mac with an Intel chip. After installing the 1.14.1 RC1 version, I got the primary instance to run, but my pre-existing instance still times out.

[ricab]

Hi @sekiskylink, that indicates you have some other problem with your instance. In order for us to help you, we'd need a new issue with the requested info, please.

[ricab]

Hi all, just wanted to let everyone involved here know that we finally got the signed packages and released 1.14.1 officially. Apologies for the delay.

[khalid3bdallah]

@ricab
My current version:

multipass   1.15.0-dev.2929.pr661+gc67ef6641.mac
multipassd  1.15.0-dev.2929.pr661+gc67ef6641.mac

Are there specific steps I need to follow to install version 1.14.1?

[ricab]

Hi @khalid3bdallah, not really, you should just be able to download the package from the link above and double click it. The version looks lower, but it is not, because yours is just a dev package (toward 1.15).

The change to multipass.run is in progress too, so people will be able to get it from there too in the next few days.

[SOFTWORK]

Has anyone the Test Package File multipass-1.15.0-dev.2929.pr661+gc67ef6641.mac-Darwin.pkg - the link provided here is dysfunctional - many thanks.

[puneetdawer]

I am not able to access test package and it gives access denied error.

[Marc-Anderson]

@puneetdawer, @SOFTWORK - I was able to successfully start and use Multipass in macOS sequoia using the latest release(1.14.1) which included a fix for this issue.

https://github.com/canonical/multipass/releases/tag/v1.14.1

[tomsonpl]

Unfortunately I am still seeing the issues even after upgrading to 1.14.1.

$ multipass shell test
shell failed: ssh connection failed: 'Failed to connect: No route to host'

It worked only after using these:

multipass set local.passphrase
sudo multipass authenticate
sudo multipass shell test

[ricab]

That's reported in #3766 @tomsonpl .

[alexcardo]

So, today is 5th of December. I've spent the entire day in an attempt to launch Multipass.

This lasts forever ... and no solution...

[ricab]

That is unrelated to this issue @alexcardo. Would you mind opening a new one, filling in the information in the form? Thank you.