0.6.7 - Initialization of base image fails / stuck at "expanding disk" - macOS Intel with vmtype vz and virtioifs #930

AndreasA · 2023-12-13T13:18:30Z

Description

After updating to colima 0.6.7 something did not work upon start, so I did the usual

colima stop
colima delete
colima prune -a

# Even did additionally:
brew remove colima lima qemu
brew install colima
brew autoremove
# reboot and reinstall

Then I did a colima start with vm-type vz and virtiofs and the new image failed to load successfully. It got stuck at expanding disk image. After this happened the mac got stuck on shutdown/reboot as well and only a force shutdown worked.

I checked the folder ~/.colima/_lima/colima and it seems the image conversion might have failed or was missing a step as it was called diffdisk..tmp instead of just diffdisk.

I tried not specifying the vz type and virtiofs and it started successfully in QEMU and sshfs mode.

I then uninstalled 0.6.7 and manually downloaded 0.6.6, did a colima start there and it worked fine. I then removed the 0.6.6 colima binary and installed 0.6.7 through brew again and now a start works as well. so it seems the issuue is indeed with 0.6.7 and image initialization. maybe the image has a bug?

I originally had 13.x installed and tried the upgrade to 14.x to check if that might solve the issue. it did not, so the issue occured in both scenarios.

Version

colima version 0.6.7
git commit: ba1be00

runtime: docker
arch: x86_64
client: v24.0.7
server: v24.0.7

limactl version 0.19.0

Operating System

Output of `colima status`

INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: x86_64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs

Reproduction Steps

see description.

Expected behaviour

colima starts even after delete on intel mac with 0.6.7

Additional context

No response

The text was updated successfully, but these errors were encountered:

AndreasA · 2023-12-13T15:06:46Z

Another one just had the same issue at my place of work:

colima start --cpu 4 --memory 12 --ssh-agent --vm-type vz --mount-type virtiofs
INFO[0000] starting colima
INFO[0000] runtime: docker
INFO[0000] creating and starting ...           context=vm
>
> 1.23 GiB / 3.50 GiB (35.10%) ? p/s
> 1.61 GiB / 3.50 GiB (46.09%) 78.80 MiB/s
> 1.91 GiB / 3.50 GiB (54.52%) 77.61 MiB/s
> 3.03 GiB / 3.50 GiB (86.64%) 87.45 MiB/s
> 3.50 GiB / 3.50 GiB (100.00%) 180.90 MiB/stime=“2023-12-13T16:03:26+01:00” level=info msg=“Expanding to 60GiB”

AndreasA · 2023-12-13T15:33:42Z

The only solution I found was to temporarily downgrade to 0.6.6, do the start there, then reinstall 0.6.7.

So something like this - probably not all deletions etc. are necessary but to be safe I did them multiple times:

colima stop -f
# Reboot macOS - warning most likely macOS will freeze during reboot and require a force shutdown.
colima delete
colima prune -a
rm -rf ~/.lima
rm -rf ~/.colima
rm -rf ~/Library/Caches/colima
rm -rf ~/Library/Caches/lima
brew uninstall colima
curl -o ~/colima -L https://github.com/abiosoft/colima/releases/download/v0.6.6/colima-Darwin-x86_64
chmod +x ~/colima
~/colima version
check that it is indeed version 0.6.6
~/colima delete
~/colima prune -a
~/colima start --vm-type vz --mount-type virtiofs ….  # add all necessary parameters.
~/colima stop
rm ~/colima
brew install colima
colima start
colima version

abiosoft · 2023-12-13T17:55:55Z

This specifically happens with vz on intel devices, right?

AndreasA · 2023-12-13T21:15:51Z

@abiosoft yes, qemu version seems to work fine and also I had another colleague try it on an M2 (delete / start) and there it worked fine. but I got the same issue on two Intel Maac with vm type VZ. not sure if virtiofs is relevant or not (though I am not sure iff anything else works in combination with VZ).

AndreasA · 2023-12-14T07:53:49Z

@abiosoft I also just tried the following lima config and manually creating it, and it fails as well with expanding disk:

images:
  - location: 'https://github.com/abiosoft/colima-core/releases/download/v0.6.7/ubuntu-23.10-minimal-cloudimg-amd64.qcow2'
    arch: 'x86_64'
    digest: 'sha512:9980ac00edc0a0b75e9c924f25f3403e76ef8fd680ee1847082de543a8ff851413992a18b065c53092edf298d11de50b0b129ece1ec17217bd8a6e9266b22345'
vmType: 'vz'
mountType: 'virtiofs'

maybe the image or the conversion from qcow2 to raw has some issue. I think the previous one was raw format?

I also found that lima issue, might be related ? lima-vm/lima#1964

though the file does not seem to be empty that has been created but it might have been before it tries to expand the file.

AndreasA · 2023-12-14T08:06:15Z

@abiosoft I did one more test and the following did indeed work (using limactl manually):

download and convert image manually to raw using qemu-img

curl -o ubuntu-23.10-minimal-cloudimg-amd64.qcow2 -L https://github.com/abiosoft/colima-core/releases/download/v0.6.7/ubuntu-23.10-minimal-cloudimg-amd64.qcow2
qemu-img convert ubuntu-23.10-minimal-cloudimg-amd64.qcow2 ubuntu-23.10-minimal-cloudimg-amd64.raw

and then using the following lime config I was able to successfully create the instance:

images:
  - location: './ubuntu-23.10-minimal-cloudimg-amd64.raw'
    arch: 'x86_64'
vmType: 'vz'
mountType: 'virtiofs'

so I guess the issue is indeed lima qcow2 auto-conversion to raw image.

abiosoft · 2023-12-14T08:07:34Z

Interesting.

Thanks for the detailed feedback.

mvmn · 2023-12-21T15:52:30Z

Got the same issue

AndreasA · 2023-12-22T06:31:00Z

@abiosoft The easiest solution would be to switch back to the raw image type until the lima bug is fixed.
at least for intel macs when using vmtype vz.
not sure how easy the check in the program would be though 😄

abiosoft · 2023-12-22T07:03:17Z

@AndreasA colima can actually do the conversion to raw as a temporary workaround.

This is specific to Intel as I am unable to reproduce on my m1 devices.

AndreasA · 2023-12-22T07:30:12Z

@abiosoft that would be even better. strictly speaking it could always be done for now for VZ vm type, as lima does the conversion anyway. Any idea when the corresponding update will be available (approximately)?

AndreasA · 2024-01-10T07:40:16Z

@abiosoft any news when this might be fixed? as it bascially prevents me to do a colima delete (though it is probably not necessary now that I have it running again with 0.6.7) without having to downgrade again.

simonjharris · 2024-02-03T11:16:17Z

I am seeing the same error on an M2 air with 0.6.8. Is there any workaround for this?

pgarrett-twc · 2024-02-05T17:01:33Z

Same error with colima 0.6.8 on M3 Pro on Sonoma 14.3. Well, it was hanging on the "expanding disk" step when I did colima start immediately after brew install colima. Not sure about the virtualization settings.

simonjharris · 2024-02-05T17:09:49Z

I downgraded to 0.5.6 and it's working fine on there. M3 is using vz by default AFAIK so likely same issue @pgarrett-twc.

realnikolaj · 2024-02-06T17:39:09Z

Had the same problem.
Reinstalling colima using
brew install --HEAD colima
did not fix the problem, but it's the version I used when the below fixed my issue.

What fixed my issues was
brew uninstall colima lima qemu
rm -rf ~/.colima ~/.lima
restarted macOS
'brew install --HEAD colima'

Used limactl to create one instance first (don't know if this did anything)
limactl create --arch x86_64 --cpus 2 --disk 20 --name default --vm-type vz

Then stopped it
limactl stop

And used colima - NOTE the smaller disk size
colima start default --arch x86_64 --cpu 8 --memory 8 --disk 20 --vm-type=vz

Then docker ps worked again

terev · 2024-02-07T10:09:29Z

If there were a raw version of the latest colima image, users could workaround this with an override right?

ralph089 · 2024-02-09T10:51:19Z

None of the above solutions worked for me (M2 Mac). I switched to Lima, which worked on the first try.

limactl start \
  --name=default \
  --cpus=4 \
  --memory=8 \
  --vm-type=vz \
  --rosetta \
  --mount-type=virtiofs \
  --mount-writable \
  --network=vzNAT \
  template://docker

wondris009 · 2024-02-12T09:14:10Z

Stopped working on M1 as well. During this phase:
> 3.50 GiB / 3.50 GiB (100.00%) 393.94 MiB/stime="2024-02-12T10:07:40+01:00" level=info msg="Expanding to 100GiB"

It eats all my HDD space. My settings is:

# Default: 2
cpu: 4
 
# Size of the disk in GiB to be allocated to the virtual machine.
# NOTE: changing this has no effect after the virtual machine has been created.
# Default: 60
disk: 100
 
# Size of the memory in GiB to be allocated to the virtual machine.
# Default: 2
memory: 8
 
# Architecture of the virtual machine (x86_64, aarch64, host).
# Default: host
arch: aarch64
 
# Container runtime to be used (docker, containerd).
# Default: docker
runtime: docker

...
 
# Virtual Machine type (qemu, vz)
# NOTE: this is macOS 13 only. For Linux and macOS <13.0, qemu is always used.
#
# vz is macOS virtualization framework and requires macOS 13
#
# Default: qemu
vmType: vz

# Utilise rosetta for amd64 emulation (requires m1 mac and vmType `vz`)
# Default: false
rosetta: true
 
# Volume mount driver for the virtual machine (virtiofs, 9p, sshfs).
#
# virtiofs is limited to macOS and vmType `vz`. It is the fastest of the options.
#
# 9p is the recommended and the most stable option for vmType `qemu`.
#
# sshfs is faster than 9p but the least reliable of the options (when there are lots
# of concurrent reads or writes).
#
# Default: virtiofs (for vz), sshfs (for qemu)
mountType: virtiofs```

egze · 2024-02-12T12:53:24Z

Works on M1 Pro, but doesn't work on M3 for me

wondris009 · 2024-02-12T15:08:24Z

I changed the space back to default (60GB, i had about 70GB left on my device when i was trying 100GB option) then the expanding finished without problem (maybe it would finish with 100GB as well if i had that space left). I was just wondering why it stopped working when i didn't make any change. But now after some docker and colima reinstalations its working again. My use case was that my ARM64 Oracle DB stopped working.

pgarrett-twc · 2024-02-13T03:31:40Z

What fixed my issues was
brew uninstall colima lima qemu
rm -rf ~/.colima ~/.lima
restarted macOS
'brew install --HEAD colima'

This worked for me on an M3 with 14.3.1 (23D60).

simonjharris · 2024-02-13T08:50:13Z

What fixed my issues was
brew uninstall colima lima qemu
rm -rf ~/.colima ~/.lima
restarted macOS
'brew install --HEAD colima'

This worked for me on an M3 with 14.3.1 (23D60).

This actually did work for me, but it still took around 5 minutes to expand the disk. So now I'm wondering if I just wasn't patient enough before 😆

On 0.5.x, initial startup is much faster.

egze · 2024-02-13T09:22:51Z

Updated my iMac to 14.3.1 from 14.2.1 and it worked with brew install --HEAD colima 🤯

egze · 2024-02-13T14:40:44Z

The --head is actually the same as 0.6.8. I reinstalled again with 0.6.8 and it also worked, but also took at least 5 minutes.

rfay · 2024-02-13T23:01:28Z

When having this problem I also see caching behavior:
Using cache "/Users/rfay/Library/Caches/lima/download/by-url-sha256/63f6e459cc61986ae586c5402c4e6e95ddc9073986f2a766ff662f19e0151407/data"

So best to rm -rf ~/Library/Caches/lima as well.

literatesnow · 2024-02-16T21:16:59Z

I also have this problem on a M2 Max, Sonoma 14.3.1, colima 0.6.8.

colima start --arch aarch64 --cpu 4 --disk 128 --memory 6 --vm-type=vz --vz-rosetta

Took 1 hour 20 minutes, activity monitor shows around 15MB/sec disk writes.

Using limactl start ... (from above) created the VM in under a minute.

mastertheif · 2024-02-20T08:49:16Z

Same issue here on an older intel MacOS Ventura 13.6.4, with colima 0.6.7, if i recall correctly the version 0.6.6 had the same issue.
I was able to circumvent the issue by downgrading to colima 0.6.5, start the vm, stop it and then upgrade

Here is how if it helps someone

brew uninstall colima && rm -rf ~/.colima # Removing the folder just in case I don´t think it is necessary

# this is the formulae for the version 0.6.5
curl https://raw.githubusercontent.com/Homebrew/homebrew-core/c3c2a6daa274d5f4b9cc7bb3d25b3cfde50b4e0a/Formula/c/colima.rb -o /tmp/colima.rb 

brew install --formulae /tmp/colima.rb
colima start --vm-type=vz --mount-type=virtiofs --disk 60
colima stop

brew update && brew upgrade

igarridot · 2024-02-21T20:17:25Z

Same issue on Apple Silicon M2 running MacOS Sonoma 14.3.1 and Colima 0.6.8 but working fine with Colima 0.6.5.

rfay · 2024-02-22T21:08:49Z

I also found that I was able to get past this by using --disk 60 instead of --disk 100. I have plenty of disk. I have no idea why that would be an issue.

I also noted that during several failures I saw that there were zombie instances of limactl that could not be killed, so when I saw them (ps -ef |grep lima) I had to restart to get rid of them.

blvp · 2024-03-09T16:58:34Z

The only solution I found was to temporarily downgrade to 0.6.6, do the start there, then reinstall 0.6.7.

So something like this - probably not all deletions etc. are necessary but to be safe I did them multiple times:

colima stop -f
# Reboot macOS - warning most likely macOS will freeze during reboot and require a force shutdown.
colima delete
colima prune -a
rm -rf ~/.lima
rm -rf ~/.colima
rm -rf ~/Library/Caches/colima
rm -rf ~/Library/Caches/lima
brew uninstall colima
curl -o ~/colima -L https://github.com/abiosoft/colima/releases/download/v0.6.6/colima-Darwin-x86_64
chmod +x ~/colima
~/colima version
check that it is indeed version 0.6.6
~/colima delete
~/colima prune -a
~/colima start --vm-type vz --mount-type virtiofs ….  # add all necessary parameters.
~/colima stop
rm ~/colima
brew install colima
colima start
colima version

Facing the same issue, going through 0.6.6 initialization and then upgrading to latest did the job well.

rsau · 2024-03-11T21:39:43Z

I experimented with different disk sizes on M1, Sonoma 14.4.

If disk is 40 then it starts very quickly without any issues but for other sizes it gets stuck

colima 0.6.8

For last 10 mins
> 3.50 GiB / 3.50 GiB (100.00%) 413.59 MiB/stime="2024-03-12T08:27:54+11:00" level=info msg="Expanding to 160GiB"

I have 273 GB free

sgtsquiggs · 2024-03-12T20:19:15Z

Using brew install --HEAD colima worked for me on a fresh M3

Flexicon · 2024-03-14T09:17:44Z

Can confirm that using brew install --HEAD colima worked for a few people in my organization.

It seems to specifically be people that are on M2/3 that are affected. Intel and M1 seems fine for the most part.

krystof-k · 2024-03-27T07:12:53Z

Mac mini M1, this

colima start --cpu 8 --memory 12 --disk 128 --arch aarch64 --vm-type=vz --vz-rosetta

took several hours writing 15 MB/s to the disk. Version 0.6.8.

AndreasA · 2024-05-08T05:10:09Z

I haven't tried this yet but the newest lima 0.22.0 version should include a bugfix regarding the issue lima-vm/lima#1964 so I hope this is fixed now as well. Bugfix: lima-vm/lima#2255 which is mentioned in the full changes: https://github.com/lima-vm/lima/milestone/45?closed=1

abiosoft · 2024-05-08T05:23:30Z

This has been fixed in the latest head. brew install --head colima.

It seems the issue is due to the way the Colima disk image was compressed. With compression disabled, it works as desired. The only downside is heavier download during first startup.

Flexicon · 2024-05-08T14:57:49Z

@abiosoft any timeline on when the change will be released and we can just install latest (v0.6.9)?

luisdavim · 2024-05-16T19:58:02Z

I had the same issue and using brew install --head colima fixed it, any chances of a new release soon?

I ran:

brew uninstall colima
rm -rf ~/Library/Caches/lima ~/.colima ~/.lima
brew install --head colima
colima start --mount $HOME/workingCopies:w -c 10 -m 16 --disk 200 --ssh-agent --vm-type vz --vz-rosetta --mount-type virtiofs

abiosoft · 2024-05-17T17:49:04Z

I had the same issue and using brew install --head colima fixed it, any chances of a new release soon?

Yeah, over the weekend.

abiosoft · 2024-05-18T05:55:48Z

v0.6.9 has been released and should be available on homebrew shortly.

literatesnow · 2024-06-28T03:29:18Z

I also have this problem on a M2 Max, Sonoma 14.3.1, colima 0.6.8.

Version 0.6.9 confirmed to have fixed the issue for me, thanks!

abiosoft · 2024-07-29T19:24:49Z

Colima now does this instead #930 (comment) as a permanent fix and would be part of the imminent next release.

The disk image downloads are smaller again but without the disk expansion issues.

Anyone can try the development version with brew install --head colima.

AndreasA changed the title ~~0.6.7 - Initialization of base image fails / stuck at "expanding disk" (macOS Intel) and vz with virtioifs~~ 0.6.7 - Initialization of base image fails / stuck at "expanding disk" - macOS Intel with vmtype vz and virtioifs Dec 13, 2023

AndreasA mentioned this issue Dec 14, 2023

vz A lima _disk converted from qcow2 > raw on startup truncates to 0 bytes and can't be used in the vm. lima-vm/lima#1964

Closed

AndreasA mentioned this issue Dec 21, 2023

macOS 14 VM Type VZ / Virtiofs - SSH agent forward fails after colima stop / start (if container with agent was still running) #942

Open

5 tasks

terev mentioned this issue Feb 7, 2024

colima start error #961

Closed

5 tasks

joshuarli mentioned this issue Mar 15, 2024

fix(devenv): downgrade to colima 0.6.6 getsentry/sentry#67071

Merged

abiosoft closed this as completed Jul 29, 2024

0.6.7 - Initialization of base image fails / stuck at "expanding disk" - macOS Intel with vmtype vz and virtioifs #930

0.6.7 - Initialization of base image fails / stuck at "expanding disk" - macOS Intel with vmtype vz and virtioifs #930

Comments

AndreasA commented Dec 13, 2023 • edited Loading

Description

Version

Operating System

Output of colima status

Reproduction Steps

Expected behaviour

Additional context

AndreasA commented Dec 13, 2023 • edited Loading

AndreasA commented Dec 13, 2023

abiosoft commented Dec 13, 2023

AndreasA commented Dec 13, 2023

AndreasA commented Dec 14, 2023 • edited Loading

AndreasA commented Dec 14, 2023

abiosoft commented Dec 14, 2023

mvmn commented Dec 21, 2023

AndreasA commented Dec 22, 2023 • edited Loading

abiosoft commented Dec 22, 2023

AndreasA commented Dec 22, 2023 • edited Loading

AndreasA commented Jan 10, 2024

simonjharris commented Feb 3, 2024

pgarrett-twc commented Feb 5, 2024 • edited Loading

simonjharris commented Feb 5, 2024

realnikolaj commented Feb 6, 2024

terev commented Feb 7, 2024

ralph089 commented Feb 9, 2024 • edited Loading

wondris009 commented Feb 12, 2024

egze commented Feb 12, 2024

wondris009 commented Feb 12, 2024 • edited Loading

pgarrett-twc commented Feb 13, 2024

simonjharris commented Feb 13, 2024

egze commented Feb 13, 2024

egze commented Feb 13, 2024

rfay commented Feb 13, 2024

literatesnow commented Feb 16, 2024

mastertheif commented Feb 20, 2024 • edited Loading

igarridot commented Feb 21, 2024 • edited Loading

rfay commented Feb 22, 2024

blvp commented Mar 9, 2024

rsau commented Mar 11, 2024

sgtsquiggs commented Mar 12, 2024

Flexicon commented Mar 14, 2024

krystof-k commented Mar 27, 2024 • edited Loading

AndreasA commented May 8, 2024

abiosoft commented May 8, 2024

Flexicon commented May 8, 2024

luisdavim commented May 16, 2024 • edited Loading

abiosoft commented May 17, 2024

abiosoft commented May 18, 2024

literatesnow commented Jun 28, 2024

abiosoft commented Jul 29, 2024

AndreasA commented Dec 13, 2023 •

edited

Loading

Output of `colima status`

AndreasA commented Dec 13, 2023 •

edited

Loading

AndreasA commented Dec 14, 2023 •

edited

Loading

AndreasA commented Dec 22, 2023 •

edited

Loading

AndreasA commented Dec 22, 2023 •

edited

Loading

pgarrett-twc commented Feb 5, 2024 •

edited

Loading

ralph089 commented Feb 9, 2024 •

edited

Loading

wondris009 commented Feb 12, 2024 •

edited

Loading

mastertheif commented Feb 20, 2024 •

edited

Loading

igarridot commented Feb 21, 2024 •

edited

Loading

krystof-k commented Mar 27, 2024 •

edited

Loading

luisdavim commented May 16, 2024 •

edited

Loading