Stuck on "ssh" or "user session is ready for ssh"

[warrenseine]

Description

I'm basically experiencing what is described in #18 but I prefer to open a new issue because it's been a long time. Plus it's happening with vz which is still considered experimental.

I'm using Lima as a replacement for Docker Desktop on macOS 13.1. Here's the configuration I use.

Everything works fine for a while but after some time I get stuck connecting to the VM, possibly after the laptop goes to sleep, waiting for requirement 1 or 2. It is worth mentioning that a factory-reset solves the issue. It was also reproduced by coworkers so unlikely to be specific to my computer. I have a feeling it might be related to our VPN configuration messing with the network configuration Lima is setting up, but that's hard to say.

I'm attaching logs:

• ha.stderr.log
• ha.stdout.log
• serial.log

Sorry for the lack of details on my side. I'll try to collect any correlation I can make as we go.

[balajiv113]

@warrenseine
Could you give a try with default vz example. See if that works ??

From serial.log, i could see vm started successfully. So like you said something happening with network configuration only

[thr3a]

I also have the same problem with vz.

After putting the laptop to sleep for a while, I sometimes get a connection error in the limactl shell.

In that case, limactl stop -f may fix it, and in other cases, it can't be fixed without factory reset.

I was able to reproduce the same issue with the vz default config.

❯ lima -v
limactl version 0.14.2

❯ sw_vers
ProductName:		macOS
ProductVersion:		13.0.1
BuildVersion:		22A400

[balajiv113]

@thr3a
Sure thanks for the info. Will try to reproduce this.

If possible when this happens could you check ha.stderr.log and see if it has some helpful error (check before doing stop -f and error when trying limactl shell)

[balajiv113]

I have created a feature request in vz (Code-Hex/vz#118).
With this support, hopefully we should be able to debug network error better.

I tried with sleep and wake scenarios. For some reason, it works just fine for me.
But most probably the root cause should be related to UDP socket being closed. But even with this assumption, doing limactl stop -f and limactl start should work again.

[warrenseine]

Yes, most of the time stop -f does the job, but I agree with @thr3a, in some cases a factory-reset is still needed. Maybe these are 2 separate issues.

Thank you for the feature request to vz. Let's hope this will unlock a fix for the first case already!

[balajiv113]

@thr3a / @warrenseine
Could you try the following and share the serial.log only for cases when stop -f didn't work.

Note: The below steps is purely for debugging purpose only

• Goto your instance folder (The one that is not working even after stop -f)
• Edit lima.yaml
  • change vmType to qemu
  • remove mountType, rossetta or any vz specific properties
• start the vm using limactl start

Share the current serial.log

[warrenseine]

Hello @balajiv113, this has just happened, and I'm stuck in a Waiting for the essential requirement 1 of 3: "ssh" loop after a limactl stop -f && limactl start. Here are the log files before I changed the vm type:

• ha.stderr.log
• ha.stdout.log
• serial.log
• lima.yaml.txt

And here are the ones after:

• ha.stderr.log
• ha.stdout.log
• serial.log
• lima.yaml.txt

It looks like it's still stuck after switching to qemu. And as usual, a factory reset works.

[balajiv113]

Thanks for this logs.

Looks like a case of disk corruption only. Mount of root filesystem is what didn't happen here.
Maybe after #1315 is merged. We can try again to see if this happens after that.

[warrenseine]

I've tested again with 0.15 and the issue is still present (can't SSH, still seeing kex_exchange_identification: read: Connection reset by peer in ha.stderr.log). I was not able to find any additional relevant information in the logs.

I'll factory reset and update this issue if something changes.

[thr3a]

I am currently using colima, and it is very stable and reliable.
However, since colima is just a wrapper for Lima, I don't understand why I had problems with Lima alone, but those problems were resolved by using colima.

❯ lima -v
limactl version 0.15.0
❯ sw_vers
ProductName:		macOS
ProductVersion:		13.2.1
BuildVersion:		22D68

colima start --cpu 4 --memory 4 --vm-type vz --vz-rosetta --mount-type virtiofs --runtime docker

[warrenseine]

I've tested Colima for the past week and I still have the same problem of losing the VM, requiring a restart, but the good news is that I never have to factory reset the VM, so it must be doing something.

[na0x2c6]

@warrenseine @balajiv113
Hello. I came across this issue when I was creating my own podman template, so I thought I'd comment.

In conclusion, it seems that this problem is less likely to be affected by disk corruption when booting from an iso image.

Colima seems to be using the alpine image, but this one is based on an iso image:

https://github.com/abiosoft/colima/blob/ed49fef4ad63ec48055b4b230ecbad00c54f5417/environment/vm/lima/yaml.go#L56-L57

With iso images, directories like /usr, /bin, /lib, and /sbin are provided by tmpfs, so it seems to start from a clean state upon reboot. (Reference: #255 (comment))