Faulty installation of Ubuntu (Waiting for the essential requirement 1 of 5: "ssh"): util.py[WARNING]: Failed to create user admin

[Mottl]

Description

Steps to reproduce:
limactl create template://ubuntu --name=ubuntu
or
limactl create template://ubuntu-lts --name=ubuntu
limactl start ubuntu

Output:

INFO[0000] Using the existing instance "ubuntu"
INFO[0000] QEMU binary "/usr/local/bin/qemu-system-x86_64" seems properly signed with the "com.apple.security.hypervisor" entitlement
INFO[0001] [hostagent] hostagent socket created at /Users/admin/.lima/ubuntu/ha.sock
INFO[0001] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/admin/.lima/ubuntu/serial*.log")
INFO[0001] SSH Local Port: 56893
INFO[0001] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0011] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
...

Obviously, there is a problem with creating the admin user in cloud-init
Snippet from ~/.lima/ubuntu/serial.log:

[   22.661276] cloud-init[586]: 2023-11-28 05:48:29,178 - util.py[WARNING]: Failed to create user admin
...
[   77.605933] cloud-init[2346]: + loginctl terminate-user admin
[   77.616303] cloud-init[2346]: Failed to look up user admin: No such process
...
[   77.864749] cloud-init[2346]: + grep -q # Lima BEGIN /home/admin.linux/.profile
[   77.867782] cloud-init[2346]: grep: /home/admin.linux/.profile: No such file or directory
[   77.872246] cloud-init[2346]: /mnt/lima-cidata/boot/20-rootless-base.sh: 10: cannot create /home/admin.linux/.profile: Directory nonexistent
[   77.878411] cloud-init[2346]: LIMA| WARNING: Failed to execute /mnt/lima-cidata/boot/20-rootless-base.sh
...
[   77.911972] cloud-init[2346]: + id -g admin
[   77.914698] cloud-init[2346]: id: ‘admin’: no such user
[   77.917552] cloud-init[2346]: + gid=
[   77.919394] cloud-init[2346]: LIMA| WARNING: Failed to execute /mnt/lima-cidata/boot/25-guestagent-base.sh
...

Alpine Linux works as expected

[AkihiroSuda]

Is this issue specific to your username "admin"?

[Mottl]

I don't have a clue. If that is, it's really weird 😐️️️️️️

[AkihiroSuda]

Do you see other errors around the util.py[WARNING]: Failed to create user admin line?

[Mottl]

Just a deprecation notification a few lines before:

[   11.825236] cloud-init[575]: 2023-11-28 06:25:06,549 - log.py[DEPRECATED]: Deprecated cloud-config provided:
[   11.827869] cloud-init[575]: ca-certs:  Deprecated in version 22.3. Use ``ca_certs`` instead.
[   11.830399] cloud-init[575]: 2023-11-28 06:25:06,549 - schema.py[WARNING]: Invalid cloud-config provided: Please run 'sudo cloud-init schema --system' to see the schema errors.
[   12.810965] cloud-init[575]: 2023-11-28 06:25:07,534 - log.py[DEPRECATED]: Key 'ca-certs' is deprecated in 22.1 and scheduled to be removed in 27.1. Use 'ca_certs' instead.
[   12.820379] cloud-init[575]: Updating certificates in /etc/ssl/certs...
[   13.789139] cloud-init[575]: 0 added, 0 removed; done.

[jandubois]

Obviously, there is a problem with creating the admin user in cloud-init

Yes, I think the problem is that cloud-init by default creates both a user and a group with the same name, and the admin group already exists on Ubuntu.

This behaviour can be disabled with the no_user_group option in the user definition in user-data:

users:
- name: admin
  no_user_group: true
  ...

I'm not sure why we need to create the user group in Lima; maybe we should disable this for all users. What do you think @AkihiroSuda? I just wonder if this will break existing use-cases, and we would have to make it configurable, which I would like to avoid. I would rather say: "Any instance created by Lima 1.0 or later will not create an additional group for the user".

Otherwise I expect any user names that match existing group names can't be used. We do have logic to use lima as the user name when the host username is invalid on Linux, but we can't know the list of existing group names until the instance is booted, as which time it is too late to switch the username.

[jandubois]

Hmm, we also have #1015 to allow the user to specify a different username for the VM. Maybe that is the best way to deal with this because even when we avoid creating the group name, we would still have the potential clash with existing user names in the Linux distro (bin, daemon, sys, etc), which would still throw an error unless #1015 is implemented.

[Mottl]

I'm not familiar with Lima, so the question is: what is the reason of falling back to lima username instead of using lima by default (i.e. not using host username at all).

[jandubois]

what is the reason of falling back to lima username instead of using lima by default (i.g. not using host username at all).

Lima provides the illusion of seamlessly blending the host and the guest environment. You can mount volumes from the host and use them inside the guest. It makes it look like nerdctl actually runs on the host which technically it can't. So showing the host username as the owner of the mounted filesystems helps to maintain this feeling of an integrated environment.

So I think providing a mechanism for users to override the username should help to deal with the conflicting names that we can't detect automatically (but maybe we should have a hard-coded list of "known" legacy Unix usernames that are almost always unavailable).

Note that you wouldn't have to specify the alternate username all the time; you could create a default.yaml or override.yaml to specify the alternative name once, and then limactl start would automatically use it with all the templates.

[Mottl]

@jandubois, could you provide with the snippets of default.yaml and override.yaml? What exactly should I put there and where these files should be placed?

[AkihiroSuda]

I'm not sure why we need to create the user group in Lima; maybe we should disable this for all users. What do you think @AkihiroSuda?

I don't remember, but I assumed that corresponds to the "standard".

[afbjorklund]

Apparently "admin" is a special group only created in Ubuntu, which is why it doesn't fail Alpine or Fedora.

https://wiki.ubuntu.com/PrecisePangolin/ReleaseNotes/UbuntuDesktop#PrecisePangolin.2FReleaseNotes.2FCommonInfrastructure.Common_Infrastructure

Up until Ubuntu 11.10, administrator access using the sudo tool was granted via the "admin" Unix group. In Ubuntu 12.04, administrator access will be granted via the "sudo" group. This makes Ubuntu more consistent with the upstream implementation and Debian. For compatibility purposes, the "admin" group will continue to provide sudo/administrator access in 12.04.

Trying to use "shadow" also fails: #974

[afbjorklund]

The current CIDATA is missing entries for group, so some scripts are currently hardcoding group==user assumption.

LIMA_CIDATA_USER=anders
LIMA_CIDATA_UID=1000

Could probably be a good idea to also export the group, in parallel to whatever the resolution is for system names....

LIMA_CIDATA_GROUP=anders
LIMA_CIDATA_GID=1000

[jandubois]

some scripts are currently hardcoding group==user assumption

As I mentioned above, this is the default behaviour of cloud-init.

If we don't want to disable this behaviour (via no_user_group: true), then my suggestion would be that we enhance the "valid user name" check to also check against a list of "well-known" default user names and groups, and we will use the fallback username lima if the host user name matches any of them.

This would of course be guarded by the new lima-version mechanism from #2107, so only applies to instances created by 0.20.0 (or 0.21.0, or whenever this gets implemented).

[afbjorklund]

I just meant that we could add another variable (or two), so that users could update (or improve) their scripts

Similar to using $HOME instead of hardcoding /home/$USER, or whatever the best analogy would be here.

[jandubois]

I just meant that we could add another variable (or two), so that users could update (or improve) their scripts

I see, but not sure why that is necessary. First we have the guarantee from cloud-init that the group will be the same as the user. But you can also always use id to query the proper value. You can do this even from a system provisioning script, as long as you have the username:

# export LIMA_CIDATA_USER=jan
# sudo -u $LIMA_CIDATA_USER id -g -n
jan
# sudo -u $LIMA_CIDATA_USER id -g
1000

But even if these values would not be readily available, we should first turn provisioning scripts into templates instead of making things available as unsupported env variables (see #2055).

my suggestion would be that we enhance the "valid user name" check to also check against a list of "well-known" default user names and groups

My suggestion was supposed to address the issue of users with conflicting user names. We have seen at least 3 instances of people using admin on the host. By using the lima fallback automatically we would make Lima work for them out-of-the-box.