unable to set up remote access

[mishac]

Post reboot, and post installation, I'm trying to set up remote access to zbm,and it's failing.

Here's the output when I run ubuntu_server_encrypted_root_zfs.sh remoteaccess:

[root:~] # ./ubuntu_server_encrypted_root_zfs.sh remoteaccess
Boot environment check passed. Found EFI boot environment.
Mon Feb 20 09:17:20 AM EST 2023
Running remote access to ZFSBootMenu install. Press Enter to Continue or CTRL+C to abort.

***WARNING*** This script could wipe out all your data, or worse! I am not responsible for your decisions. Press Enter to Continue or CTRL+C to abort.


WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
dracut-network is already the newest version (056-3).
dropbear is already the newest version (2022.82-4).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
fatal: destination path 'dracut-crypt-ssh' already exists and is not an empty directory.
mkdir: cannot create directory ‘/usr/lib/dracut/modules.d/60crypt-ssh’: File exists
cp: -r not specified; omitting directory '/tmp/dracut-crypt-ssh/modules/60crypt-ssh/helper'
Generating public/private rsa key pair.
/etc/dropbear/ssh_host_rsa_key already exists.
Overwrite (y/n)? y
Your identification has been saved in /etc/dropbear/ssh_host_rsa_key
Your public key has been saved in /etc/dropbear/ssh_host_rsa_key.pub
The key fingerprint is:
SHA256:lgVpBeE9O829TQRoRXdK/eWVRhDAbMJbbPl/ZMUpNOo root@mishac
The key's randomart image is:
+---[RSA 3072]----+
|        ===.+OO+=|
|       .o= O+o.BB|
|       .. Xo. +.*|
|         +.= o .=|
|        S oEo oo.|
|       .   .   =.|
|              . o|
|                 |
|                 |
+----[SHA256]-----+
Generating public/private ecdsa key pair.
/etc/dropbear/ssh_host_ecdsa_key already exists.
Overwrite (y/n)? y
Your identification has been saved in /etc/dropbear/ssh_host_ecdsa_key
Your public key has been saved in /etc/dropbear/ssh_host_ecdsa_key.pub
The key fingerprint is:
SHA256:6CjrLvtymb6YAF3sPNKT82/0lvIMD9Yx4VvxCt8mgi8 root@mishac
The key's randomart image is:
+---[ECDSA 256]---+
|                 |
|   .             |
|    o     . .    |
| . = . . . . o   |
|. o O . S = . .  |
|.  . B  .o B o   |
|. .o. o.=.+.+ o  |
|+o+o   oE*+. o   |
|+XB.   ..==      |
+----[SHA256]-----+
  ##Copy dropbear welcome message
  inst /etc/zfsbootmenu/dracut.conf.d/banner.txt /etc/banner.txt
}
Synchronizing state of dropbear.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable dropbear
No initramfs generator specified; using dracut
## No version found in path /boot/vmlinuz, using 5.19.0-31-generic from kernel strings
## Identified version 5.19.0-31-generic for kernel /boot/vmlinuz
## Identified version 5.19.0-31-generic for kernel /boot/vmlinuz-5.19.0-31-generic
## No version found in path /boot/vmlinuz.old, using 5.19.0-31-generic from kernel strings
## Identified version 5.19.0-31-generic for kernel /boot/vmlinuz.old
## Latest kernel: 5.19.0-31-generic
## No version found in path /boot/vmlinuz.old, using 5.19.0-31-generic from kernel strings
## Executing: dracut -f --confdir /etc/zfsbootmenu/dracut.conf.d /tmp/oKx5Z_4VYc/zfsbootmenu.img 5.19.0-31-generic
Creating ZFSBootMenu 2.1.0 from kernel /boot/vmlinuz.old
dracut: Executing: /usr/bin/dracut -f --confdir /etc/zfsbootmenu/dracut.conf.d /tmp/oKx5Z_4VYc/zfsbootmenu.img 5.19.0-31-generic
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'systemd-coredump' will not be installed, because command 'coredumpctl' could not be found!
dracut: dracut module 'systemd-coredump' will not be installed, because command '/usr/lib/systemd/systemd-coredump' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut: dracut module 'dbus-daemon' depends on 'systemd', which can't be installed
dracut: dracut module 'dbus' depends on 'dbus-daemon', which can't be installed
dracut: dracut module 'systemd-networkd' depends on 'dbus', which can't be installed
dracut: dracut module 'network' depends on 'systemd-networkd', which can't be installed
dracut: dracut module 'crypt-ssh' depends on 'network', which can't be installed
dracut: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut: dracut module 'pcsc' will not be installed, because command 'pcscd' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'cifs' depends on 'network', which can't be installed
dracut: dracut module 'fcoe' will not be installed, because command 'dcbtool' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'fipvlan' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'lldpad' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'fcoemon' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'fcoeadm' could not be found!
dracut: dracut module 'fcoe-uefi' will not be installed, because command 'dcbtool' could not be found!
dracut: dracut module 'fcoe-uefi' will not be installed, because command 'fipvlan' could not be found!
dracut: dracut module 'fcoe-uefi' will not be installed, because command 'lldpad' could not be found!
dracut: dracut module 'iscsi' depends on 'network', which can't be installed
dracut: dracut module 'nbd' depends on 'network', which can't be installed
dracut: dracut module 'nfs' depends on 'network', which can't be installed
dracut: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found!
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: dracut module 'crypt-ssh' cannot be found or installed.
Failed to create /tmp/oKx5Z_4VYc/zfsbootmenu.img

It seems to be missing all the dracut modules for reasons I'm unclear on.

[Sithuk]

I was able to replicate an error when I ran a test on the remoteaccess feature just now. Unfortunately it wasn't the error you've experienced.

The error I experienced was that the system couldn't find the dracut-network package. I think the error might have been related to recent apt mirror handling changes. I've added a step to update the package sources before installing dracut-network. Could you test it? You'll need to re-install (or rollback to before you ran the remoteaccess feature) to ensure there is no conflict with the previously failed remoteaccess install attempt.

[mishac]

unfortunately not :(

it still ends with an error that crypt ssh could not be found:

dracut: dracut module 'crypt-ssh' cannot be found or installed.
Failed to create /tmp/aeIkLWeKU0/zfsbootmenu.img

[Sithuk]

You rolled back or tried a re-install before running the remoteaccess feature again?

Dracut-network shouldn't have been installed at that point so you shouldn't see the following in your log:
"dracut-network is already the newest version (056-3)."

You also shouldn't have seen the following, which is indicative of remoteaccess having been previously run.
"fatal: destination path 'dracut-crypt-ssh' already exists and is not an empty directory.
mkdir: cannot create directory ‘/usr/lib/dracut/modules.d/60crypt-ssh’: File exists"

[mishac]

the paste in my original issue was like the eleventeenth time I tried to run it. I'd spent an hour or two trying to manually fix it (removing dropbear.conf, uninstalling dracut-network, clearing the 60-cryppt-ssh folder etc)

the latest one was after rolling back, but perhaps I didn't rollback far enough.

[Sithuk]

It might be quicker just to do a reinstall.

[mishac]

yeah that's my next step I think. It's my main workstation, so I'll be doing it after work hours

[Sithuk]

Which variant are you installing? Server? Ubuntu Desktop?

[mishac]

server but I later installed ubuntu-desktop-minimal, which I did before doing the remoteaccess script.

Honestly this installation was an absolute shitshow, I need to do it again for a myriad of reasons.

On 4 other machines I installed ubuntu-server with your script with no issues whatsoever

[Sithuk]

I've just tested an ubuntu-desktop-minimal install with remoteaccess and it all worked fine. I didn't do a server install first. Let me know how your re-install goes.

[mishac]

so it seems to work on a jammy install, however it fails on kinetic. Which isn't even listed as supported here, so I guess I should have thought more carefully before opening the issue!

[Sithuk]

No problem. You’ve got me curious as to what has changed in kinetic.

[Sithuk]

It looks like the relevant dracut issue is here.
dracutdevs/dracut#1756

A pull request was created to resolve the issue here.
dracutdevs/dracut#2181

[Sithuk]

Add the following line:
add_dracutmodules+=" network-legacy "

after this line:
add_dracutmodules+=" crypt-ssh "

to resolve the issue with remoteaccess on 23.04. The fix is a workaround suggested in the dracut issue thread linked above. When upstream rolls out a fix then the workaround shouldn't be required. I've tested with 23.04 server but not with any of the desktop variants which install network manager.

[Sithuk]

Ive found an issue with the remoteaccess on 23.04 that it only works when using root's authorized_keys. I've identified the problem and suggested a fix for upstream at the link below.
dracut-crypt-ssh/dracut-crypt-ssh#61

I have a fix ready for the script that I can roll out, but I'll see if upstream fixes first.

[Sithuk]

I've gone ahead and added the fixes to the scripts. They shouldn't conflict even if the issues are fixed upstream.
43a3622