SSH password needs to be generated dynamically during auto-install

[abhay-krishna]

Environment

• Make target: Multiple
  • build-node-ova-vsphere-rhel-8
  • build-node-ova-vsphere-ubuntu-2204
  • build-nutanix-ubuntu-2204
  • build-nutanix-ubuntu-2004
  • build-qemu-rhel-8
  • build-qemu-rhel-9
  • build-raw-rhel-8
  • build-raw-rhel-9
  • build-raw-ubuntu-2004
• Run using container image? (Y/N): No

What steps did you take and what happened?

We use image-builder to build images for different CAPI providers (VSphere/CloudStack/Nutanix/Raw).

Yesterday, we migrated from image-builder v0.1.37 to v0.1.38 to leverage the CVE fixes for CVE-2024-9486 and CVE-2024-9594. Immediately after doing so, all our builds across all providers started failing (after a 30 minute SSH connection wait) with the following error:

Waiting for SSH to become available...
Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

When we temporarily reverted to v0.1.37, we didn't see this behavior and all the builds passed. This pointed to something in the v0.1.38 changelog. The only plausible cause we could think of was #1596, since it affected the SSH password used to connect to VM after boot.

After some digging, we figured out that only the SSH password to be used with builder user was changed to be dynamic but the actual password when creating the builder user was still hardcoded to builder (example) . This caused a mismatch during the SSH connection because the SSH was using a random generated UUID, while the expected password was builder, so the SSH connection would never go through and just timed out.

In terms of a fix, since the UUID thing is internal to Packer, it's never printed or returned to the user. Also the builder user and password are created before the ssh_password is used by Packer, so anything generated in Packer will be an after-the-fact thing with respect to user creation, since the flow is unidirectional.

So I think we will need to generate it outside of Packer in the Makefile or script and replace in the JSON ssh_password field as well as in the user creation step.

/kind bug