SSH Handshake Error When Trying to connect to a RHEL9 image (using shell provisioner)

[safeaim]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

When filing a bug, please include the following headings if possible. Any
example text in this template can be deleted.

Overview of the Issue

In our build pipeline we are having issues with the shell provisioner not being able to connect to VM through the shell provisioner once the builder has finished setting up the VM. Once the vsphere-iso-builder has finished installing the VM and reboots, the shell provisioner is supposed to connect to the VM and run a script. However that fails with the following error seen in the packer-log:

2022/11/01 15:09:50 packer-builder-null plugin: [INFO] Attempting SSH connection to 1.1.1.1:22...
2022/11/01 15:09:50 packer-builder-null plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/11/01 15:09:50 packer-builder-null plugin: [DEBUG] handshaking with SSH
2022/11/01 15:09:50 packer-builder-null plugin: [DEBUG] SSH handshake err: ssh: handshake failed: EOF 

From the OS it looks like this.

Nov 01 15:09:43 localhost.localdomain sshd[7336]: error: kex_exchange_identification: Connection closed by remote host
Nov 01 15:09:43 localhost.localdomain sshd[7336]: Connection closed by $PACKER_PROVISIONER_IP port 56930
Nov 01 15:09:43 localhost.localdomain sshd[7337]: fatal: mm_answer_sign: sign: error in libcrypt

I have setup a small test packer-template that is only using the null builder and the shell provisioner to rule out involvement of the vsphere-iso-plugin, and even setup packer on a fresh machine without any older packer-references. And even then I'm still seeing the same error when packer tries to connect to the VM.

Reproduction Steps

Install a CentOS Stream 9/RHEL9 VM and set the following OpenSSH hardening settings in sshd.config:

KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
HostkeyAlgorithms ssh-ed25519,ssh-rsa

Then run the following hcl-template against the VM:

source "null" "basic-example" {
    ssh_host =  "1.1.1.1"
    ssh_username =  "root"
    ssh_password = "rootpw"

 }

build {
  sources = ["sources.null.basic-example"]
  provisioner "shell" {
    inline = ["echo foo"]
  }
}

Packer will then retry connecting forever until it times out.

Packer version

Packer v1.8.4

Simplified Packer Template

source "null" "basic-example" {
    ssh_host =  "1.1.1.1"
    ssh_username =  "root"
    ssh_password = "rootpw"

 }

build {
  sources = ["sources.null.basic-example"]
  provisioner "shell" {
    inline = ["echo foo"]
  }
}

Operating system and Environment details

Red Hat Enterprise Linux release 9.0 (Plow) (5.14.0-70.26.1.el9_0.x86_64)

Log Fragments and crash.log files

packerlog.txt

[aaronyeung]

I ran into this issue this morning building a new RHEL9 template using vsphere-iso.

Digging into this a bit more it looks similar to a Red Hat issue [1] where they dropped support for ssh-rsa (signing with SHA-1)

I then found #8609 [2] and saw that the fix had been merged to main already.

After upgrading my packer vsphere plugin from 1.0.2 -> 1.1.0 and my packer version to latest (1.8.4) it appears that my issue is resolved now.

I inserted this snippet into my template and ran packer init -upgrade.

packer {
  required_plugins {
    vsphere = {
      version = ">= 1.1.0"
      source  = "github.com/hashicorp/vsphere"
    }
  }
}

Hope that helps

1. https://bugzilla.redhat.com/show_bug.cgi?id=2088750
2. ssh handshake failure after changing system-wide crypto policy to FIPS #8609

[nywilken]

@safeaim are you still running into issues here?

If you haven't already I would suggest trying with the latest version the vSphere plugin. There was a fix to the crypto libraries used by Packer to address the ssh-rsa issue that was affecting new Linux distros; especially RHEL variants. Please let us know if this issue gets resolved after upgrading. Cheers!

[oyvindkolbu]

We've had the same problems and after upgrading to the latest vpshere plugin the ssh issue with newer Rocky 9 images was resolved.

[tenthirtyam]

Recommended for transfer to packer-plugin-vsphere and subsequent closure.

cc @nywilken @lbajolet-hashicorp

[lbajolet-hashicorp]

Hi @tenthirtyam,

This was indeed a vsphere issue needing to be updated to use the updated SDK where we made the fix, but since this is already fixed let's skip the transfer and close it here now :)

Thanks for the ping!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.