[BUG] Windows EC2 minion provisioned with salt-cloud keyed to master incorrectly

[jmcmillan89]

Description
When using salt-cloud -p to provision a Windows EC2 instance in AWS, the resulting minion is keyed to the master incorrectly. When running salt-key -L, two thumbprints appear -- one in Accepted Keys and one in Denied Keys -- the thumbprint in Denied Keys is correct (validated using salt-call key.finger --local from the EC2 machine).

We are installing version 3004.2 of the salt-minion onto Windows Server 2016 using the base/core Windows Server 2016 AMI provided by Amazon (AMI ID ami-0d79078d185bf79b7). This behavior does not occur for Linux EC2 instances.

Setup
provider configuration

my-ec2:
  driver: ec2
  ssh_interface: private_ips

  minion:
    master: salt.foo.bar

  id: *******
  key: *******

  subnetid: subnet-abc1234

  securitygroupname:
    - tools

  private_key: /root/.ssh/salt-cloud.pem
  keyname: salt-cloud

  win_deploy_auth_retries: 10
  win_deploy_auth_retry_delay: 1
  win_installer: /root/Salt-Minion-3004.2-Py3-AMD64-Setup.exe
  win_username: Administrator
  win_password: auto

profile configuration

winserver2016_medium:
  provider: my-ec2
  image: ami-0d79078d185bf79b7
  size: t2.medium
  userdata_file: /etc/salt/cloud.userdata.d/windows-firewall.ps1

windows-firewall.ps1

<powershell>
# See https://docs.saltproject.io/en/latest/topics/cloud/windows.html#firewall-settings
New-NetFirewallRule -Name "SMB445" -DisplayName "SMB445" -Protocol TCP -LocalPort 445
Set-Item (dir wsman:\localhost\Listener\*\Port -Recurse).pspath 445 -Force

# Not sure if these are actually needed
New-NetFirewallRule -Name "SaltStack4505" -DisplayName "SaltStack4505" -Protocol TCP -LocalPort 4505
New-NetFirewallRule -Name "SaltStack4506" -DisplayName "SaltStack4506" -Protocol TCP -LocalPort 4506

Restart-Service winrm
</powershell>

Provision with salt-cloud -p winserver2016_medium winserver2016_medium -l debug. Once the VM is provisioned and command output is successfully returned:

[root@salt.foo.bar cloud.profiles.d]# salt-key -L
Accepted Keys:
winserver2016_medium
Denied Keys:
winserver2016_medium
Unaccepted Keys:
Rejected Keys:

The minion appears in both Accepted Keys and Denied Keys.

A test.ping does not return successfully:

[root@salt.foo.bar cloud.profiles.d]# salt -L winserver2016_medium test.ping
winserver2016_medium:
    Minion did not return. [Not connected]
ERROR: Minions returned with non-zero exit code

Showing the fingerprints:

[root@salt.foo.bar cloud.profiles.d]# salt-key -f winserver2016_medium
Accepted Keys:
winserver2016_medium:  b3:84:5e:03:76:0b:93:a8:1e:fb:d2:2d:76:76:38:ee:5d:54:28:a2:95:4b:62:f1:9e:2d:35:45:7b:e8:b7:a4
Denied Keys:
winserver2016_medium:  f1:7b:01:c1:dd:4d:a5:b3:fb:57:c8:60:3b:11:89:7e:32:6f:d0:b4:63:03:ce:08:f9:2a:7b:d5:a0:58:8f:f2

And then hopping onto the Windows EC2 instance:

C:\Program Files\Salt Project\Salt>salt-call key.finger --local
local:
    f1:7b:01:c1:dd:4d:a5:b3:fb:57:c8:60:3b:11:89:7e:32:6f:d0:b4:63:03:ce:08:f9:2a:7b:d5:a0:58:8f:f2

The incorrect thumbprint is accepted on the Salt master -- how is there even 2 thumbprints to begin with?

I did some poking around and noticed that there are 2 conf dirs however -- one at C:\salt\conf (old pre-3000 salt-minion installations) and one at C:\ProgramData\Salt Project\Salt\conf (newer salt-minion installs). Each has a minion.pub in the pki dir and they contain different contents:

PS C:\Users\Administrator> Get-Content "C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pub"
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvuEBHubzkrOuseYyrypg
jLaBn3ki6djDTi9OTdVhiBJO5kz9MV3qTQe1EcGKjnNGVxXtQlLjykilPQsN539I
AcowFTCyPTqnt0qCvbIv2NERviqj9846T/F7dA/6Fa66IiJ4ZP3WW2LHylCaRGb4
UHR0vY2JW3DLksb9UiPDO7mn1raUy5gQJeCFDZ7VSkkVkfB61G0RwFyKJTnPps0l
67J3nVoGoiTKLAYaWwIIlOib0tv3mv7H/4kfrjo/Fgzy8ltMgl52Zr7BIxZrtDhn
CJ9oDDO5SZ2ZqSH3xwqB7mN/41A8gOP1uHu4fTjrukct1/ZYnPhrrTzWnGCk1Hms
zwIDAQAB
-----END PUBLIC KEY-----

PS C:\Users\Administrator> Get-Content "C:\salt\conf\pki\minion\minion.pub"
-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAl5NIEsXYMij89MAyGANH
8elsPbltA25BewKzySKDJIoqAFIQqVQNAatBA5RXv7cLJbmtukqul2UhbZp+Wi6Q
Kr61qB7oCkfcmZXVow9qThk7XnqdhhuR48PTurEERpqYbNaNZyzlj6HQEXhO6laa
1IY0WMMyYAcVpNSh+gNQt1us3HJKq+j0DHvuDbzdzrQDctIVzVwjE2pss8DnMqiw
50MxTGahUzXqF566kBOXpunrNVnefzvSvuorbVgSVCCFWK6FVdeN6b57LDIkUnb6
piuUVPeUENgmAmF9P81+HPNjgf0I6CDQXjHGEdImdjvCAC7Xi9YumSlw7nrpJQTW
MfR3Qlqyo9J808JdJd27RW9COLrBfGdZ0UPLA12hZFo9XGN1pIzDEP3y2YoWP6G9
LHs1CZnq2u5j1j/bgNtHr6u4xLTpzLtjJRzKMZP4Iq8xkVWkeXa8ksfE5YsXjgPo
bqDYzibu0GxuRyVg8qk0DDJy3l6+9QEHOLs7g4aCxcR4xIqaEgsskHmh7aYhDYPL
WlMFQk1YH6pGfiqCE6PY6YFA+oc+9Ik8IDrthW3QP1ia0ntT+kS5oDTUMc1+uaWN
LBKxp0fRozwGYP9P0FtuUIMqDjrrclgBrldiVFL0Y3WD76cQNvXgizbbpp2ebpDC
toP793tCnz88BbAPZ9rOcOcCAwEAAQ==
-----END PUBLIC KEY-----

Is it possible this is the cause of the 2 thumbprints? My theory is that part of the salt-minion bootstrapping is creating the conf in C:\salt because when we onboard on-prem Windows VMs into Salt using Salt-Minion-3004.2-Py3-AMD64-Setup.exe it creates the conf dir within C:\ProgramData\Salt Project\Salt.

The workaround is to simply unkey with salt-key -d and restart the salt-minion service and it will attempt to re-key with the correct thumbprint. Note however that I did not delete either one of the pki dirs.

Expected behavior
The provisioned EC2 instance is keyed to the Salt master using the correct thumbprint.

Versions Report

[root@salt.foo.bar cloud.profiles.d]# salt --versions-report
Salt Version:
          Salt: 3004.2

Dependency Versions:
          cffi: 1.15.0
      cherrypy: unknown
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.1
       libgit2: 1.3.1
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: 1.7.0
        Python: 3.6.8 (default, Aug 13 2020, 07:46:32)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

System Versions:
          dist: rhel 7.9 Maipo
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-957.21.3.el7.x86_64
        system: Linux
       version: Red Hat Enterprise Linux Server 7.9 Maipo

[whytewolf]

one question. does your windows image already have a copy of salt installed? because remember if salt starts and there is no key where it is expecting it will create one. and these two keys look totally different, like different versions of salt generated them. the accepted key would be the one that is generated by the salt master that gets moved over. the other is generated by a salt-minion. and pushed towards the salt-master on start up.

[jmcmillan89]

That's a great question -- I am just using the base Windows Server 2016 AMI (Microsoft Windows Server 2016 with Desktop Experience Locale English AMI provided by Amazon to be exact). I did manually spin up an EC2 instance using this image and there is no Salt stuff installed.

As an extra test, I went ahead and installed salt-minion using the same binary I am using in salt-cloud and it installed to C:\ProgramData\Salt Project\Salt and C:\Program Files\Salt Project\Salt -- no C:\salt to be found.

[whytewolf]

interesting. so. one of the setups is that salt should detect if c:\salt exists and use it if it already exists. which i think salt-cloud takes advantage of. i know in my personal windows desktop that i have been upgrading forever my config is still found in c:\salt and I'm on 3005.1. and looking at the deploy for salt-cloud it does create c:\salt but it also tells the config to use that directory for the config. so i am not sure why it ends up moving over to the new default when it should already be using the old config directory.

[infantvin]

Is there any way to pass arguments to the installer which we declare for Windows

win_installer: /root/Salt-Minion-3004.2-Py3-AMD64-Setup.exe

there are install arguments like /S and /move-config which may help to have only one directory for the configuration etc.
Please let me know if there is any. If yes maybe it can be a workaround for this issue here.

[twangboy]

The issue has been fixed by: #63030
It is available in the 3006 release.

[knine]

Opinion: Given that users of SaltStack Config are stuck with 3005 until SSC can work with Onedir packages (and Salt decided to release SSC with that limitation and only package 3006 with Onedir), it would seem reasonable that this and other serious bugs be back-ported to 3005 with the older packages.

Yes, support gave me a patch, but that is messing with RHEL file packages which can cause issues if 3005.X is released with CVE security fixes and blows away the changed file.