Can't SSH in using public key authentication when running as a service, can when sshd is run from cmd line

[TCS-Taylor]

"OpenSSH for Windows" version
sshd version OpenSSH_for_Windows_7.7, LibreSSL 2.6.4

Server OperatingSystem
Windows 7

Client OperatingSystem
Windows 10

What is failing
unable to ssh in using public key when the service is running.

Expected output
I expect I should be able to ssh in using public key authentication

Actual output
I am unable to ssh in using public key when the service is running (started thru services.msc). If I stop the service and run c:\program files\openssh\sshd.exe -ddd and then attempt to ssh in using a public key, it works just fine.
When I attempt to ssh in using public key authentication with the sshd service running (started thru services.msc) I get three entries in my eventvwr.exe (in this order)

(ERROR) Event ID 2: sshd:error: unable to generate token on 2nd attempt for user DOMAINNAME\root
(ERROR) Event ID 2: sshd:error: unable to get security token for user DOMAINNAME\root
(CRITICAL) Event ID 1: sshd:fatal: fork of unprivileged child failed

Any help you can provide would be most appreciated!

Taylor

[manojampalam]

Most likely due to you domain not being accessible. If you are outside your domain network, try VPN or other means to connect to your domain first and retry.

[TCS-Taylor]

manojampalam - My domain is accessible, as is evident by the fact that I can SSH into the windows 7 machine when sshd is being run interactively.

[manojampalam]

When sshd is running interactively (to be used only for testing/diagnostics), the authentication flow is different - it doesn't invoke S4U token generation logic that needs access to domain.

[TCS-Taylor]

Apologies, I was unaware of the difference. I tested SSHing in using a local user, with the service running and authenticating with public keys, and it worked great. So it definitely has something to do with the domain users.

My domain is accessible as far as I can tell (I can log into it, access network resources, etc). Any other suggestions? Any way I can turn up the logging of sshd running in service mode?

[manojampalam]

Adding @NoMoreFood to this thread.

Has it been working on the previous version. We had user name resolution changes in 7.7.0.0 and I want to rule out if they would factor in.

See https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps for info on how to generate traces.

[TCS-Taylor]

@manojampalam , This is a clean install and my first attempt at using SSH on windows. I'm putting together a proof of concept to show my team, we are planning on using SSH to help us manage our windows clients easier.

I will go thru the troubleshooting steps right now. Thanks!

[manojampalam]

Sure. It should be very easy to try out v7.6.1.0, just download in a different folder and run install-sshd.ps1 from in there. You can easy switch between the versions by re-running install-sshd.ps1 from the respective versions.

[NoMoreFood]

Does DOMAINNAME\root have a user principal name in Active Directory? And the machine you're running it on is domain-joined?

The most useful diagnostic step will be getting the log with DEBUG3 enabled. If the new algorithm were to fail it would be because TranslateNameW fails for whatever reason.

[TCS-Taylor]

I have tried using v7.6.1.0p1, v7.7.0.0p1 and 7.7.1.0p1, all with the same results (unable to ssh in, and the same three events in event vwr).

I have attached the DEBUG3 logs of my attempt with each version. Lines 228 and 231 of v7.7.0 and 7.7.1 are of interest to me. Thinking that might point us in the right direction...

@NoMoreFood - yes, tcsbasys\root does have a UPN in active directory and the machine i'm running sshd on is domain joined to the tcsbasys domain.

sshd-TEST-LOG-v7.6.1.0p1.txt
sshd-TEST-LOG-v7.7.0.0p1.txt
sshd-TEST-LOG-v7.7.1.0p1.txt

[TCS-Taylor]

I should also note that our Active Directory isn't technically Microsoft. We are running Samba Version 4.7.3-Debian, which emulates a windows 2008r2 environment.

[manojampalam]

Ok. This does look specific to your environment. Can you add more details on your AD setup?

[TCS-Taylor]

I followed these instructions for setting up our Samba4 AD environment.

https://github.com/thctlo/samba4/tree/master/howtos

Samba4 emulates 2008r2 AD VERY well. All AD maintenance happens thru Microsoft's built in tools (IE, RSAT, Active directory users and computers, etc).

[TCS-Taylor]

I just built a clean Microsoft Windows 2008r2 domain test environment, with all the same settings as our SAMBA4 domain environment (usernames, passwords, forest name, etc) and tested OpenSSH 7.7.0.0p1 public key authentication using a clean client joined to the Windows domain, and it worked great. So it has to be something with the SAMBA4 setup. Next step for me is to run some packet captures against the working ssh server on the test domain, as well as the not-working ssh server on the production domain and then compare the packet captures...

[NoMoreFood]

@TCS-Taylor I suspect, at least in the new version (v7.7.1.0p1), the problem is the inability to discover the UPN for the user. This may be something related to SAMBA not properly servicing LDAP searches that the Windows TranslateName() function uses. You should see this in your packet capture; it must be failing for some reason. The error code (1332) indicates that Windows cannot get the UPN of the user by doing a lookup using the NetBiosDomain\SamAccountName.

For the old version (v7.6.1.0p1) it may actually work if you use the UserPrincipalName as your login ID. Please give this a try. This will tell us whether Kerberos S4U works properly, in general, with SAMBA.

[TCS-Taylor]

Alright, I captured packets from attempts to SSH in using v7.6.1.0p1 (both using username only AND UPN), v7.7.0.0p1 (using username) and v7.7.1.0p1 (using username). Below are the packet captures.

OpenSSH_v7.6.1.0p1-SAMBA4-PCAP-justusername.zip
OpenSSH_v7.6.1.0p1-SAMBA4-PCAP-upn.zip
OpenSSH_v7.7.0.0p1-SAMBA4-PCAP.zip
OpenSSH_v7.7.1.0p1-SAMBA4-PCAP.zip

[TCS-Taylor]

Thanks again for helping with this folks! If we can ferret out the problem that would be FANTASTIC! If it does end up being an issue on the SAMBA4 side, maybe if we let them know they can fix it.

[NoMoreFood]

So the PREAUTH errors are "normal" and can be ignored. Concentrating on the 7.6.1.0p1 code which removes the added complication of a name lookup, the only thing I see is a potential disconnect on Kerberos Encryption key types (i.e. MD5 vice AES256). The other immediate thing I question is whether SAMBA actually supports Kerberos S4U. I'll try to replicate your setup, time permitting. While I carve out time to do that, can you let me know the following:

• Specific version of SAMBA.
• The SAMBA domain functional level.
• The settings within your krb5.conf file.
• The value of msds-supportedencryptiontypes on the computer object, if set.
• The user acocunts AES128/256 etype settings.

If you're unfamiliar with the latter two, see this: https://blogs.msdn.microsoft.com/openspecification/2011/05/30/windows-configurations-for-kerberos-supported-encryption-type/

[NoMoreFood]

Alright, I got Ubuntu running with Samba. TranslateName() does not work properly (it seems to work for anything other than acquiring UPN). More importantly though, I receive a STATUS_NOT_SUPPORTED when attempting to acquire Kerberos S4U token. So it looks both may be due to feature limitations in SAMBA.

Update: Latest version of SAMBA does actually support TranslateName() properly, but S4U still returns STATUS_NOT_SUPPORTED. Lack of S4U support is documented so I think this is going to be a non-starter.

[jamieshaw]

I'm seeing this issue also; fresh install of Windows Server 2016 with Windows Server Essentials configured on it.

Happy to assist in any way that I can but I'm primary a Mac user so not sure where to start digging into the logs. Same issues as OP. Cannot log in via Public Key but can via password; when starting SSHD from the command line, it works fine.

Setup was working fine before configuring Windows Server Essentials so I'm assuming Active Directory / Domain binding is causing the issue somewhere.

[NoMoreFood]

@jamieshaw Please attached DEBUG3 logs when running as a service. Also please make sure the user has a user principal name defined in Active Directory.

[wvoyek]

I'm having the same issue. I can sign-in using a password but the connection gets reset when I use my key. I have attached a log of the attempt.

My domain account does have a proper userPrincipalName. The Domain Controller is Windows Server 2012 R2. The domain functional level is 2012 R2 and the forest functional level is 2008 R2.

sshd_log.txt

[NoMoreFood]

@wvoyek So the error you are seeing is a different breed from the others. 0xC000006D = STATUS_LOGON_FAILURE (From Docs: "The logon attempt failed. The reason for the failure is not specified, but typical reasons include misspelled user names and misspelled passwords."). Since you got this far, I doubt it's a misspelled username but I don't see any UPN lookup error, so I assume that translation is fine. That account works fine with username and password?

[wvoyek]

@NoMoreFood Yes, username/password works fine.

[NoMoreFood]

@wvoyek Weird. Do you happen to see any Security logon events like 4625 around the time when the error occurs? Anything particularly weird about the account or your setup that you think could be impacting it? I'm close to creating a Kerberos S4U based on all the random issues than can occur when doing impersonation on a variety of platforms.

[wvoyek]

@NoMoreFood Nope, no 4625 but a 4624 event. I've attached the event log entries that occurred during the login event in the sshd_log.
sshd_log.xml.txt

edited to add:

Nothing weird about the account or the setup either. Or at least I don't think so. It's my regular daily use account I'm using. I'm trying to sign-in from an Ubuntu guest running on the host that's running sshd. And it does work fine w/ user/pass.

[NoMoreFood]

@wvoyek How about on the domain controller?

[wvoyek]

@NoMoreFood Nothing additional there. Just the same 4624 event.

[pldmgg]

Just noticed this thread after creating a duplicate issue a couple minutes ago (#1213).

I'm having the same issue. Here's all of my log information.

https://gist.github.com/pldmgg/edfb798404302ea73ba5c8e42f6fb591

@manojampalam You asked @TCS-Taylor if this worked correctly in a previous version...The last time I tried this (i.e. Public Cert Auth) was with 7.6.0.0, and it seemed to work fine. I'm not able to re-test with 7.6.0.0 at this point in time though.

[infokiller]

When sshd is running interactively (to be used only for testing/diagnostics), the authentication flow is different - it doesn't invoke S4U token generation logic that needs access to domain.

Is there a way to make the service use the same auth flow as if it was run interactively?
I'm asking because I have the same issue as OP and looking for a workaround.

[bagajjal]

Please check if "users" on the machine has access to sshd.exe

[infokiller]

Thanks @bagajjal but unfortunately users has access...
I think this is expected: like OP password auth works fine, only key auth fails.

[maertendMSFT]

Cannot Repro

[cattrace]

switch to openssh x64 version works for me .
env: win7 x64

[infokiller]

@maertendMSFT can you please respond to #1177 (comment) ?

[bagajjal]

@infokiller - Unfortunately we can't do it. when sshd service is running in interactive mode it runs in the current user context. It can't generate the token for other users as the user will not have enough privileges. sshd interactive is meant for debugging purpose only.

What is the error message you are seeing in the sshd.log file? fyi, we don't support logging in with AAD (Azure Active Directory) credentials. There is work in progress.

[infokiller]

That indeed may be the issue, I think I'm logging in with AAD creds (how do I verify this?).

[winest]

I got exactly the same issue with the fresh installed OpenSSH_for_Windows_8.0p1, LibreSSL 2.6.5.

The solutions works for me is to connect to VPN on my Windows(the machine that runs sshd) so username can be resolved. Hope this bug can be fixed as I don't think it's reasonable for user who already can be verified by public key authentication still need to check name of user by connecting to AD, that's too annoying for me. (I use ssh in WSL2 to connect to local host Windows to run some ansible scripts)

Problem description:
If I started with sshd.exe -d, I can connect with public key authentication.
However, if I started it as service by PowerSell: Start-Service sshd, my connection will be reset
debug1: Found key in /home/username/.ssh/known_hosts:30 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after 134217728 blocks debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521> debug1: SSH2_MSG_SERVICE_ACCEPT received Connection reset by 172.20.16.1 port 22

And from windows event log
sshd: error: lookup_principal_name: User principal name lookup failed for user 'company_domain\username' (explicit: 1355, implicit: 1355)
sshd: error: get_user_token - unable to generate token on 2nd attempt for user company_domain\username
sshd: fatal: ga_init, unable to resolve user company_domain\username

[bagajjal]

@winest - error 1355 means "The specified domain either does not exist or could not be contacted."
OpenSSH first check the incoming user is valid or not before proceeding to the authentication. If user is not resolved then it fails the authentication. This is the same behavior with *nix OpenSSH.

[ghost]

@bagajjal I have the same issue on a Windows 2016 installation. I don't use an AD, it's a standalone VM. On installation, after starting the service I'm able to connect, but after a reboot I get sshd: error: lookup_principal_name: User principal name lookup failed for user myHostname\\Administrator. I'm 'fighting' with this problem for at least 6 months without resolve. The same installation on Windows 2012 Server works with no issues.

[dharrop]

OpenSSH for windows appears to be authenticating using RC4_HMAC. A recent change in my environment to remove that cipher using group policy "Local Policies\Security Options\Network security: Configure encryption types allowed for Kerberos" broke public/private key authentication exactly as this bug report describes. Keyboard interactive continued to work fine. Given that Microsoft is in the process of discontinuing RC4_HMAC it appears that OpenSSH for windows needs to be retrofitted for AES when doing public/private key authentication. For the meantime, ensuring RC4_HAMC is allowed via the aforementioned policy key corrected my problem. This seems to be the case for cygwin's OpenSSH as well - FYI