Unable to run commandline-based AppX (Store Apps)

[DataGhost]

"OpenSSH for Windows" version
((Get-Item (Get-Command sshd).Source).VersionInfo.FileVersion)
7.7.2.3 (Installed through optional features)
8.1.0.0 (Installed manually)

Server OperatingSystem
((Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows nt\CurrentVersion\" -Name ProductName).ProductName)
Windows 10 Pro (x64, 2004, 10.0.19041.329), freshly-installed from an ISO created with the Media Creation Tool yesterday, fully updated and no other software installed.

Client OperatingSystem
Linux (Kubuntu 20.04)

What is failing
I installed Python 3.8 through the Windows Store, expecting to be able to execute Python scripts remotely through ssh. Everything works fine on the machine itself (directly or over RDP) but the interpreter won't start over ssh. One or more log entries appear in Event Viewer\Applications and Services Logs\Microsoft\Windows\AppModel-Runtime\Admin in either case. I have tried both the OpenSSH server version bundled with Windows, and the latest 8.1.0.0p1-Beta release. I have not had them installed side-by-side.

Expected output
Using cmd over RDP, both in regular and elevated prompts, in powershell as well:

C:\Users\Username>python3
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

or

C:\Users\Username>"c:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1008.0_x64__qbz5n2kfra8p0\python3.8.exe"
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

Event log (excerpts):

Created Desktop AppX container 4c052416-bd27-11ea-9194-000c29ce0202 for package PythonSoftwareFoundation.Python.3.8_3.8.1008.0_x64__qbz5n2kfra8p0.
Event ID: 210
Keywords: DesktopAppXContainer
User: SYSTEM

Added process 6184 to Desktop AppX container 4c052416-bd27-11ea-9194-000c29ce0202 for package PythonSoftwareFoundation.Python.3.8_3.8.1008.0_x64__qbz5n2kfra8p0.
Event ID: 211
Keywords: DesktopAppXContainer
User: SYSTEM

Created process 6184 for application PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0!Python in package PythonSoftwareFoundation.Python.3.8_3.8.1008.0_x64__qbz5n2kfra8p0. [FinishPackageActivation]
Event ID: 201
Keywords: DesktopAppXProcess
User: SYSTEM
EventData:
  ProcessID 6184 
  PackageName PythonSoftwareFoundation.Python.3.8_3.8.1008.0_x64__qbz5n2kfra8p0 
  ImageName python3.8.exe 
  ApplicationName PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0!Python 
  Message [FinishPackageActivation] 

Actual output
Using cmd over ssh:

username@MACHINENAME C:\Users\Username>python3
The system cannot execute the specified program.

or

username@MACHINENAME C:\Users\Username>"c:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1008.0_x64__qbz5n2kf
ra8p0\python3.8.exe"
Access is denied.

Event log (excerpts):

0x5200000: Cannot create the process for package  because an error was encountered. 耇[GetPackageToken]
Event ID: 202
Keywords: DesktopAppXProcess
User: SYSTEM
EventData:
  PackageName  
  ImageName 邅翹 
  ApplicationName L> 
  ErrorCode 85983232 
  Message 耇[GetPackageToken] 

Note the Chinese/garbled characters here as opposed to the cmd over RDP case.

Using powershell over ssh:

PS C:\Users\Username> python3
Program 'python3.exe' failed to run: The file cannot be accessed by the systemAt line:1 char:1
+ python3
+ ~~~~~~~.
At line:1 char:1
+ python3
+ ~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

or

PS C:\Users\Username> & "c:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1008.0_x64__qbz5n2kfra8p0\python3.8.e
xe"
Program 'python3.8.exe' failed to run: Access is deniedAt line:1 char:1
+ & "c:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.
At line:1 char:1
+ & "c:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Event log shows the same as cmd over ssh, but three times (all identical) in the same second instead of once.

Oddity
Maybe interesting is that the Chinese/garbled characters in the Event Log are identical for each log entry in the same boot, even when changing OpenSSH versions (without reboot), restarting OpenSSH or reconnecting to it, also with different user accounts. After a reboot, the characters remain the same in all EventData fields except for ImageName, which gets different characters. Seems to be some uninitialised, though constant, buffer or pointer. None of the fields have garbled characters when starting python successfully (i.e. over RDP).

[WSLUser]

If you want to run python scripts remotely, then you'll need to use fabric. This will still require python to be installed on the Linux host. Also keep in mind the latest fabric is different syntax than older ones but the newer one will still work on older python. Running fabric using win32 openssh does work as I've done it.

[DataGhost]

That is a workaround, just as installing Python from the Python website rather than the Windows Store is a workaround. I've done the latter and everything is working for me now but that doesn't mean there's no bug in the way Win32-OpenSSH interacts with Store apps. I always had the idea that the Store was marketed as an analogue to Linux package managers, though over the last few months I've started to question that given the limitations imposed on and quirks experienced with Store apps.
Curious, though, are you using Python from the Windows Store to run fabric? Because I've got a very strong feeling that it's going to suffer from the same issue.
Cmd over SSH:

username@MACHINENAME C:\Users\Username>fab
Fatal error in launcher: Unable to create process using '"c:\users\username\appdata\local\microsoft\windowsapps\pythonsoftwarefound
ation.python.3.8_qbz5n2kfra8p0\python.exe"  "C:\Users\Username\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfr
a8p0\LocalCache\local-packages\Python38\Scripts\fab.exe" '

Cmd over RDP:

C:\Users\Username>fab
Can't find any collection named 'fabfile'!

Edit: Verified it myself:
fabfile.py:

from fabric import task
@task
def cmdhello(c):
	c.run("cmd /C echo hello")
@task
def pyhello(c):
	c.run("python hello.py")
@task
def wronghello(c):
	c.run("wrongcommand")

>fab -H MACHINENAME --prompt-for-login-password cmdhello
Enter login password for use with SSH auth:
hello"
>fab -H MACHINENAME --prompt-for-login-password pyhello
Enter login password for use with SSH auth:
The system cannot execute the specified program.
>fab -H MACHINENAME --prompt-for-login-password wronghello
Enter login password for use with SSH auth:
'wrongcommand' is not recognized as an internal or external command,
operable program or batch file.

[WSLUser]

That looks to be an issue with your PATH. Yes I'm using Python 3.8 from the Store, running a fabric task that also requires a password prompt due to use of sudo using win32-openssh to execute scripts(can be bash, python, etc.) on a remote Linux server using PS7 in Windows Terminal (but I have tested cmd as well). Be sure to follow the wiki installation for ssh and by the same token, ensure your python Scripts folder is also on PATH. Fabric and a number of other python packages go into the Script directory. CMD or PS rely on PATH in order to execute any number of tasks and is more obvious when a programming language like Python is being used. Additionally, most of the scripts I ran were used on a remote server. I only did the bare minimum on Windows for a connection to be established and execute each of the scripts on multiple hosts from a remote Linux server responsible for hosting those scripts

[DataGhost]

Are you using Python 3.8 from the Store to invoke fab (locally) so that it executes Python code on another machine, which is running some Python not from the Store? Can you test from some machine A with my fabfile to some other machine B (MACHINENAME) which has Python 3.8 from the Store and correctly setup PATH?

I explicitly added wronghello to show that there'd be a different error in case my PATH were wrong. The three test cases show correct execution of normal executables, no execution of python and a different error message for missing executables, respectively. Additionally, I got the same DesktopAppXProcess Error in the Event Log just like in the original bug report.

My test setup for my previous comment was:
Machine A: Fresh Win10 x64, Python 3.8 from Store, pip install fabric pypiwin32, fabfile.py running said commands
Machine B (MACHINENAME): Fresh Win10 x64, OpenSSH server through Features, Python 3.8 from Store, pip install fabric pypiwin32, nothing else.
I connected to the (static) IP of machine B in the -H argument of fab on machine A.

To explicitly clear this up, the Scripts directory (%USERPROFILE%\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\Scripts) has been in my PATH during each test, on both machines.

[DataGhost]

Just an FYI and clarification that I'm going to assume WSLUser is using OpenSSH client from Windows to a Linux box rather than server on Windows from Linux to Windows (my situation) to do this. I have still not been able to run Python from the Windows Store on a Windows machine through OpenSSH-Server.

[SpecLad]

I have encountered a very similar issue, although not with OpenSSH, but with my own service that runs Python from the Windows Store. I believe it's the service part that is important - that is, Windows seems to be prohibiting services from running Windows Store apps.

I tested this by writing a trivial program that just runs Python using CreateProcess. When I run it by itself, it works; when I run it as a Windows service (using WinSW), CreateProcess fails with the error code 1920 (ERROR_CANT_ACCESS_FILE). This produces the same event in the event log as in the original post (with the same Chinese character).

This issue appeared after I updated to Windows 10 version 2004, so I assume that's where the change (bug?) was introduced.

[peter-dolkens]

I can confirm that this is an issue with appx apps running remotely via OpenSSH.

For me, it occurs when I try to run MS Store Installed winget via OpenSSH Server

[brunovieira97]

I installed PowerShell 7 from the Store on my Windows desktop (that basically acts as a server), and I couldn't set it as the default shell. Well, I could set it on the regkey, but trying to connect would give me a "session was closed" or something like that on client-side. Now I see that OpenSSH on the server could not load up pwsh.exe from WindowsApps or even my user folder symlink (store version creates it on install process).

Installing the MSI from GitHub worked. If this is a limitation of Win32-OpenSSH, firstly that should be stated on Wiki or some doc, and then be fixed by Microsoft.

[bagajjal]

@brunovieira97 - I guess the installation path is different when you try MSI vs windows store.
Please provide the ssh server side logs (see below), ssh client side logs (ssh -vvv user@ip)
By default, ssh server logging is ETW based. To change it to file based follow the below instructions,

1. Add below config to c:\programdata\ssh\sshd_config file
   SyslogFacility LOCAL0
   LogLevel DEBUG3
2. net stop sshd
3. net start sshd
4. ssh -vvv user@ip

[pnbruckner]

I have encountered a very similar issue, although not with OpenSSH, but with my own service that runs Python from the Windows Store. I believe it's the service part that is important - that is, Windows seems to be prohibiting services from running Windows Store apps.

I tested this by writing a trivial program that just runs Python using CreateProcess. When I run it by itself, it works; when I run it as a Windows service (using WinSW), CreateProcess fails with the error code 1920 (ERROR_CANT_ACCESS_FILE). This produces the same event in the event log as in the original post (with the same Chinese character).

This issue appeared after I updated to Windows 10 version 2004, so I assume that's where the change (bug?) was introduced.

Hi all. Sorry for intruding on this conversation. I'm not using the same stuff, but I ran into a similar problem where using Python from a service was working fine before I upgraded Windows 10 from 1909 to 20H2. I ran across this topic while searching for the error message "The system cannot execute the specified program."

Although I haven't tried it yet, I think the solution (at least for me) is to uninstall the Python "package" from the Microsoft Store and use the "Full Install" from python.org instead. FYI, see: https://docs.python.org/3/using/windows.html#the-microsoft-store-package

[DataGhost]

As per SpecLad's comment:

I believe it's the service part that is important - that is, Windows seems to be prohibiting services from running Windows Store apps.

What I've managed to find and experience is that there's some sort of sandbox that's transparently built up around any AppX that's executed, and that that, for some reason, does not work when it's ran in the OpenSSH service context. So for whatever reason, probably some obscure security thing, this won't work in the current state and that's a bad design choice IMO if one of the end goals is to have the Microsoft Store as the de facto place to install programs from in the future, because it breaks a lot of interoperability.

It seems that it's the AppX subsystem that's disallowing this so it may need to be addressed there, but it also feels like the OpenSSH service isn't opening a proper login session for the connecting user, because the command is only started after authentication. In any case, none of my attempts at any layers of indirection have succeeded besides running a custom service in the context of the user itself, and that's not a solution I'd personally like to go with. So the AppX subsystem must not like the session I'm logged in to and the permission tokens granted to it, despite the user being on Admin level. I don't have enough knowledge of the inner working of Windows to debug this, nor do I have the time to figure it out.

And yes @pnbruckner, doing a normal install by downloading and installing it manually from the Python website "solves the problem". Since it's not executed from the protected AppX environment, it has none of the artificial restrictions. I got fed up with this and needed to go on with my work, so I went that route. But it's not specific to Python, any AppX seems to suffer from the same problem. So while it is a workaround that allows me to do what I want, it is not (IMO) the best way to solve it because it defeats one of the goals of the MS Store: (aiding in) keeping software up-to-date (possibly automatically) without having to manually go into each package's publisher's website regularly. Despite the "workaround", this bug persists.

[peter-dolkens]

In any case, none of my attempts at any layers of indirection have succeeded besides running a custom service in the context of the user itself, and that's not a solution I'd personally like to go with.

I actually run OpenSSH manually via Task Scheduler in my user account so that I can login with my AzureAD account (not supported with the default service install)

I still get the same error - so it's definitely related to security, but not to services in particular.

[SpecLad]

Isn't Task Scheduler a service? :-)

[peter-dolkens]

Isn't Task Scheduler a service? :-)

Task scheduler isn't the OpenSSH service however, meaning it's not restricted to running OpenSSH as a service, nor is it tied to operating under the Local System account, as I can replicate this as a Task Scheduler task running as an AzureAD account, even with "highest privileges". This is already a different security context, because apparently you can't run login to OpenSSH as an AzureAD user without this workaround.

More info too - running a cmd shell returns

The system cannot execute the specified program

Running a powershell shell returns

ResourceUnavailable: Program 'winget.exe' failed to run: The file cannot be accessed by the system.At line:1 char:1

[glima]

One more issue related to openssh on windows! Yay! :)

[simevo]

This hit me on a CI VM where the jobs where started ssh-ing in. I had Python installed the default way (from the microsoft store), the link in AppData\Local\Microsoft\WindowsApps was there:

dir c:\Users\xxx\AppData\Local\Microsoft\WindowsApps\python.exe 
 Volume in drive C has no label.
 Volume Serial Number is A02B-BD04

 Directory of c:\Users\simevo\AppData\Local\Microsoft\WindowsApps

06/16/2021  08:49 PM                 0 python.exe
               1 File(s)              0 bytes
               0 Dir(s)  13,636,009,984 bytes free

and python would start from the CMD ... but not in the CI job while in a ssh session!

[saschanaz]

Store-distributed cli tools are exposed via a special type of reparsed points, so it needs an explicit support like cygwin/cygwin@e6d1078, libuv/libuv#2812, or python/cpython#15231. I guess SSH needs the same type of work.

[ghost]

The comments are all over the place.

Previously in windows 10, the store was confined to MSIX packages only.
MSIX packages are installed in \program files\WindowsApps directory. You can launch MSIX packaged Cli executables by providing the absolute executable path, eg; C:\program files\Windowsapps\AppFolder\app.exe ( note that each of these node must have read and execute permission for the current user and "NETWORK" group. )

the problem is , MSIX packages are exposed in PATH through App Execution Aliases which itself are undocumented special NTFS reparse points which are stored in ~\appdata\local\microsoft\windowsapps\ directory; these Mysterious special reparse points seem to not launch from OpenSSH. that's the reason you can't launch MSIX packaged Cli tools directly without providing absolute path.

[saschanaz]

App Execution Aliases which itself are undocumented special NTFS reparse points

It's kinda documented, although not well enough:

Used by Universal Windows Platform (UWP) packages to encode information that allows the application to be launched by CreateProcess. Server-side interpretation only, not meaningful over the wire.

I think we'll want to fix somewhere around this function.

https://github.com/PowerShell/openssh-portable/blob/0b73c4636d38f0c69424721d52d0e7752db99c81/contrib/win32/win32compat/fileio.c#L1124

[bagajjal]

@saschanaz - The issue is happening in an interactive SSH session i.e., the code in fileio.c wouldn't get executed.
Including @DHowett from console team to see if it can be handled.

[ghost]

Windowsows Package Manager aka winget is a MSIX packaged tool and can't be run over SSH. this whole time some users including myself have been blaming MSIX for this only to know that the reason is openssh program itself. now it's conhost.exe!
@bagajjal so the whole time the real culprit has been conhost.exe ? 😬

[DHowett]

@mshaikhcool Welcome to bug triage in the open-source world! Since Windows has so many components, we often have to move bugs between a few owners before we can tell what's really going on.

As the owner of conhost: We don't have any control over what processes get to run, but looking at that error report (specifically, that there is an issue in GetPackageToken) it seems like something definitely changed about how packaged applications are launched.

If I had to guess: the MSIX platform requires a different sort of user session than the one SSH provides; perhaps it needs interactive logon, perhaps it needs something else. We'll have to engage with somebody from the MSIX team!

EDIT: This also suggests a possible cause for @SpecLad's service troubles, and is in line with @DataGhost's suspicion in #1632 (comment)

[ghost]

TL;DR: given executable absolute path, if you(user who installed the app and NETWORK group) have read and execute permission on the whole PATH and the executable and it's dependencies, you can launch a packaged Cli app from OpenSSH session regardless of the app.exe in question being a UWP Console app(LowIL/AppContainer app) or a FullTrust app or whether it's using MSIX virtualization or unvirtualizedResources.

it's only App Execution Aliases/special NTFS reparse points where all the trouble begin for OpenSSH.

On a local Interactive login,

1. I install a UWP console app, it has an App execution alias name hen , typing hen on a command line launches fine.
2. I go ~\appdata\local\microsoft\windowsapps and do .\hen.exe , the app launches fine.
3. Given enough permissions, I provide absolute executable path \program files\WindowsApps\HenFolder\hen.exe on a commandline, the app launches fine.
4. I forcefully remove the whole executable and dependencies from \program files\windowsApps\HenFolder, the app execution alias still present, I type hen on a command line (Powershell) throws
   ResourceUnavailable, the system can not access the program error. this means executable is missing which is fine.

on a OpenSSH session,

1. typing hen throws ResourceUnavailable, the system can not access the program error.
2. going ~\appdata\local\microsoft\windowsapps and do .\hen.exe , same above error.
3. I provide absolute executable path \program files\WindowsApps\HenFolder\hen.exe on a commandline, the app launches fine.

Both 1. & 2. step are directly related to App Execution Aliases/special NTFS reparse point. that means OpenSSH is not handling these aliases.

[DHowett]

@mshaikhcool What's actually happening when you launch the executable through its direct path (in Program Files) is that it is being spawned without an app container token. It's not the app execution alias that is broken, it's the fact that the app execution alias forces a packaged launch (where the process receives the correct container token), whereas launching it directly does not.

Before app execution aliases, the only way to execute an application that lived in a package was to explicitly break the package. That's all you're accomplishing this way, too. 😄

Further, launching a "Low IL" or "AppContainer" console application directly out of Program Files runs it with full permissions, ablating any benefit the app container could have provided.

[DHowett]

(Also: modifying the permissions on Program Files\WindowsApps is an excellent path to a system that doesn't work the way it should, as evidenced by all the sub-reports on microsoft/terminal#9452)

[ghost]

the app execution alias forces a packaged launch (where the process receives the correct container token)
Before app execution aliases, the only way to execute an application that lived in a package was to explicitly break the package.

Nice, so mystery unfolded. that means it will require heavy engineering around NT Kernel's AppContainer SID.

'modifying the permissions' purpose was to see if openssh would launch it or not.

[ghost]

@DHowett
TIL that App Execution Aliases are married to Containerization. what happens when packages (like powershell) that use unvirtualizedResources where no virtualization is involved and is launchable from an admin shell and still use Aliases ?

[DHowett]

Ah. Even if they're using unvirtualized resources and are otherwise uncontained, they still get launched with a process token that includes their "package identity." This is necessary to support package ACLs and capabilities, even if the process isn't going to use them.

[ghost]

package identity .. is necessary to support package ACLs and capabilities even if the process isn't going to use them.

Nice. Loving these App Execution Alias Mystery unfoldings. Thanks.

[cpbotha]

I would like to add this datapoint:

I spent quite a few hours last night trying to figure out why trying to execute wsl.exe or bash.exe (from the shiny new store-installed WSL 0.48.2) kept giving me "access is denied" when SSH'd in to a powershell on my Windows 11 22000.346 machine.

When I Stop-Service sshd and then run the sshd binary directly from an elevated powershell prompt, I can SSH in with my normal admin group user account and run wsl.exe without any issues.

Last night I thought this must be due to sshd running as "local system" in the service case, but based on this thread it looks like there might be additional complications due to the interaction between store apps and the openssh service?

There seems to be a known "access is denied" issue with WSL specifically when it runs from local system (i.e. openssh service): microsoft/WSL#4803 -- also note the work-around implemented by tailscale in net/dns to start the child process in a different user context: tailscale/tailscale@6b9f820

[atruskie]

Just to signal boost this: after updates (I think wsl --update but it could have been a Windows update too) I can no longer use WSL/bash.exe from a sshd service session. This has really messed up my workflow.

(ssh cannot runs things it could just one update ago)

I would appreciate any advice or workarounds.

EDIT

My issue is specifically related to this known issue: https://learn.microsoft.com/en-us/windows/wsl/store-release-notes#known-issues.

Fix: uninstall the Windows Store version of WSL.

Related issue: microsoft/WSL#7900

[tgross35]

Has this been resolved? I am able to run WSL and store-installed apps, as documented in microsoft/WSL#7900 (comment)