Value of `sys.executable` is not always a path to an executable

[BCSharp]

Follow up on one of the issues reported in #1930.

Background

In Python, the value of sys.executable should be the full path to the Python interpreter running the current script. It is being used by scripts to launch another instance of the interpreter if needed, e.g. for the execution in a separate process, or with different command-line arguments. The two notable packages using it are pip and venv. The child interpreter should run by default with the same library and environment as the parent.

Running IronPython on .NET involves two executables:

1. Platform specific but application agnostic runner dotnet/dotnet.exe (name depending on the OS)
2. Platform agnostic but application specific Python interpreter implementation ipy.dll

So the complete command is the combination of the two, e.g. dotnet ipy.dll. However the Python scripts expect sys.executable to be a single path to the executable that can be executed directly by the system. Reporting dotnet /path/to/ipy.dll as sys.executable will result in an error because a file with a path dotnet /path/to/ipy.dll does not exist.

.NET is also capable of producing an application-specific runner ipy/ipy.exe, which contains a hard-coded relative path to ipy.dll it will run. Although the running assembly is ipy.dll, it can report sys.executable as ipy/ipy.exe. In this case, Python scripts can use this runner to launch another instance of the interpreter, by executing a single executable.

The code in ipy.dll has a heuristic of looking for ipy/ipy.exe in the same directory as ipy.dll to use as the value of sys.executable. As a fallback, it also looks for ipy.sh/ipy.bat to use a single executable. The shell scripts themselves (the launchers) contain the logic to run ipy.dll with the dotnet runner. Those scripts are created by Install-IronPython.ps1, when the IronPython is being installed from a zip file.

Module sys also has a few more relevant constants, all of them being paths to directories:

• sys.prefix is the path to the installation root of Python that is currently being executed.
  • On Windows, it is something like C:\Python34 by default, but can be any location as chosen during the installation. This folder contains subfolder Lib with the Python Standard Library installed in it, Scripts with pip.exe, and DLLs with .pyd's.
  • On Linux, it is /usr or /usr/local depending on the installation. This directory is expected to contain lib/pythonX.Y which contains the StdLib. The executable is usually somewhere else, e.g. in /usr/bin or /usr/local/bin.
• sys.exec_prefix is for platform dependent files, which in practice looks like it is always the same as sys.prefix (for platform independent files).

The above description is valid for system installations. It is different in the case of a venv:

• sys.prefix is the path to the venv directory. The venv directory has a different structure than a system installation, the most interesting being:
  • On Windows: Scripts with the executables which are copies of the original; Lib (empty except for the site packages); and file pyvenv.cfg that a.o. contains the path to the directory where the original python executable resides.
  • On other systems: bin with the executables (python is a link to the executable at the system location); lib/pythonX.Y (empty except for the site packages); and file pyvenv.cfg that a.o. contains the path to the directory where the original python executable resides.
• sys.base_prefix contains the value of sys.prefix of the environment from which this venv was created, presumably extracted from pyvenv.cfg.
• Again, sys.exec_prefix is the same as sys.prefix, and sys.base_exec_prefix is the same as sys.base_prefix, adding nothing useful.

Problems when installed as dotnet tool

pip

When IronPython is installed as a dotnet tool, the runner ipy/ipy.exe is created in a different directory than ipy.dll. Specifically, ipy.dll is in subdirectory .store/ironpython.console/3.4.2/ironpython.console/3.4.2/tools/net8.0/any/ relative to the directory containing the runner. Consequently, sys.executable is the full path to ipy.dll, which cannot be executed by the OS directly.

venv

Creating a new venv copies only sys.executable to the new venv, so a number of DLLs are missing. Interestingly, venv on Windows will copy a whole bunch from Scripts as well, but IronPython does not use Scripts so only whatever sys.executable points to is copied.

Approaches

I have considered the following approaches, each of which comes with some caveats.

Expand the search for the runner

Let sys.executable point to the precompiled runner (e.g. ~/.dotnet/tools/ipy) but keep sys.prefix where ipy.dll and lib is.

Since the runner is in a relative position to ipy.dll, it is not (easily) relocatable. In particular, if a venv is created, linking/copying the runner to bin/Scripts will break the runner. Even if the remaining DLLs are linked/copied there as well, the runner will not work. This is a problem for package venv, which only supports CPythons directory structure. However, venv does not work with IronPython for several other reasons as well, and dotnet tool --tool-path serves as an alternative to venv, so this approach seems acceptable for now. Also Install-IronPython.ps1 and Enter-IronPythonEnvironment.ps1 function as reasonable alternatives to venv.

Add launcher scripts

The NuGet package can be expanded to contain ipy.sh and ipy.bat launchers. In such case sys.prefix will be the absolute path of one of them, depending on the OS; the other script would be unused, but since it is hidden deep in the directory structure that is not on PATH, it should not be a problem.

The risks with this approach are always possible incompatibilities of shells on various target systems. Another issue is that on Windows, this will create always two processes: the parent shell process, and the child dotnet process. Besides inefficiency, this may create problems with applications that launch Python programmatically and expect that the handle of their child process is the handle of the actual Python subprocess.

Also, it still does not work with venv: On Windows, only ipy.bat is copied, not the ipy.dll, and it is a non-trivial task to somehow parse pyvenv.cfg in a BAT file to find its way back to where the original DLLs are located. On Linux/macOS, ipy.sh will be a soft link, which, when followed, leads directly to the directory with ipy.dll, so it can easily be launched with dotnet, but following the link breaks the containment of venv and IronPython runs as if it was run from the original environment. Perhaps parsing env:VIRTUAL_ENV can be used to find the way back to the containment (though only if the environment was properly activated using the activation scripts), or an additional argument to ipy can be defined to pass the venv path (e.g. -X VenvPath=...).

Make ipy.dll runnable

The path to ipy.dll is reported as sys.executable if all else fails. There are several ways how an executable can be started from within a Python shell: os.system, os.execve, subprocess.Popen (afaik, other ways are build on top of one of those functions). It is possible to modify the implementation of those functions (os.execve is not implemented yet) to make a DLL "runnable" by silently modifying the command to execute by adding dotnet in front of it.

This should work with subprocesses created directly by a Python script, but will fail if the script passes the value of sys.executable to another program, say, pip3.exe and that program tries to "execute" the DLL.

Also, venv is still broken. It will copy the DLL to bin/Scripts, which is not runnable by itself and still misses a bunch of satellite DLLs.

Conclusions

1. It seems that setting sys.executable to the ipy/ipy.exe runner created by dotnet tool is the preferred solution, at least for dotnet tool installations.

2. To make venv work with IronPython on .NET will require some modifications to that package, regardless of the approach chosen. I think that venv does already work with IronPython on .NET Framework, which is more important than working on .NET, given that there are no alternative mechanisms to easily create separate environments for .NET Framework. I have no experience with venv with IronPython on Mono. However, various tooling (e.g. Visual Studio) depend on venv to create local development environments, so it is still useful to support it in all cases.