Can not use 'python -m my_module' // Importing python packages with *.pth-import fails

[monzelr]

Describe the bug

I have just tested the latest release v0.3.12 which also implements #782 (Include a stub app in Windows projects).

Since this stub app I have two problems after releasing my app with briefcase:

1. Python's python38._pth file does not contain app and app_packages, thus I can not start another python instance with python -m my_module in my application. I am using a lot of python instances to run the software on different CPUs. I am wondering how my application (*.exe) is finding all the python modules?? In my logs, I see that my app adds the two folders app and app_packages:

PreInitializing Python runtime...
PythonHome: C:\Program Files\SOFTWARE\AutoLucid
PYTHONPATH:
- C:\Program Files\SOFTWARE\AutoLucid\python38.zip
- C:\Program Files\SOFTWARE\AutoLucid
- C:\Program Files\SOFTWARE\AutoLucid\app_packages
- C:\Program Files\SOFTWARE\AutoLucid\app
Configure argc/argv...
Initializing Python runtime...
Running app module: autolucid

2. This is the same issue as in app_packages in the app bundle is not declared among the site packages #669 and Fails if dependencies include .pth files #381:
   Python packages who are using *.pth files are not correct imported. For example the python package pywin32 is not correct imported:

Configure argc/argv...
Initializing Python runtime...
Running app module: autolucid
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "autolucid\gui\evaluationviewer.py", line 29, in autolucid.gui.evaluationviewer
  File "\app_packages\win32com\__init__.py", line 5, in <module>
    import win32api, sys, os
ModuleNotFoundError: No module named 'win32api'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "\app\autolucid\__main__.py", line 1, in <module>
    from .main import main
  File "autolucid\main.py", line 2, in init autolucid.main
  File "autolucid\gui\autolucid.py", line 49, in init autolucid.gui.autolucid
  File "autolucid\gui\evaluationviewer.py", line 46, in init autolucid.gui.evaluationviewer
  File "\app_packages\win32com\__init__.py", line 5, in <module>
    import win32api, sys, os
ModuleNotFoundError: No module named 'win32api'

---------------------------------------------------------------------------
Application quit abnormally (Exit code -6)!

For me it looks like the pywin32 is only partial imported.
Note here, that the python38._pth file does not use import site.
This code here

import site
print(site.USER_SITE)

prints None, thus the user site is not set and the *.pth from python packages are ignored.

Steps to reproduce

1. Create a brief case project with requirement pywin32==303:

[tool.briefcase.app.MyApp.windows]
requires = ["pywin32==303"]

2. In C:\programe files\MyName\MySoftware\ try to launch any module which is installed in app_packages with the command python -m MyModule. This is not working since briefcase version v0.3.9.

Expected behavior

1. Calling python -m my_module is running. (This feature have been eliminated since v0.3.9 -> windows stub app)
2. There is no error due the import of the pywin32 package.

Screenshots

No response

Environment

• Operating System: Windows 10
• Python version: 3.8
• Software versions:
  • Briefcase: 0.3.12

Logs

Additional context

I think I found a solution, but I do not now how to integrate this into briefcase...

I would suggest the following:

1. Add app and app_packages to the python38._pth file again (this is missing since briefcase version 0.3.9).
   With briefcase v0.3.9 my python38._pth is looking like this:

python38.zip
.
..\\app
..\\app_packages

2. Add code in the MySoftware.exe file were the site.USER_SITE variable is set with a path to the folder app_packages. In python it would be this:

import sys
import site
app_packages = ""
for path in sys.path:
    if path.endswith("app_packages"):
        app_packages = path
if app_packages != "":
    site.USER_SITE = app_packages  # correct the 'site-packages' path to 'app_packages' path
    site.main()  # recall site package thus all .pth in 'app_packages' will be called

(Note here, that calling python -m my_module needs to set USER_SITE on his own.)

[freakboy3742]

Thanks for the report. If I'm understanding this report correctly, there's 2 problems being reported here:

1. Invoking python -m module doesn't work; and
2. .pth files aren't being honored.

The first problem is a use case that Briefcase doesn't support. Briefcase doesn't guarantee the existence of a literal python executable - therefore invoking subprocess (or similar) to invoke python won't work. The only executable that is guaranteed to exist is the binary of the app you've started, and that binary doesn't honour the same command line API as python.exe. The fact that a Windows app bundle includes a python.exe is an artefact of how the bundle is produced, and its existence should be considered a bug - and it should be stripped by Briefcase at time of packaging.

This also means that packages like multiprocessing don't work in spawn mode (although they should work in fork and forkserver mode). This is something that needs to be addressed on a per-stub basis; i.e., each app stub needs to be modified to allow multiprocessing to work as expected.

The second problem is literally #381; it's a known issue, and the win32api package in particular is known to be succeptible to this problem. It looks like something akin to the workaround (2) could/should definitely be integrated into the app stubs of each platform.

I've opened beeware/briefcase-windows-app-template#6 and beeware/briefcase-windows-VisualStudio-template#7 to track the stray python.exe/pythow.exe binaries; I've also opened beeware/briefcase-windows-VisualStudio-template#8, beeware/briefcase-macOS-Xcode-template#20 and beeware/briefcase-linux-flatpak-template#10 to formally track the spawn/multiprocessing issue.

On the basis that the underlying problem is either (1) with the template repositories, or (2) an existing known issue, I'm going to close this ticket.