CPython2 support for macOS Big Sur

[glandium]

Issue

After creating a virtualenv with python2 from the Command Line Tools, running the python executable from that virtualenv fails:

% python virtualenv.pyz venv   
created virtual environment CPython2.7.16.final.0-64 in 221ms
  creator CPython2macOsFramework(dest=/Users/glandium/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/Users/glandium/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.2, pip==20.2.4, setuptools==44.1.1, wheel==0.35.1
  activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
% venv/bin/python
zsh: killed     venv/bin/python

The problem comes from code signature not matching:

% codesign -v venv/bin/python
venv/bin/python: invalid Info.plist (plist or signature have been modified)
In architecture: x86_64

This doesn't happen with python3.

[glandium]

I actually failed to reproduce on a x86_64 mac, despite the signature not being valid there too. I did originally hit this on an arm64 DTK, and got someone else to reproduce on one of the new M1-based Mac minis, and there appear to be two levels with this:

• one is the codesign issue, which can be worked around by replacing the signature manually with codesign -s - -f
• but once the codesign issue is solved, running still fails and now gives the following in the console log: `exec_mach_imgact: not running binary "python" built against preview arm64e ABI"

And indeed, the executable behind sys.executable is a Universal x86_64/arm64e binary for python, but is a Universal x86_64/arm64 binary for python3, thus why there is no problem with python3.

[glandium]

Editing create/via_global_ref/builtin/cpython/mac_os.py to always fallback to install_name_tool makes it emit a warning about the first half of the issue:

/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/glandium/venv/bin/python (for architecture x86_64)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/glandium/venv/bin/python (for architecture arm64e)

[glandium]

FWIW, I confirmed both are independent:

• After disabling SIP (csrutil disable), the codesign issue doesn't appear (no need to replace the signature), but the arm64e problem still does
• After enabling the arm64e preview abi (sudo nvram boot-args=-arm64e_preview_abi), the virtualenv python works just fine
• After re-enabling SIP (csrutil enable), but keeping the arm64e preview abi, it doesn't work anymore.

[gaborbernat]

Let's spin out the M1 part into #2024, as for the Big Sur one we can follow it here. It's a new OS/platform so virtualenv not working on them is a feature request rather than a bug one. I don't have at the moment access to Big Sur machine, so can't help with it, so PRs welcome.

[glandium]

As mentioned in the second comment, the codesign validity preventing python from running actually doesn't happen on x86_64 based Big Sur, for some reason, although the code signature is invalid.

[pradyunsg]

Any way to check if the certificate/whatever had an expiry date or compatibility info that makes it incompatible with Big Sur?

[glandium]

The problem is virtualenv is editing the executable to change its dependencies to use @executable_path, invalidating the signature.

[gaborbernat]

The problem is without the edit, the executable will not run. Changing that path invalids the signature, but it is the only way we can create a virtual environment. Not sure what else we can do. We could resign it, but for that, you need a key, and we'd not even be the same key, etc. Considering CPython 2 is EOL on macOS, I'd not expect them to offer any solutions for it. @glandium ideas?

[glandium]

codesign -s - -f venv/bin/python works, until it doesn't and says the codesign_allocate helper tool cannot be found or used, which I'm not sure under which circumstances it happens (although it seems to involve Rosetta), but it does:

% python virtualenv.pyz venv      
created virtual environment CPython2.7.16.final.0-64 in 192ms
  creator CPython2macOsFramework(dest=/Users/glandium/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/Users/glandium/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.4, setuptools==44.1.1, wheel==0.35.1
  activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
% arch --x86_64 venv/bin/python   
zsh: killed     arch --x86_64 venv/bin/python
% codesign -s - -f venv/bin/python
venv/bin/python: replacing existing signature
venv/bin/python: the codesign_allocate helper tool cannot be found or used

Another one:

% arch --x86_64 python virtualenv.pyz venv
created virtual environment CPython2.7.16.final.0-64 in 263ms
  creator CPython2macOsFramework(dest=/Users/glandium/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/Users/glandium/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.4, setuptools==44.1.1, wheel==0.35.1
  activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
% venv/bin/python
zsh: killed     venv/bin/python
% codesign -s - -f venv/bin/python        
venv/bin/python: replacing existing signature
venv/bin/python: the codesign_allocate helper tool cannot be found or used

But it doesn't happen here:

% arch --x86_64 python virtualenv.pyz venv
created virtual environment CPython2.7.16.final.0-64 in 263ms
  creator CPython2macOsFramework(dest=/Users/glandium/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/Users/glandium/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.4, setuptools==44.1.1, wheel==0.35.1
  activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
% codesign -s - -f venv/bin/python        
venv/bin/python: replacing existing signature

So I guess if virtualenv itself would do the code signing, it would work™.

But the arm64e problem is unsolvable if virtualenv can't be made to work with symbolic links. That said, after codesign -s - -f venv/bin/python succeeds, arch --x86_64 venv/bin/python does work.

It really sucks that the failure mode is that the process is killed rather than a more useful error message directly on the command line (instead of hidden in the console log).

[gaborbernat]

But the arm64e problem is unsolvable if virtualenv can't be made to work with symbolic links.

It's not that virtualenv can't be made to work with, but rather there's no way to make the cpython 2 interpreter work with symlinks.