[BUG] KeyError: 'file' with setuptools 60.0.0 #2941

LuisHenri · 2021-12-20T14:35:43Z

setuptools version

setuptools==60.0.0

Python version

Python 3.8.12

OS

Windows 10

Additional environment information

I use Anaconda with Python 3.8.12.
I use PyArmor 7.0.3.
I user PyInstaller 4.7.

Description

I use PyArmor to pack my project (Obfuscation + Executable with PyInstaller).
It is used inside a Conda Environment.
I have a script that calls PyArmor.

    call_pyarmor(
        [
            "pack",
            "-s",
            "./main.spec",
            "-x",
            " --exclude .venv*"
            " --enable-suffix"
            " --with-license outer"
            " --plugin pyarmor_plugins/plugin_example.py",
            "--clean",
            "./main.py",
        ]
    )

Expected behavior

To the project's Executable be properly generated.

How to Reproduce

Set the environment:

conda create --yes --quiet --name .venv python=3.8
call activate .venv
python -m pip install --upgrade pip setuptools wheel
python -m pip install pyarmor==7.0.3 pyinstaller[encryption]==4.7

Create a sample default project (with main.spec and a plugin)
Run PyArmor from a script:

from PyInstaller.__main__ import run as call_pyinstaller
call_pyarmor(
        [
            "pack",
            "-s",
            "./main.spec",
            "-x",
            " --exclude .venv*"
            " --enable-suffix"
            " --with-license outer"
            " --plugin pyarmor_plugins/plugin_example.py",
            "--clean",
            "./main.py",
        ]
)

Output

2021-12-20T12:51:26.2832859Z INFO     PyArmor Version 7.0.3
2021-12-20T12:51:26.2833655Z INFO     Python 3.8.12
2021-12-20T12:51:26.2837167Z INFO     Target platforms: Native
2021-12-20T12:51:26.2837966Z INFO     Source path is "D:\a\1\s"
2021-12-20T12:51:26.2838558Z INFO     Entry scripts are ['.\\main.py']
2021-12-20T12:51:26.2839403Z INFO     Use cached capsule C:\Users\VssAdministrator\.pyarmor\.pyarmor_capsule.zip
2021-12-20T12:51:26.2840362Z INFO     Search scripts mode: Recursive
2021-12-20T12:51:26.2841345Z INFO     Exclude path "dist"
2021-12-20T12:51:26.2844968Z INFO     Exclude path ".venv*"
2021-12-20T12:51:26.2851366Z INFO     Auto exclude output path "dist\obf"
2021-12-20T12:51:26.3066863Z no previously-included directories found matching 'dist'
2021-12-20T12:51:26.3073564Z no previously-included directories found matching '.venv*'
2021-12-20T12:51:26.3115156Z no previously-included directories found matching 'dist\obf'
2021-12-20T12:51:26.3119135Z INFO     Save obfuscated scripts to "dist\obf"
2021-12-20T12:51:26.3120364Z INFO     Read product key from capsule
2021-12-20T12:51:26.3151411Z INFO     Obfuscate module mode is 2
2021-12-20T12:51:26.3152052Z INFO     Obfuscate code mode is 1
2021-12-20T12:51:26.3152539Z INFO     Wrap mode is 1
2021-12-20T12:51:26.3153215Z INFO     Restrict mode is 1
2021-12-20T12:51:26.3153770Z INFO     Advanced value is 0
2021-12-20T12:51:26.3155708Z INFO     Super mode is False
2021-12-20T12:51:26.3156579Z INFO     Super plus mode is not enabled
2021-12-20T12:51:26.3163577Z INFO     Generating runtime files to dist\obf
2021-12-20T12:51:26.3169121Z INFO     Extract pytransform.key
2021-12-20T12:51:26.3171851Z INFO     Use outer license file
2021-12-20T12:51:26.3174756Z INFO     Copying C:\Miniconda\envs\.venv\lib\site-packages\pyarmor\platforms\windows\x86_64\_pytransform.dll
2021-12-20T12:51:26.3175495Z INFO     Rename it to _pytransform_vax_0001.dll
2021-12-20T12:51:26.3210551Z INFO     Patch library dist\obf\_pytransform_vax_0001.dll
2021-12-20T12:51:26.6123821Z INFO     Patch library file OK
2021-12-20T12:51:27.2509613Z INFO     Copying C:\Miniconda\envs\.venv\lib\site-packages\pyarmor\pytransform.py
2021-12-20T12:51:27.2511236Z INFO     Rename it to pytransform_vax_0001.py
2021-12-20T12:51:27.2521723Z INFO     Generate runtime files OK
2021-12-20T12:51:27.2527229Z INFO     Start obfuscating the scripts...
2021-12-20T12:51:27.7791265Z INFO     Found plugin check_processor_id at: pyarmor_plugins/example_plugin.py
2021-12-20T12:51:27.7800488Z INFO     Patch this script with plugins
2021-12-20T12:51:27.7803068Z INFO     Apply plugin example_plugin
2021-12-20T12:51:27.7807320Z INFO     Patch this entry script with protection code
2021-12-20T12:51:27.8445234Z INFO     Insert bootstrap code to entry script dist\obf\main.py
2021-12-20T12:51:27.8456836Z INFO     Obfuscate 9 scripts OK.
2021-12-20T12:51:30.6824262Z 2464 INFO: PyInstaller: 4.7
2021-12-20T12:51:30.6825312Z 2464 INFO: Python: 3.8.12 (conda)
2021-12-20T12:51:30.6996302Z 2483 INFO: Platform: Windows-10-10.0.17763-SP0
2021-12-20T12:51:30.7045110Z 2488 INFO: UPX is not available.
2021-12-20T12:51:30.7052400Z 2490 INFO: Removing temporary files and cleaning cache in C:\Users\VssAdministrator\AppData\Local\pyinstaller
2021-12-20T12:51:30.7111993Z Traceback (most recent call last):
2021-12-20T12:51:30.7112816Z   File "C:\Miniconda\envs\.venv\lib\runpy.py", line 194, in _run_module_as_main
2021-12-20T12:51:30.7113603Z     return _run_code(code, main_globals, None,
2021-12-20T12:51:30.7114264Z   File "C:\Miniconda\envs\.venv\lib\runpy.py", line 87, in _run_code
2021-12-20T12:51:30.7114913Z     exec(code, run_globals)
2021-12-20T12:51:30.7115338Z   File "C:\Miniconda\envs\.venv\lib\site-packages\PyInstaller\__main__.py", line 134, in <module>
2021-12-20T12:51:30.7115776Z     run()
2021-12-20T12:51:30.7118603Z   File "C:\Miniconda\envs\.venv\lib\site-packages\PyInstaller\__main__.py", line 124, in run
2021-12-20T12:51:30.7119564Z     run_build(pyi_config, spec_file, **vars(args))
2021-12-20T12:51:30.7120046Z   File "C:\Miniconda\envs\.venv\lib\site-packages\PyInstaller\__main__.py", line 58, in run_build
2021-12-20T12:51:30.7120607Z     PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
2021-12-20T12:51:30.7121099Z   File "C:\Miniconda\envs\.venv\lib\site-packages\PyInstaller\building\build_main.py", line 782, in main
2021-12-20T12:51:30.7122155Z     build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
2021-12-20T12:51:30.7123140Z   File "C:\Miniconda\envs\.venv\lib\site-packages\PyInstaller\building\build_main.py", line 714, in build
2021-12-20T12:51:30.7124574Z     exec(code, spec_namespace)
2021-12-20T12:51:30.7124905Z   File "main-patched.spec", line 5, in <module>
2021-12-20T12:51:30.7125488Z     from pip._internal.operations import freeze
2021-12-20T12:51:30.7126946Z   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2021-12-20T12:51:30.7128349Z   File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
2021-12-20T12:51:30.7128859Z   File "<frozen importlib._bootstrap>", line 914, in _find_spec
2021-12-20T12:51:30.7129415Z   File "C:\Miniconda\envs\.venv\lib\site-packages\_distutils_hack\__init__.py", line 83, in find_spec
2021-12-20T12:51:30.7129816Z     return method()
2021-12-20T12:51:30.7130701Z   File "C:\Miniconda\envs\.venv\lib\site-packages\_distutils_hack\__init__.py", line 104, in spec_for_pip
2021-12-20T12:51:30.7131192Z     if self.pip_imported_during_build():
2021-12-20T12:51:30.7131784Z   File "C:\Miniconda\envs\.venv\lib\site-packages\_distutils_hack\__init__.py", line 115, in pip_imported_during_build
2021-12-20T12:51:30.7132215Z     return any(
2021-12-20T12:51:30.7132809Z   File "C:\Miniconda\envs\.venv\lib\site-packages\_distutils_hack\__init__.py", line 116, in <genexpr>
2021-12-20T12:51:30.7133491Z     frame.f_globals['__file__'].endswith('setup.py')
2021-12-20T12:51:30.7133835Z KeyError: '__file__'
2021-12-20T12:51:30.8726971Z Traceback (most recent call last):
2021-12-20T12:51:30.8759812Z   File "deployment.py", line 70, in <module>
2021-12-20T12:51:30.8761995Z     main(sys.argv)
2021-12-20T12:51:30.8765357Z   File "deployment.py", line 27, in main
2021-12-20T12:51:30.8766912Z     run_pyarmor()
2021-12-20T12:51:30.8768238Z   File "deployment.py", line 40, in run_pyarmor
2021-12-20T12:51:30.8768744Z     call_pyarmor(
2021-12-20T12:51:30.8769402Z   File "C:\Miniconda\envs\.venv\lib\site-packages\pyarmor\pyarmor.py", line 1572, in main
2021-12-20T12:51:30.8769991Z     args.func(args)
2021-12-20T12:51:30.8770836Z   File "C:\Miniconda\envs\.venv\lib\site-packages\pyarmor\packer.py", line 540, in packer
2021-12-20T12:51:30.8771583Z     _pyinstaller(src, entry, output, extra_options, xoptions, args)
2021-12-20T12:51:30.8772362Z   File "C:\Miniconda\envs\.venv\lib\site-packages\pyarmor\packer.py", line 440, in _pyinstaller
2021-12-20T12:51:30.8773051Z     run_command([sys.executable] + packcmd + ['-y', '--clean', patched_spec])
2021-12-20T12:51:30.8773787Z   File "C:\Miniconda\envs\.venv\lib\site-packages\pyarmor\packer.py", line 86, in run_command
2021-12-20T12:51:30.8774419Z     raise RuntimeError('Run command failed')
2021-12-20T12:51:30.8774985Z RuntimeError: Run command failed

Code of Conduct

I agree to follow the PSF Code of Conduct

ogrisel · 2021-12-20T14:42:59Z

It seems that we have observed a similar crash on our CI: https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=36235&view=logs&jobId=aabdcdc3-bb64-5414-b357-ed024fe8659e&j=aabdcdc3-bb64-5414-b357-ed024fe8659e&t=08761b20-68b0-57e8-3399-a73e66658a1f

Traceback:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/io/sklearn/utils/_show_versions.py", line 85, in show_versions
    deps_info = _get_deps_info()
  File "/io/sklearn/utils/_show_versions.py", line 69, in _get_deps_info
    mod = importlib.import_module(modname)
  File "/io/testvenv/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 963, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 906, in _find_spec
  File "/io/testvenv/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 83, in find_spec
    return method()
  File "/io/testvenv/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 104, in spec_for_pip
    if self.pip_imported_during_build():
  File "/io/testvenv/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 117, in pip_imported_during_build
    for frame, line in traceback.walk_stack(None)
  File "/io/testvenv/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 117, in <genexpr>
    for frame, line in traceback.walk_stack(None)
KeyError: '__file__'

lesteve · 2021-12-20T15:22:58Z

A simple snippet reproducing the problem:

if [ -d /tmp/test-venv ]; then
    rm -rf /tmp/test-venv
fi
python -m venv /tmp/test-venv
source /tmp/test-venv/bin/activate
pip install 'setuptools>=60'

pip list
python -c 'import importlib; importlib.import_module("pip")'

A few comments:

this fails with setuptools 60.0.0 with a similar error as shown above but succeeds with setuptools 59.8

Full stack-trace

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/local/lesteve/miniconda3/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 982, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 925, in _find_spec
  File "/tmp/test-venv/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 83, in find_spec
    return method()
  File "/tmp/test-venv/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 104, in spec_for_pip
    if self.pip_imported_during_build():
  File "/tmp/test-venv/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 115, in pip_imported_during_build
    return any(
  File "/tmp/test-venv/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 116, in <genexpr>
    frame.f_globals['__file__'].endswith('setup.py')
KeyError: '__file__'

this is only a problem with python -c, executing a .py file (with the same content) works fine

AdmiralNemo · 2021-12-20T15:27:20Z

This breaks python -m ensurepip:

python -m venv test-venv
test-venv/bin/python -m pip install 'setuptools>=60'
test-venv/bin/python -m ensurepip

Traceback (most recent call last):
  File "<string>", line 6, in <module>
  File "/usr/lib64/python3.9/runpy.py", line 206, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/usr/lib64/python3.9/runpy.py", line 130, in _get_module_details
    spec = importlib.util.find_spec(mod_name)
  File "/usr/lib64/python3.9/importlib/util.py", line 103, in find_spec
    return _find_spec(fullname, parent_path)
  File "<frozen importlib._bootstrap>", line 925, in _find_spec
  File "/home/dhatch/test-venv/lib64/python3.9/site-packages/_distutils_hack/__init__.py", line 83, in find_spec
    return method()
  File "/home/dhatch/test-venv/lib64/python3.9/site-packages/_distutils_hack/__init__.py", line 104, in spec_for_pip
    if self.pip_imported_during_build():
  File "/home/dhatch/test-venv/lib64/python3.9/site-packages/_distutils_hack/__init__.py", line 115, in pip_imported_during_build
    return any(
  File "/home/dhatch/test-venv/lib64/python3.9/site-packages/_distutils_hack/__init__.py", line 116, in <genexpr>
    frame.f_globals['__file__'].endswith('setup.py')
KeyError: '__file__'
Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.9/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/usr/lib64/python3.9/ensurepip/__init__.py", line 221, in _main
    return _bootstrap(
  File "/usr/lib64/python3.9/ensurepip/__init__.py", line 140, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/usr/lib64/python3.9/ensurepip/__init__.py", line 52, in _run_pip
    return subprocess.run([sys.executable, "-c", code], check=True).returncode
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/dhatch/test-venv/bin/python', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpijdjrgp5/setuptools-53.0.0-py3-none-any.whl\', \'/tmp/tmpijdjrgp5/pip-21.0.1-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpijdjrgp5\', \'setuptools\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.

ironstar94 · 2021-12-20T15:48:46Z

The bug is occuring because setuptools bundled distutils that's breaking stuff. It was apparently only tested on Fedora #2896

Temporary workaround is setting
SETUPTOOLS_USE_DISTUTILS=stdlib in your environment.

mariocj89 · 2021-12-20T15:50:45Z

Changing to ignore __file__ if it is not present does seems to indeed fix ensurepip.
If the code there is just to check if this is being invoked from a setup.py it should be safe to just do something like:
frame.f_globals.get('__file__', '').endswith('setup.py')

Happy to send a PR.

jaraco · 2021-12-21T00:52:35Z

This issue is pretty much a duplicate of #2940.

lesteve · 2021-12-21T07:26:35Z

Thanks, I can confirm 60.0.3 fixes the problem for the snippet I posted above.

LuisHenri · 2021-12-21T07:46:26Z

LuisHenri added bug Needs Triage Issues that need to be evaluated for severity and status. labels Dec 20, 2021

lesteve mentioned this issue Dec 20, 2021

setuptools v60.0.0 conda-forge/setuptools-feedstock#220

Closed

3 tasks

ogrisel mentioned this issue Dec 20, 2021

ENH Loss module LogisticRegression scikit-learn/scikit-learn#21808

Merged

glemaitre mentioned this issue Dec 20, 2021

CI temporary force setuptools to use distutils from stdlib scikit-learn/scikit-learn#22028

Merged

pfmoore mentioned this issue Dec 20, 2021

updating setuptools causes pip to no longer import as a module successfully pypa/pip#10737

Closed

1 task

phofl mentioned this issue Dec 20, 2021

CI: Builds failing in pandas.show_versions() pandas-dev/pandas#44980

Closed

mroeschke mentioned this issue Dec 21, 2021

CI: Pin setuptools for python dev pandas-dev/pandas#44989

Merged

1 task

jaraco closed this as completed Dec 21, 2021

jaraco added duplicate and removed Needs Triage Issues that need to be evaluated for severity and status. labels Dec 21, 2021

This was referenced Dec 21, 2021

Revert temporary 60.0.0 setuptools work-around scikit-learn/scikit-learn#22036

Merged

setuptools v60.0.3 conda-forge/setuptools-feedstock#222

Merged

setuptools v60.0.1 conda-forge/setuptools-feedstock#221

Closed

jreback mentioned this issue Dec 21, 2021

refactor: use nix to manage development dependencies ibis-project/ibis#2924

Merged

jamesls mentioned this issue Dec 28, 2021

"chalice package" fails in download_all_dependencies() aws/chalice#1857

Closed

petrushy mentioned this issue Jan 24, 2022

JCC 3.11 update conda-forge/jcc-feedstock#56

Merged

5 tasks

petrushy mentioned this issue Feb 6, 2022

Fails with dll not found on win when python is higher than 3.9.7 conda-forge/jcc-feedstock#57

Closed

1 task

akesandgren mentioned this issue Jul 6, 2022

{devel}[system/system] ReFrame v3.12.0 easybuilders/easybuild-easyconfigs#15814

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] KeyError: 'file' with setuptools 60.0.0 #2941

[BUG] KeyError: 'file' with setuptools 60.0.0 #2941

LuisHenri commented Dec 20, 2021 •

edited

Loading

ogrisel commented Dec 20, 2021 •

edited

Loading

lesteve commented Dec 20, 2021 •

edited

Loading

AdmiralNemo commented Dec 20, 2021

ironstar94 commented Dec 20, 2021

mariocj89 commented Dec 20, 2021 •

edited

Loading

jaraco commented Dec 21, 2021

lesteve commented Dec 21, 2021

LuisHenri commented Dec 21, 2021

[BUG] KeyError: '__file__' with setuptools 60.0.0 #2941

[BUG] KeyError: '__file__' with setuptools 60.0.0 #2941

Comments

LuisHenri commented Dec 20, 2021 • edited Loading

setuptools version

Python version

OS

Additional environment information

Description

Expected behavior

How to Reproduce

Output

Code of Conduct

ogrisel commented Dec 20, 2021 • edited Loading

lesteve commented Dec 20, 2021 • edited Loading

AdmiralNemo commented Dec 20, 2021

ironstar94 commented Dec 20, 2021

mariocj89 commented Dec 20, 2021 • edited Loading

jaraco commented Dec 21, 2021

lesteve commented Dec 21, 2021

LuisHenri commented Dec 21, 2021

[BUG] KeyError: 'file' with setuptools 60.0.0 #2941

[BUG] KeyError: 'file' with setuptools 60.0.0 #2941

LuisHenri commented Dec 20, 2021 •

edited

Loading

ogrisel commented Dec 20, 2021 •

edited

Loading

lesteve commented Dec 20, 2021 •

edited

Loading

mariocj89 commented Dec 20, 2021 •

edited

Loading