-
-
Notifications
You must be signed in to change notification settings - Fork 30.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
modulefinder chokes on numpy - dereferencing None in spec.loader #84530
Comments
Issue: Running ModuleFinder.run_script() on numpy versions 1.16.1 to 1.18.3 (maybe more) fails with backtrace. See steps to reproduce below. I do not see this problem on earlier versions of python than 3.8 (tested 3.4, 3.5, 3.6 on ubuntu LTSs), but the code has changed around 3.8. The failure comes to this line of modulefinder.py https://github.com/python/cpython/blame/master/Lib/modulefinder.py#L79 if spec.loader.is_package(name):
return None, os.path.dirname(file_path), ("", "", _PKG_DIRECTORY) I can work around it by changing that to check for None if spec.loader is not None and spec.loader.is_package(name):
return None, os.path.dirname(file_path), ("", "", _PKG_DIRECTORY) Environment: Ubuntu 20.04 with default python3, python3-pip Steps to reproduce: # note any nump version I've tried 1.16.1 and greater fails - I included 1.18.3 to be precise for reproduciton $ python3
>>> from modulefinder import ModuleFinder
>>> finder = ModuleFinder()
>>> finder.run_script("test.py")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/modulefinder.py", line 165, in run_script
self.load_module('__main__', fp, pathname, stuff)
... 300 lines of stack elided - see attached fulllog.txt ...
Obviously I can't tell if numpy or modulefinder is the real culprit. Let me know if I can give any more information. |
Ah, namespace packages. :) Yeah, the code is not taking the "spec.loader is None" case into account. I expect the fix would be to add handling of that case a few lines up in the code, right after handling BuiltinImporter and FrozenImporter. Offhand I'm not sure if the "type" should be _PKG_DIRECTORY or some new one just for namespace packages. How does imp.find_module() (on which modulefinder._find_module() is based) respond to namespace packages? [1] https://docs.python.org/3/library/importlib.html#importlib.machinery.ModuleSpec.loader |
Reproduced on Python3.9.0a5+ imp.find_module() simply raised an ImportError in my tests with an implicitly namespaced package (without an __init__.py) About the "type_", I think it should be consistent with _PKG_DIRECTORY, since PEP-420 states the following[1]:
I'd be happy to submit a patch if you think this is alright. |
Turns out using _PKG_DIRECTORY as a type for namespace packages ended up making I've raised a PR with a new type _NSP_DIRECTORY for namespace-directories. |
Anything still left to do that is stalling this? I just got bitten by it when trying to use modulefinder. |
I opened up a new PR that should fix this properly. The root issue was that PathFinder was not setting the loader attribute for namespace packages in the spec, which it should. After fixing that, _find_module just needed to be updated to deal with NamespaceLoader.
Previously:
|
The modulefinder._find_module has an unpatched bug (python/cpython#84530) in py3.8, 3.9, 3.10. Neither imp.find_module, nor the buggy modulefinder._find_module, handles namespace packages... This fixes #130. #140, #141.
So I found another use case where this same error occurs. I recently learnt about
I am really hoping this can be resolved soon-ish. 🤞 |
While this is still a valid bug, I'd recommend you use something like PyInstaller for that use-case, instead of |
Thank you! |
When compiling multi-file projects that contain a namespace pacakge (folder without __init__.py), we hit Python bug python/cpython#84530. This adds a try/except block to catch the error and raise a more helpful error message. Issue: pybricks/support#1602
When compiling multi-file projects that contain a namespace pacakge (folder without __init__.py), we hit Python bug python/cpython#84530. This adds a try/except block to catch the error and raise a more helpful error message. Issue: pybricks/support#1602
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: