Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Resolving relative dependencies? #200

Open
JohnPeng47 opened this issue Nov 6, 2023 · 7 comments
Open

Resolving relative dependencies? #200

JohnPeng47 opened this issue Nov 6, 2023 · 7 comments

Comments

@JohnPeng47
Copy link

Does Pydeps not resolve relative dependencies? (Aside: are relative dependencies considered malpractise in python?)

Following command:
python -m pydeps --show-deps --debug route.py

Does not show any of the relative dependencies imported like so
from .schema import CreateGraphReq, ... from .service import GraphManager, ....

Cheers (also thanks for making this, especially looking forward to using the intermediate output for a project I'm working on for writing automatic test cases)
@thebjorn
Copy link
Owner

thebjorn commented Nov 6, 2023

Does Pydeps not resolve relative dependencies?

It should (there are tests: https://github.com/thebjorn/pydeps/blob/master/tests/test_relative_imports.py). If you have a testcase I can look into it.

Aside: are relative dependencies considered malpractise in python?

Relative imports are kosher in Python.

@JohnPeng47
Copy link
Author

Hey thanks for the quick response, in the middle of restructuring my Python dir layout, was using really weird pathing with potentially some circular dependencies along the way, will rerun pydeps after and let you know

@JohnPeng47
Copy link
Author

JohnPeng47 commented Nov 9, 2023 •
edited
Loading

Here, I ran my command from the root of this directory:
python -m pydeps --show-deps --max-module-depth 2 src\server\routes\graph\route.py

Here is the file:
https://github.com/JohnPeng47/KongMono/blob/master/KongServer/src/server/routes/graph/route.py

Here is a snippet of the generated deps file:
{ "fastapi": { "bacon": 1, "imported_by": [ "fastapi.applications", "fastapi.param_functions", "fastapi.routing", "route.py" ], "imports": [ "fastapi.applications", "fastapi.background", "fastapi.datastructures", "fastapi.exceptions",

The generated deps file only shows fastapi and networkx, but it misses both my relative imports and the src.KongBot imports

@itaowei
Copy link

itaowei commented Dec 6, 2023 •
edited
Loading

I also have the similar problem and one test case for that issue is this file under the repository matplotlib.

The error message is shown below

Traceback (most recent call last):
..........
  File "/$HOME/anaconda3/envs/AA/lib/python3.11/site-packages/pydeps/py2depgraph.py", line 237, in py2dep
    mf.run_script(dummy.fname)
  File "/$HOME/anaconda3/envs/AA/lib/python3.11/site-packages/pydeps/py2depgraph.py", line 122, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "/$HOME/anaconda3/envs/AA/lib/python3.11/site-packages/pydeps/py2depgraph.py", line 159, in load_module
    module = mf27.ModuleFinder.load_module(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/$HOME/anaconda3/envs/AA/lib/python3.11/site-packages/pydeps/mf27.py", line 153, in load_module
    self.scan_code(co, m)
  File "/$HOME/anaconda3/envs/AA/lib/python3.11/site-packages/pydeps/mf27.py", line 206, in scan_code
    parent = self.determine_parent(m, level=level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/$HOME/anaconda3/envs/AA/lib/python3.11/modulefinder.py", line 188, in determine_parent
    raise ImportError("relative importpath too deep")
ImportError: relative importpath too deep

@thebjorn
Copy link
Owner

thebjorn commented Dec 6, 2023 •
edited
Loading

@itaowei interesting. The error is coming from Python's modulefinder module, and indeed if you run in directly you'll see the same result:

(pydeps200) go|C:\srv\venv\pydeps200\Lib\site-packages> python -m modulefinder matplotlib\__init__.py
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 664, in <module>
    mf = test()
         ^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 657, in test
    mf.run_script(script)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 153, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 436, in scan_code
    parent = self.determine_parent(m, level=level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 188, in determine_parent
    raise ImportError("relative importpath too deep")
ImportError: relative importpath too deep

If I modify the modulefinder.py code to print out what is causing the problem:

    def determine_parent(self, caller, level=-1):
        self.msgin(4, "determine_parent", caller, level)
        if not caller or level == 0:
            self.msgout(4, "determine_parent -> None")
            return None
        pname = caller.__name__
        if level >= 1: # relative import
            if caller.__path__:
                level -= 1
            if level == 0:
                parent = self.modules[pname]
                assert parent is caller
                self.msgout(4, "determine_parent ->", parent)
                return parent
            if pname.count(".") < level:
                raise ImportError(f"relative importpath too deep for {pname}, level {level}")

I get the following output:

(pydeps200) go|C:\srv\venv\pydeps200\Lib\site-packages> python -m modulefinder matplotlib\__init__.py
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 664, in <module>
    mf = test()
         ^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 657, in test
    mf.run_script(script)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 153, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 436, in scan_code
    parent = self.determine_parent(m, level=level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 188, in determine_parent
    raise ImportError(f"relative importpath too deep for {pname}, level {level}")
ImportError: relative importpath too deep for __main__, level 1

Which tells me that it isn't due to excessive directory depth in the source tree...

Looking at the matplotlib/__init__.py file I can see that they're doing some hacks to get around circular imports. I don't have the time to dig into it right now, but my assumption is that they're doing "things" with partially initialized modules that modulefinder can't follow...

ps: running modulefinder with maximum debug options doesn't seem to give much more clarity

[...]
                            load_module -> Module('numpy', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy'])
                        import_module -> Module('numpy', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy'])
                    find_head_package -> (Module('numpy', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy']), '')
                        load_tail Module('numpy', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy']) ''
                    load_tail -> Module('numpy', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy'])
                    import_hook 'packaging.version' Module('__main__', 'matplotlib\\__init__.py') None 0
                        determine_parent Module('__main__', 'matplotlib\\__init__.py') 0
                    determine_parent -> None
                        find_head_package None 'packaging.version'
                            import_module 'packaging' 'packaging' None
                        import_module -> Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging'])
                    find_head_package -> (Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging']), 'version')
                        load_tail Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging']) 'version'
                            import_module 'version' 'packaging.version' Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging'])
                        import_module -> Module('packaging.version', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\version.py')
                    load_tail -> Module('packaging.version', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\version.py')
                    import_hook 'packaging.version' Module('__main__', 'matplotlib\\__init__.py') ['parse'] 0
                        determine_parent Module('__main__', 'matplotlib\\__init__.py') 0
                    determine_parent -> None
                        find_head_package None 'packaging.version'
                            import_module 'packaging' 'packaging' None
                        import_module -> Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging'])
                    find_head_package -> (Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging']), 'version')
                        load_tail Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging']) 'version'
                            import_module 'version' 'packaging.version' Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging'])
                        import_module -> Module('packaging.version', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\version.py')
                    load_tail -> Module('packaging.version', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\version.py')
                        determine_parent Module('__main__', 'matplotlib\\__init__.py') 1
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 664, in <module>
    mf = test()
         ^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 657, in test
    mf.run_script(script)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 153, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 436, in scan_code
    parent = self.determine_parent(m, level=level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 188, in determine_parent
    raise ImportError(f"relative importpath too deep for {pname}, level {level}")
ImportError: relative importpath too deep for __main__, level 1

@thebjorn
Copy link
Owner

thebjorn commented Dec 6, 2023

Hmm... it might actually be numpy..?

(pydeps200) go|C:\srv\venv\pydeps200\Lib\site-packages> python -m modulefinder numpy\__init__.py
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 664, in <module>
    mf = test()
         ^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 657, in test
    mf.run_script(script)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 153, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 436, in scan_code
    parent = self.determine_parent(m, level=level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 188, in determine_parent
    raise ImportError(f"relative importpath too deep for {pname}, level {level}")
ImportError: relative importpath too deep for __main__, level 1

@ZiqianXu
Copy link

Hey @thebjorn, were you able to root cause the issue? I'm having similar issue with relative path import via from . import <moduleName>.
It's for a simple OSS module grpc python file _server.py

python3 -m pydeps grpc/aio/_server.py --show-deps --pylib --no-output --noshow --include-missing
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/pydeps/__main__.py", line 2, in <module>
    pydeps()
  File "/usr/local/lib/python3.8/dist-packages/pydeps/pydeps.py", line 175, in pydeps
    return _pydeps(inp, **_args)
  File "/usr/local/lib/python3.8/dist-packages/pydeps/pydeps.py", line 39, in _pydeps
    dep_graph = py2depgraph.py2dep(trgt, **kw)
  File "/usr/local/lib/python3.8/dist-packages/pydeps/py2depgraph.py", line 237, in py2dep
    mf.run_script(dummy.fname)
  File "/usr/local/lib/python3.8/dist-packages/pydeps/py2depgraph.py", line 122, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "/usr/local/lib/python3.8/dist-packages/pydeps/py2depgraph.py", line 159, in load_module
    module = mf27.ModuleFinder.load_module(
  File "/usr/local/lib/python3.8/dist-packages/pydeps/mf27.py", line 153, in load_module
    self.scan_code(co, m)
  File "/usr/local/lib/python3.8/dist-packages/pydeps/mf27.py", line 206, in scan_code
    parent = self.determine_parent(m, level=level)
  File "/usr/lib/python3.8/modulefinder.py", line 198, in determine_parent
    raise ImportError("relative importpath too deep")
ImportError: relative importpath too deep

Checking my python3.8 source path directory, all file with from . import <moduleName> returns the same type of error from pydeps.

Thanks for your help on debugging this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@thebjorn @JohnPeng47 @itaowei @ZiqianXu