Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support submodules #1602

Open
TomJuri opened this issue Apr 16, 2024 · 9 comments
Open

[Feature] Support submodules #1602

TomJuri opened this issue Apr 16, 2024 · 9 comments
Labels
enhancement New feature or request software: pybricksdev Issues related to the pybrickdev Python package

Comments

@TomJuri
Copy link

TomJuri commented Apr 16, 2024

Describe the bug
Custom python modules (folders) don't get compiled onto the hub. Everything works fine when I put every python file that gets imported by any other python file into the project root. However, when I try to import a file that's within a directory, this error occurs:

[tom@neon:~/Coding/2425-RobotCode]$ pybricksdev run ble main.py 
Searching for any hub with Pybricks service...
Traceback (most recent call last):
  File "/home/tom/Coding/2425-RobotCode/venv/bin/pybricksdev", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/tom/Coding/2425-RobotCode/venv/lib/python3.11/site-packages/pybricksdev/cli/__init__.py", line 383, in main
    asyncio.run(subparsers.choices[args.tool].tool.run(args))
  File "/home/tom/.nix-profile/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/tom/Coding/2425-RobotCode/venv/lib/python3.11/site-packages/pybricksdev/cli/__init__.py", line 207, in run
    await hub.run(script_path, args.wait)
  File "/home/tom/Coding/2425-RobotCode/venv/lib/python3.11/site-packages/pybricksdev/connections/pybricks.py", line 572, in run
    mpy = await compile_multi_file(py_path, abi)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/Coding/2425-RobotCode/venv/lib/python3.11/site-packages/pybricksdev/compile.py", line 119, in compile_multi_file
    finder.run_script(path)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 153, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 411, in scan_code
    self._safe_import_hook(name, m, fromlist, level=0)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 367, in _safe_import_hook
    self.import_hook(name, caller, level=level)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 165, in import_hook
    q, tail = self.find_head_package(parent, name)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 221, in find_head_package
    q = self.import_module(head, qname, parent)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 308, in import_module
    fp, pathname, stuff = self.find_module(partname,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 489, in find_module
    return _find_module(name, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 69, in _find_module
    if spec.loader.is_package(name):
       ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'is_package'

When I add an init.py file to the directory, it successfully uploads the file to the hub but fails importing it:

[tom@neon:~/Coding/2425-RobotCode]$ pybricksdev run ble main.py 
Searching for any hub with Pybricks service...
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 316/316 [00:00<00:00, 2.14kB/s]
Traceback (most recent call last):
  File "main.py", line 1, in <module>
ImportError: can't import name actual_module

However, when I remove the actual_module file and put the functions into the init.py file and remove the actual_module part from the import statement, everything works fine (except for my IDE not picking up the function)

To reproduce
Steps to reproduce the behavior:

  1. Create a pybricks project
  2. Create main file
  3. Create module
  4. Import module in main

Expected behavior
It compiles every single file successfully, and it runs without any issues.

Screenshots
image

I hope my explanation of this issue was sufficient, and thanks in advance!

@TomJuri TomJuri added the triage Issues that have not been triaged yet label Apr 16, 2024
@dlech dlech added software: pybricksdev Issues related to the pybrickdev Python package and removed triage Issues that have not been triaged yet labels Apr 16, 2024
@dlech
Copy link
Member

dlech commented Apr 16, 2024

Can you share you main.py?

@TomJuri
Copy link
Author

TomJuri commented Apr 16, 2024

from my_module import actual_module

actual_module.my_func()

@laurensvalk
Copy link
Member

laurensvalk commented Apr 16, 2024

Everything works fine when I put every python file that gets imported by any other python file into the project root.

This is currently the only officially supported way.

Custom python modules (folders) don't get compiled onto the hub.

If this doesn't require firmware changes, I think we could consider supporting this in Pybricksdev.

I don't know if we can do full packages, but submodules might work.

@laurensvalk laurensvalk added the enhancement New feature or request label Apr 16, 2024
@laurensvalk laurensvalk changed the title [Bug] Custom python modules (directories) don't get compiled onto the hub. [Feature] Support submodules Apr 16, 2024
@TomJuri
Copy link
Author

TomJuri commented Apr 16, 2024

A possible workaround might be parsing all the python files and moving them to the project root and then obviously changing the imports in a script before running pybricksdev run, however it would be quite nice if pybricksdev would support this natively. Looking forward to what you figure out!

@laurensvalk
Copy link
Member

laurensvalk commented Apr 16, 2024

Support for this might have been removed in: pybricks/pybricksdev@ff32826

@laurensvalk
Copy link
Member

Just hacking around with Pybricksdev a bit to put some of pybricks/pybricksdev@ff32826 back in, it appears that the firmware still works for the following use case (see folder structure on the left).

image

pybricks/pybricksdev@ff32826 was probably done for good reason (as it does make the code cleaner), but maybe there's a way to achieve the same with the ModuleFinder introduced there.

@TomJuri
Copy link
Author

TomJuri commented Apr 17, 2024

Seems like this would just work without any modifications required if this bug in python got fixed. python/cpython#84530 It's probably going to take forever until that is fixed though. Would you be able to provide me with a patch with what exactly you changed in pybricksdev to make it work?

@dlech
Copy link
Member

dlech commented May 4, 2024

Recapping, there are two "bugs" here, one in the CPython standard library and one is a firmware limitation. And both have workarounds.

  1. Not being able to find imports when there is an implicit namespace package

  2. The following code fails in Pybricks MicroPython

    from my_module import actual_module
    
    actual_module.my_func()

    with ImportError: can't import name actual_module

    • The module is actually found and compiled into the .mpy file like it is supposed to be.
    • This is probably a limitation of how we implemented imports in the firmware.
    • Workaround is to write the code like this instead:
    from my_module.actual_module import my_func
    
    my_func()

dlech added a commit to dlech/pybricksdev that referenced this issue May 5, 2024
When compiling multi-file projects that contain a namespace pacakge
(folder without __init__.py), we hit Python bug python/cpython#84530.

This adds a try/except block to catch the error and raise a more helpful
error message.

Issue: pybricks/support#1602
@LeanderGlanda
Copy link

I'm affected by this bug too. It would be really nice to be able to have submodules so I could structure code better.

I don't fully get what's wrong.
So there is a bug in CPython, which makes that me need the init.py in a submodule/folder, right?
And the other bug is in the pybricks firmware or in pybricksdev?

dlech added a commit to pybricks/pybricksdev that referenced this issue Jun 30, 2024
When compiling multi-file projects that contain a namespace pacakge
(folder without __init__.py), we hit Python bug python/cpython#84530.

This adds a try/except block to catch the error and raise a more helpful
error message.

Issue: pybricks/support#1602
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request software: pybricksdev Issues related to the pybrickdev Python package
Projects
None yet
Development

No branches or pull requests

4 participants