[Feature] Support submodules #1602

TomJuri · 2024-04-16T13:15:58Z

Describe the bug
Custom python modules (folders) don't get compiled onto the hub. Everything works fine when I put every python file that gets imported by any other python file into the project root. However, when I try to import a file that's within a directory, this error occurs:

[tom@neon:~/Coding/2425-RobotCode]$ pybricksdev run ble main.py 
Searching for any hub with Pybricks service...
Traceback (most recent call last):
  File "/home/tom/Coding/2425-RobotCode/venv/bin/pybricksdev", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/tom/Coding/2425-RobotCode/venv/lib/python3.11/site-packages/pybricksdev/cli/__init__.py", line 383, in main
    asyncio.run(subparsers.choices[args.tool].tool.run(args))
  File "/home/tom/.nix-profile/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/tom/Coding/2425-RobotCode/venv/lib/python3.11/site-packages/pybricksdev/cli/__init__.py", line 207, in run
    await hub.run(script_path, args.wait)
  File "/home/tom/Coding/2425-RobotCode/venv/lib/python3.11/site-packages/pybricksdev/connections/pybricks.py", line 572, in run
    mpy = await compile_multi_file(py_path, abi)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/Coding/2425-RobotCode/venv/lib/python3.11/site-packages/pybricksdev/compile.py", line 119, in compile_multi_file
    finder.run_script(path)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 153, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 411, in scan_code
    self._safe_import_hook(name, m, fromlist, level=0)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 367, in _safe_import_hook
    self.import_hook(name, caller, level=level)
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 165, in import_hook
    q, tail = self.find_head_package(parent, name)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 221, in find_head_package
    q = self.import_module(head, qname, parent)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 308, in import_module
    fp, pathname, stuff = self.find_module(partname,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 489, in find_module
    return _find_module(name, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tom/.nix-profile/lib/python3.11/modulefinder.py", line 69, in _find_module
    if spec.loader.is_package(name):
       ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'is_package'

When I add an init.py file to the directory, it successfully uploads the file to the hub but fails importing it:

[tom@neon:~/Coding/2425-RobotCode]$ pybricksdev run ble main.py 
Searching for any hub with Pybricks service...
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 316/316 [00:00<00:00, 2.14kB/s]
Traceback (most recent call last):
  File "main.py", line 1, in <module>
ImportError: can't import name actual_module

However, when I remove the actual_module file and put the functions into the init.py file and remove the actual_module part from the import statement, everything works fine (except for my IDE not picking up the function)

To reproduce
Steps to reproduce the behavior:

Create a pybricks project
Create main file
Create module
Import module in main

Expected behavior
It compiles every single file successfully, and it runs without any issues.

Screenshots

I hope my explanation of this issue was sufficient, and thanks in advance!

The text was updated successfully, but these errors were encountered:

dlech · 2024-04-16T13:59:02Z

Can you share you main.py?

TomJuri · 2024-04-16T14:11:44Z

from my_module import actual_module

actual_module.my_func()

laurensvalk · 2024-04-16T14:35:28Z

Everything works fine when I put every python file that gets imported by any other python file into the project root.

This is currently the only officially supported way.

Custom python modules (folders) don't get compiled onto the hub.

If this doesn't require firmware changes, I think we could consider supporting this in Pybricksdev.

I don't know if we can do full packages, but submodules might work.

TomJuri · 2024-04-16T14:40:57Z

A possible workaround might be parsing all the python files and moving them to the project root and then obviously changing the imports in a script before running pybricksdev run, however it would be quite nice if pybricksdev would support this natively. Looking forward to what you figure out!

laurensvalk · 2024-04-16T14:44:43Z

Support for this might have been removed in: pybricks/pybricksdev@ff32826

laurensvalk · 2024-04-16T14:59:53Z

Just hacking around with Pybricksdev a bit to put some of pybricks/pybricksdev@ff32826 back in, it appears that the firmware still works for the following use case (see folder structure on the left).

pybricks/pybricksdev@ff32826 was probably done for good reason (as it does make the code cleaner), but maybe there's a way to achieve the same with the ModuleFinder introduced there.

TomJuri · 2024-04-17T18:01:20Z

Seems like this would just work without any modifications required if this bug in python got fixed. python/cpython#84530 It's probably going to take forever until that is fixed though. Would you be able to provide me with a patch with what exactly you changed in pybricksdev to make it work?

dlech · 2024-05-04T23:55:38Z

Recapping, there are two "bugs" here, one in the CPython standard library and one is a firmware limitation. And both have workarounds.

Not being able to find imports when there is an implicit namespace package
- This is modulefinder chokes on numpy - dereferencing None in spec.loader python/cpython#84530
- Workaround is to include __init__.py so there isn't an implicit namespace package
The following code fails in Pybricks MicroPython
```
from my_module import actual_module

actual_module.my_func()
```
with ImportError: can't import name actual_module
- The module is actually found and compiled into the .mpy file like it is supposed to be.
- This is probably a limitation of how we implemented imports in the firmware.
- Workaround is to write the code like this instead:
```
from my_module.actual_module import my_func

my_func()
```

When compiling multi-file projects that contain a namespace pacakge (folder without __init__.py), we hit Python bug python/cpython#84530. This adds a try/except block to catch the error and raise a more helpful error message. Issue: pybricks/support#1602

LeanderGlanda · 2024-06-12T12:44:45Z

I'm affected by this bug too. It would be really nice to be able to have submodules so I could structure code better.

I don't fully get what's wrong.
So there is a bug in CPython, which makes that me need the init.py in a submodule/folder, right?
And the other bug is in the pybricks firmware or in pybricksdev?

When compiling multi-file projects that contain a namespace pacakge (folder without __init__.py), we hit Python bug python/cpython#84530. This adds a try/except block to catch the error and raise a more helpful error message. Issue: pybricks/support#1602

TomJuri added the triage Issues that have not been triaged yet label Apr 16, 2024

dlech added software: pybricksdev Issues related to the pybrickdev Python package and removed triage Issues that have not been triaged yet labels Apr 16, 2024

laurensvalk added the enhancement New feature or request label Apr 16, 2024

laurensvalk changed the title ~~[Bug] Custom python modules (directories) don't get compiled onto the hub.~~ [Feature] Support submodules Apr 16, 2024

dlech mentioned this issue May 5, 2024

pybricksdev/compile.py: Add better error message pybricks/pybricksdev#85

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Support submodules #1602

[Feature] Support submodules #1602

TomJuri commented Apr 16, 2024

dlech commented Apr 16, 2024

TomJuri commented Apr 16, 2024

laurensvalk commented Apr 16, 2024 •

edited

Loading

TomJuri commented Apr 16, 2024 •

edited

Loading

laurensvalk commented Apr 16, 2024 •

edited

Loading

laurensvalk commented Apr 16, 2024

TomJuri commented Apr 17, 2024 •

edited

Loading

dlech commented May 4, 2024 •

edited

Loading

LeanderGlanda commented Jun 12, 2024

[Feature] Support submodules #1602

[Feature] Support submodules #1602

Comments

TomJuri commented Apr 16, 2024

dlech commented Apr 16, 2024

TomJuri commented Apr 16, 2024

laurensvalk commented Apr 16, 2024 • edited Loading

TomJuri commented Apr 16, 2024 • edited Loading

laurensvalk commented Apr 16, 2024 • edited Loading

laurensvalk commented Apr 16, 2024

TomJuri commented Apr 17, 2024 • edited Loading

dlech commented May 4, 2024 • edited Loading

LeanderGlanda commented Jun 12, 2024

laurensvalk commented Apr 16, 2024 •

edited

Loading

TomJuri commented Apr 16, 2024 •

edited

Loading

laurensvalk commented Apr 16, 2024 •

edited

Loading

TomJuri commented Apr 17, 2024 •

edited

Loading

dlech commented May 4, 2024 •

edited

Loading