Exception on lint when Thonny installed with bundled Python

[BorysekOndrej]

Versions:

Thonny: 4.1.1 with bundled Python (likely since 4.0.0)
Thonny-edulint: 0.3.2 (which installs edulint 2.9.2)

Thonny 4.0.0 release notes

Use isolated mode for launching Thonny. This includes hiding user-site packages from Thonny's GUI process (not from user programs). Plug-ins now get installed under Thonny's data folder. (for links to PRs check the release notes)

Problem

When thonny-edulint plugin is installed to Thonny with bundled Python than thonny-edulint throws exception during file linting.
The root cause of this exception is thonny-edulint being unable to import edulint, which should have been installed as a dependency. This dependency is installed into a path, that is currently not part of the python path (see thonny/thonny#2816).

The actual exception is only a side effect of the unsuccessful import. For more information more detailed logs are attached bellow.

Resolution:

Will be fixed in Thonny 4.1.2. thonny/thonny#2816

Workaround:

• We will add clearer error message in thonny-edulint, explaining to temporarily (until bug is fixed):

  • Install edulint also as a package: Tools -> Manage packages... -> edulint -> Install

• Also note that it works out of the box when thonny and thonny-edulint is installed in existing python environment (i.e. using python3 -m pip install thonny thonny-edulint)

Steps to reproduce:

1. Install Thonny 4.1.1 (current version) with bundled Python (e.g. the .exe file for Windows)
2. Thonny -> Tools -> Manage plug-ins... -> thonny-edulint -> Install
3. Restart Thonny
4. Open arbitrary Python file
5. Click the Lint button (has a broomstick icon)
6. Exception window is thrown - it will complain about invalid JSON, but that is sideeffect of not being able to import edulint package.

Plugin install log

C:\Users\ondre\AppData\Local\Programs\Thonny\python.exe -m pip --disable-pip-version-check --no-color install --no-warn-script-location --user --user thonny-edulint
Collecting thonny-edulint
  Using cached thonny_edulint-0.3.2-py3-none-any.whl (28 kB)
Collecting Pygments
  Using cached Pygments-2.16.1-py3-none-any.whl (1.2 MB)
Collecting m2r2
  Using cached m2r2-0.3.3.post2-py3-none-any.whl (11 kB)
Collecting edulint>=2.6.5
  Using cached edulint-2.9.2-py3-none-any.whl (144 kB)
Requirement already satisfied: thonny>=3.0.0 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from thonny-edulint) (4.1.1)
Collecting dataclasses-json
  Using cached dataclasses_json-0.5.14-py3-none-any.whl (26 kB)
Collecting requests
  Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Collecting flake8>=3.8.4
  Using cached flake8-6.1.0-py2.py3-none-any.whl (58 kB)
Collecting flake8-json==21.7.0
  Using cached flake8_json-21.7.0-py2.py3-none-any.whl (4.4 kB)
Requirement already satisfied: tomli in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from edulint>=2.6.5->thonny-edulint) (2.0.1)
Requirement already satisfied: pylint>=2.15.10 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from edulint>=2.6.5->thonny-edulint) (2.17.4)
Requirement already satisfied: setuptools>=45.2 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from thonny>=3.0.0->thonny-edulint) (65.5.0)
Requirement already satisfied: Send2Trash>=1.5 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from thonny>=3.0.0->thonny-edulint) (1.8.2)
Requirement already satisfied: jedi>=0.18.1 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from thonny>=3.0.0->thonny-edulint) (0.18.2)
Requirement already satisfied: wheel in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from thonny>=3.0.0->thonny-edulint) (0.40.0)
Requirement already satisfied: asttokens>=2.0 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from thonny>=3.0.0->thonny-edulint) (2.2.1)
Requirement already satisfied: mypy>=0.761 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from thonny>=3.0.0->thonny-edulint) (1.3.0)
Requirement already satisfied: docutils>=0.16 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from thonny>=3.0.0->thonny-edulint) (0.20.1)
Requirement already satisfied: pyserial>=3.4 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from thonny>=3.0.0->thonny-edulint) (3.5)
Collecting mistune==0.8.4
  Using cached mistune-0.8.4-py2.py3-none-any.whl (16 kB)
Requirement already satisfied: six in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from asttokens>=2.0->thonny>=3.0.0->thonny-edulint) (1.16.0)
Requirement already satisfied: mccabe<0.8.0,>=0.7.0 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from flake8>=3.8.4->edulint>=2.6.5->thonny-edulint) (0.7.0)
Collecting pycodestyle<2.12.0,>=2.11.0
  Using cached pycodestyle-2.11.0-py2.py3-none-any.whl (31 kB)
Collecting pyflakes<3.2.0,>=3.1.0
  Using cached pyflakes-3.1.0-py2.py3-none-any.whl (62 kB)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from jedi>=0.18.1->thonny>=3.0.0->thonny-edulint) (0.8.3)
Requirement already satisfied: typing-extensions>=3.10 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from mypy>=0.761->thonny>=3.0.0->thonny-edulint) (4.6.3)
Requirement already satisfied: mypy-extensions>=1.0.0 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from mypy>=0.761->thonny>=3.0.0->thonny-edulint) (1.0.0)
Requirement already satisfied: colorama>=0.4.5 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from pylint>=2.15.10->edulint>=2.6.5->thonny-edulint) (0.4.6)
Requirement already satisfied: astroid<=2.17.0-dev0,>=2.15.4 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from pylint>=2.15.10->edulint>=2.6.5->thonny-edulint) (2.15.5)
Requirement already satisfied: isort<6,>=4.2.5 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from pylint>=2.15.10->edulint>=2.6.5->thonny-edulint) (5.12.0)
Requirement already satisfied: dill>=0.2 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from pylint>=2.15.10->edulint>=2.6.5->thonny-edulint) (0.3.6)
Requirement already satisfied: platformdirs>=2.2.0 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from pylint>=2.15.10->edulint>=2.6.5->thonny-edulint) (3.5.3)
Requirement already satisfied: tomlkit>=0.10.1 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from pylint>=2.15.10->edulint>=2.6.5->thonny-edulint) (0.11.8)
Collecting typing-inspect<1,>=0.4.0
  Using cached typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)
Collecting marshmallow<4.0.0,>=3.18.0
  Using cached marshmallow-3.20.1-py3-none-any.whl (49 kB)
Collecting urllib3<3,>=1.21.1
  Using cached urllib3-2.0.4-py3-none-any.whl (123 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2023.7.22-py3-none-any.whl (158 kB)
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.2.0-cp310-cp310-win_amd64.whl (96 kB)
Requirement already satisfied: wrapt<2,>=1.11 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from astroid<=2.17.0-dev0,>=2.15.4->pylint>=2.15.10->edulint>=2.6.5->thonny-edulint) (1.15.0)
Requirement already satisfied: lazy-object-proxy>=1.4.0 in c:\users\ondre\appdata\local\programs\thonny\lib\site-packages (from astroid<=2.17.0-dev0,>=2.15.4->pylint>=2.15.10->edulint>=2.6.5->thonny-edulint) (1.9.0)
Collecting packaging>=17.0
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Installing collected packages: mistune, urllib3, typing-inspect, Pygments, pyflakes, pycodestyle, packaging, m2r2, idna, flake8-json, charset-normalizer, certifi, requests, marshmallow, flake8, dataclasses-json, edulint, thonny-edulint
Successfully installed Pygments-2.16.1 certifi-2023.7.22 charset-normalizer-3.2.0 dataclasses-json-0.5.14 edulint-2.9.2 flake8-6.1.0 flake8-json-21.7.0 idna-3.4 m2r2-0.3.3.post2 marshmallow-3.20.1 mistune-0.8.4 packaging-23.1 pycodestyle-2.11.0 pyflakes-3.1.0 requests-2.31.0 thonny-edulint-0.3.2 typing-inspect-0.9.0 urllib3-2.0.4
Done!

Frontend log noting the two errors

c:\Users\ondre\AppData\Roaming\Thonny\frontend.log

12:54:01.388 [MainThread] INFO    thonny: Thonny version: 4.1.1
12:54:01.389 [MainThread] INFO    thonny: cwd: C:\Users\ondre\AppData\Local\Programs\Thonny
12:54:01.389 [MainThread] INFO    thonny: original argv: []
12:54:01.389 [MainThread] INFO    thonny: sys.executable: C:\Users\ondre\AppData\Local\Programs\Thonny\pythonw.exe
12:54:01.389 [MainThread] INFO    thonny: sys.argv: ['C:\\Users\\ondre\\AppData\\Local\\Programs\\Thonny\\thonny.exe']
12:54:01.389 [MainThread] INFO    thonny: sys.path: ['C:\\Users\\ondre\\AppData\\Local\\Programs\\Thonny\\python310.zip', 'C:\\Users\\ondre\\AppData\\Local\\Programs\\Thonny\\DLLs', 'C:\\Users\\ondre\\AppData\\Local\\Programs\\Thonny\\lib', 'C:\\Users\\ondre\\AppData\\Local\\Programs\\Thonny', 'C:\\Users\\ondre\\AppData\\Local\\Programs\\Thonny\\lib\\site-packages']
12:54:01.389 [MainThread] INFO    thonny: sys.flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=1, no_site=0, ignore_environment=1, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=1, dev_mode=False, utf8_mode=0, warn_default_encoding=0, int_max_str_digits=-1)
12:54:01.438 [MainThread] INFO    thonny.workbench: Starting Workbench
12:54:01.552 [MainThread] INFO    thonny.workbench: Creating runner
12:54:01.554 [MainThread] INFO    thonny.workbench: Start loading plugins
12:54:02.801 [MainThread] INFO    thonny.workbench: Done loading plugins
12:54:02.879 [MainThread] INFO    thonny.workbench: Opening views
12:54:03.170 [MainThread] INFO    thonny.plugins.cpython_frontend.cp_front: Creating LocalCPythonProxy
12:54:03.170 [MainThread] INFO    thonny.running: Starting the backend: ['C:\\Users\\ondre\\AppData\\Local\\Programs\\Thonny\\python.exe', '-u', '-B', 'C:\\Users\\ondre\\AppData\\Local\\Programs\\Thonny\\lib\\site-packages\\thonny\\plugins\\cpython_backend\\cp_launcher.py', 'C:\\Users\\ondre', "{'run.warn_module_shadowing': True}"] C:\Users\ondre
12:54:42.889 [MainThread] INFO    thonny.ui_utils: Restoring focus to .
12:54:42.889 [MainThread] INFO    thonny.editors: Save dialog returned '' with typevariable ''
12:55:02.420 [MainThread] INFO    thonny.ui_utils: Restoring focus to .
12:55:02.420 [MainThread] INFO    thonny.editors: Save dialog returned 'C:/Github/anicka/tmp/asdads' with typevariable 'Python files'
12:55:04.891 [MainThread] ERROR   thonny: Edulint: C:\Users\ondre\AppData\Local\Programs\Thonny\pythonw.exe: No module named edulint

12:55:04.891 [MainThread] ERROR   thonny.workbench: Internal error
Traceback (most recent call last):
  File "C:\Users\ondre\AppData\Local\Programs\Thonny\lib\tkinter\__init__.py", line 1921, in __call__
    return self.func(*args)
  File "C:\Users\ondre\AppData\Local\Programs\Thonny\lib\tkinter\__init__.py", line 839, in callit
    func(*args)
  File "C:\Users\ondre\AppData\Local\Programs\Thonny\lib\site-packages\thonny\ui_utils.py", line 2294, in poll
    on_completion(proc, out_lines, err_lines)
  File "C:\Users\ondre\AppData\Roaming\Thonny\plugins\Python310\site-packages\thonnycontrib\edulint\__init__.py", line 45, in _parse_and_output_warnings
    edulint_findings = json.loads("\n".join(out_lines))
  File "C:\Users\ondre\AppData\Local\Programs\Thonny\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Users\ondre\AppData\Local\Programs\Thonny\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\ondre\AppData\Local\Programs\Thonny\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
12:55:04.907 [MainThread] INFO    thonny.ui_utils: Parent y: 50, rooty: 101, vrooty: 0
12:55:04.908 [MainThread] INFO    thonny.ui_utils: Placing .!longtextdialog with geometry +283+243

[GiraffeReversed]

Workaround (error message to install edulint separately) implemented: a924165

[BorysekOndrej]

So Thonny 4.1.2 fixed the bug -- sort of. Now when you do Thonny -> Tools -> Manage plug-ins... -> thonny-edulint -> Install than both thonny-edulint and edulint will appear to be installed both as a Plugin and as a Package. And more problematically -- when package is installed from a Plugin as a dependency than the button to Install from Packages dialog disappears.

If you've only installed from Plugins dialog than during the lint the edulint module is not found (and we have a good error message for that). So we still have to install edulint as a package. And we have to do it before we install thonny-edulint as a plugin.

Interestingly enough this leads to a weird behavior. Update dialog reads the version of edulint Plugin, but linting uses version of edulint Package. And even more curiously the version comparison sometimes triggers the dialog even if all version numbers look identical.

I haven't found a good way how to solve it yet. I'll keep looking tommorow.

[BorysekOndrej]

The current behavior (Thonny 4.1.4 from Nov 19, 2023) seems identical to the originally reported behavior (i.e. first install Thonny-EduLint as a plugin, then you have to install EduLint as a package).

The commit da8fe76 replaces the current error msg (which was just barely on screen due to stack trace) with two new error msgs. One is a dialog, that appears during the failure. Second is a fake edulint finding that persists in the right window pane.

[BorysekOndrej]

A more comprehensive workaround was in thonny-edulint 0.6.2.