User-installed flake8 is using system-level pycodestyle dependency and crashing

[BTOdell]

Please describe how you installed Flake8

$ sudo apt install python3-flake8
$ python3 -m pip install --user --upgrade flake8

Flake8 is installed at the system-level by administrator on Ubuntu 18.04:

admin@host:~$ sudo apt install python3-flake8
...
admin@host:~$ flake8 --version
3.5.0 (mccabe: 0.6.1, pycodestyle: 2.3.1, pyflakes: 1.6.0) CPython 3.6.9 on Linux

Then, a user (jenkins) installs a newer version of flake8 (3.9.2) to their user directory:

jenkins@host:~$ python3 -m pip install --user --upgrade flake8
...
jenkins@host:~$ flake8 --version
Traceback (most recent call last):
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/plugins/manager.py", line 157, in load_plugin
    self._load()
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/plugins/manager.py", line 134, in _load
    self._plugin = self.entry_point.load()
  File "/usr/local/lib/python3.6/dist-packages/importlib_metadata/__init__.py", line 98, in load
    return functools.reduce(getattr, attrs, module)
AttributeError: module 'pycodestyle' has no attribute 'break_around_binary_operator'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/scratch/jenkins/.local/bin/flake8", line 8, in <module>
    sys.exit(main())
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/main/cli.py", line 22, in main
    app.run(argv)
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/main/application.py", line 363, in run
    self._run(argv)
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/main/application.py", line 350, in _run
    self.initialize(argv)
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/main/application.py", line 330, in initialize
    self.find_plugins(config_finder)
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/main/application.py", line 159, in find_plugins
    self.check_plugins.load_plugins()
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/plugins/manager.py", line 415, in load_plugins
    plugins = list(self.manager.map(load_plugin))
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/plugins/manager.py", line 302, in map
    yield func(self.plugins[name], *args, **kwargs)
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/plugins/manager.py", line 413, in load_plugin
    return plugin.load_plugin()
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/plugins/manager.py", line 164, in load_plugin
    raise failed_to_load
flake8.exceptions.FailedToLoadPlugin: Flake8 failed to load plugin "pycodestyle.break_around_binary_operator" due to module 'pycodestyle' has no attribute 'break_around_binary_operator'.

Please provide the exact, unmodified output of flake8 --bug-report

jenkins@host:~$ flake8 --bug-report
Traceback (most recent call last):
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/plugins/manager.py", line 157, in load_plugin
    self._load()
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/plugins/manager.py", line 134, in _load
    self._plugin = self.entry_point.load()
  File "/usr/local/lib/python3.6/dist-packages/importlib_metadata/__init__.py", line 98, in load
    return functools.reduce(getattr, attrs, module)
AttributeError: module 'pycodestyle' has no attribute 'break_around_binary_operator'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/scratch/jenkins/.local/bin/flake8", line 8, in <module>
    sys.exit(main())
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/main/cli.py", line 22, in main
    app.run(argv)
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/main/application.py", line 363, in run
    self._run(argv)
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/main/application.py", line 350, in _run
    self.initialize(argv)
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/main/application.py", line 330, in initialize
    self.find_plugins(config_finder)
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/main/application.py", line 159, in find_plugins
    self.check_plugins.load_plugins()
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/plugins/manager.py", line 415, in load_plugins
    plugins = list(self.manager.map(load_plugin))
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/plugins/manager.py", line 302, in map
    yield func(self.plugins[name], *args, **kwargs)
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/plugins/manager.py", line 413, in load_plugin
    return plugin.load_plugin()
  File "/scratch/jenkins/.local/lib/python3.6/site-packages/flake8/plugins/manager.py", line 164, in load_plugin
    raise failed_to_load
flake8.exceptions.FailedToLoadPlugin: Flake8 failed to load plugin "pycodestyle.break_around_binary_operator" due to module 'pycodestyle' has no attribute 'break_around_binary_operator'.

Please describe the problem or feature

It seems like flake8 is importing the system-level pycodestyle package instead of the user-level pycodestyle package.
If the system-level flake8 package is uninstalled and all dependencies (inc. pycodestyle) are removed, then the user-level flake8 starts working.

admin@host:~$ sudo apt --purge autoremove python3-flake8
...
jenkins@host:~$ flake8 --version
3.9.2 (pycodestyle: 2.7.0, pyflakes: 2.3.1) CPython 3.6.9 on Linux

Perhaps a bug in flake8's plugin resolution order?

[asottile]

that's just how user installs work -- your local site-packages and the system site-packages are both available

use a virtualenv

[BTOdell]

@asottile It should prioritize using the user-installed version of pycodestyle though, right?

[asottile]

it does, but they're both available so they both get loaded

[sigmavirus24]

@BTOdell also, how exactly do you expect Flake8 itself to isolate itself from the system? It's not possible. We rely on external libraries to find the installations and they have no means of distinguishing between the two either. As @asottile has said, use a virtualenv. If you don't want to configure that yourself, you can use a tool like pipx or pipsi to achieve a user installation inside the virtualenv.

[BTOdell]

I expect flake8 to load and use the correct version of pycodestyle that has been installed for the user. If flake8 can't load the library that is literally installed right next to it, then that's a bug with flake8. Why would flake8 load both versions of the library? Does it need to use two different versions? Obviously not, so it should only load the library that is installed "closest" to itself.

I have other linting tools installed in this same way. They have dependencies too and they load the correct user-installed versions. Why can't flake8?

[BTOdell]

Python searches sys.path when importing a package or module. The sys.path that flake8 is using (on my computer) includes the user site before the system site:

jenkins@host:~$ python3 -m site
sys.path = [
    '/scratch/jenkins',
    '/usr/lib/python36.zip',
    '/usr/lib/python3.6',
    '/usr/lib/python3.6/lib-dynload',
    '/scratch/jenkins/.local/lib/python3.6/site-packages',
    '/usr/local/lib/python3.6/dist-packages',
    '/usr/lib/python3/dist-packages',
]
USER_BASE: '/scratch/jenkins/.local' (exists)
USER_SITE: '/scratch/jenkins/.local/lib/python3.6/site-packages' (exists)
ENABLE_USER_SITE: True

Is flake8 not using standard Python importing?

[sigmavirus24]

We do use it

flake8/src/flake8/_compat.py

Line 5 in 03401a6

import importlib.metadata as importlib_metadata

(and it's backport) which is in the standard library. It is what loads these dependencies. I suspect your other tools are "doing the right thing" because they aren't being incorrectly installed and used like Flake8 is. In fact, if we weren't using the standard library and instead some kind of home-brewed, not-invented-here we might be able to magic what you want, but what you're asking for doesn't exist and is unreasonable.

[sigmavirus24]

I have other linting tools installed in this same way. They have dependencies too and they load the correct user-installed versions. Why can't flake8?

It's also condescending, bordering on hostile, to insist that the maintainers of a widely used package who have a combined over 2 decades of experience with this don't know what we're doing when we provide you with the exact explanation of the problem as well as a solution.

[BTOdell]

I am using pylint and bandit in the same way. Older versions are installed at the system-level and I've installed newer versions for the user. Both tools work fine but flake8 doesn't. I know I could use a virtualenv but I need the tools to be available from the command line for a user without setting up an isolated environment. A user install is a perfectly valid and documented way of installing packages, so I don't know why you keep saying it's been installed incorrectly and used wrong.

You haven't provided an "exact explanation" at all. In fact, you've provided conflicting answers. You said before that you are using "external libraries" to find the installations and now you say you're using the standard, built-in importlib library.

I'm being persistent because you're being dismissive of this problem and there is a chance that you're not understanding what my expectation is. I don't know how the expectation of a tool importing it's dependencies correctly is unreasonable.

From the stack trace, it seems like the crash is originating from your (dare I say) home-brew plugin manager. If the sys.path variable has the user site-packages listed first, then I would expect that importlib would import the version of pycodestyle that is installed there before it finds the system-installed version. You claim that flake8 is loading both versions of pycodestyle but you never answered why flake8 would need to do that.

[asottile]

enough.

[sigmavirus24]

To wrap this up

A user install is a perfectly valid and documented way of installing packages, so I don't know why you keep saying it's been installed incorrectly and used wrong.

User-installed packages are an occasionally okay way to install things which is why the PyPA - the folks behind importlib.metadata (it's backport as well) and pip have built pipx to improve the user-level installation with the same conveniences but with far better behaviour that doesn't run into issues exactly like this.

User-installed packages also presume you're not going to try to have multiple copies (system-level and user-level). The fact that this sometimes works is almost certainly undefined behaviour.

In fact, you've provided conflicting answers.

No. You just haven't done enough to understand the underlying mechanisms at play. We've used importlib for ages. I think you were confused by "external" which does not mean "3rd party" (e.g., installed from PyPI - although on older Python versions a backport is used from PyPI) but "external to this tool".

I don't know how the expectation of a tool importing it's dependencies correctly is unreasonable.

It's unreasonable when the behaviour is that of the standard library.

it seems like the crash is originating from your (dare I say) home-brew plugin manager

Which provides logic around the standard library to make our lives simpler. It's a convenience layer for the rest of the application which you might know if you'd looked at the maintainer docs or read up on importlib.metadata during this time (instead of implying you know more and better than other folks).

You claim that flake8 is loading both versions of pycodestyle but you never answered why flake8 would need to do that.

At the very least it's loading the standard library first. It may be loading both simultaneously but it's not behaviour we can anticipate nor is the added complexity of fixing "One hostile user complained about this" worth the effort or the maintenance cost when the authorities have built better ways of doing what you want which when used get you exactly the behaviour you want. But I'm guessing you didn't bother reading the docs that I found and linked for you or understanding how those tools better suit your needs.