Plugins with conflicting entry-point error-code specifications do not auto-load properly

[asottile]

In GitLab by @bskinn on Apr 14, 2019, 22:32

My apologies if this has been reported before, or is a known limitation; I didn't find anything similar after searching the issues for a while.

Please describe how you installed Flake8

(venv) $ pip install -r requirements-dev.txt
(venv) $ cat requirements-dev.txt
...
flake8
...

Please provide the exact, unmodified output of flake8 --bug-report

$ flake8 --bug-report
{
  "dependencies": [
    {
      "dependency": "entrypoints",
      "version": "0.3"
    }
  ],
  "platform": {
    "python_implementation": "CPython",
    "python_version": "3.6.6",
    "system": "Linux"
  },
  "plugins": [
    {
      "is_local": false,
      "plugin": "flake8-colors",
      "version": "0.1.0"
    },
    {
      "is_local": false,
      "plugin": "flake8-docstrings",
      "version": "1.3.0, pydocstyle: 3.0.0"
    },
    {
      "is_local": false,
      "plugin": "flake8-pie",
      "version": "0.0.4"
    },
    {
      "is_local": false,
      "plugin": "flake8_builtins",
      "version": "1.4.1"
    },
    {
      "is_local": false,
      "plugin": "mccabe",
      "version": "0.6.1"
    },
    {
      "is_local": false,
      "plugin": "naming",
      "version": "0.8.2"
    },
    {
      "is_local": false,
      "plugin": "pycodestyle",
      "version": "2.5.0"
    },
    {
      "is_local": false,
      "plugin": "pyflakes",
      "version": "2.1.0"
    }
  ],
  "version": "3.7.7"
}

Please describe the problem or feature

First, the following annotated snip of pip list:

flake8                   3.7.7      
flake8-bugbear           19.3.0     <=== NOTE THIS
flake8-builtins          1.4.1      
flake8-colors            0.1.6      
flake8-docstrings        1.3.0      
flake8-pie               0.0.4      <=== AND THIS
flake8-polyfill          1.0.2 

Note that both flake8-bugbear and flake8-pie are installed in the venv.

Second, the output of flake8 --version:

3.7.7 (flake8-colors: 0.1.0, flake8-docstrings: 1.3.0, pydocstyle: 3.0.0, flake8-pie: 0.0.4, flake8_builtins: 1.4.1, mccabe: 0.6.1, naming: 0.8.2, pycodestyle: 2.5.0, pyflakes: 2.1.0) CPython 3.6.6 on Linux

EXPECT: Both flake8-bugbear and flake8-pie would show in flake8 --bug-report and flake8 --version.

ACTUAL: flake8-pie shows in both, but flake8-bugbear shows in neither.

It appears that flake8 plugins with identical error-code specifications in setup(entry_points=...) collide with one another, and only one ends up loaded by flake8.

From setup.py of flake8-bugbear @ current master:

entry_points={"flake8.extension": ["B = bugbear:BugBearChecker"]},

From env/lib/.../entry-points.txt of flake8-pie v0.0.4, after install from PyPI (there are some odd discrepancies between the GitHub repo and the packages on PyPI):

[flake8.extension]
B = flake8_pie:Flake8PieCheck

Both plugins declare just B as their associated error code/code-category.

Weirdly, if I upgrade flake8-pie to v0.1.0 in the venv (pip install -U flake8-pie==0.1.0), it's now flake8-pie that fails to be loaded, whereas flake8-bugbear loads fine:

$ flake8 --version
3.7.7 (flake8-bugbear: 19.3.0, flake8-colors: 0.1.0, flake8-docstrings: 1.3.0, pydocstyle: 3.0.0, flake8_builtins: 1.4.1, mccabe: 0.6.1, naming: 0.8.2, pycodestyle: 2.5.0, pyflakes: 2.1.0) CPython 3.6.6 on Linux

$ flake8 --bug-report
{
  "dependencies": [
    {
      "dependency": "entrypoints",
      "version": "0.3"
    }
  ],
  "platform": {
    "python_implementation": "CPython",
    "python_version": "3.6.6",
    "system": "Linux"
  },
  "plugins": [
    {
      "is_local": false,
      "plugin": "flake8-bugbear",
      "version": "19.3.0"
    },
    {
      "is_local": false,
      "plugin": "flake8-colors",
      "version": "0.1.0"
    },
    {
      "is_local": false,
      "plugin": "flake8-docstrings",
      "version": "1.3.0, pydocstyle: 3.0.0"
    },
    {
      "is_local": false,
      "plugin": "flake8_builtins",
      "version": "1.4.1"
    },
    {
      "is_local": false,
      "plugin": "mccabe",
      "version": "0.6.1"
    },
    {
      "is_local": false,
      "plugin": "naming",
      "version": "0.8.2"
    },
    {
      "is_local": false,
      "plugin": "pycodestyle",
      "version": "2.5.0"
    },
    {
      "is_local": false,
      "plugin": "pyflakes",
      "version": "2.1.0"
    }
  ],
  "version": "3.7.7"
}

The contents of env/lib/.../entry_points.txt for flake8-pie==0.1.0 are identical to that for 0.0.4, above.

With flake8-pie==0.1.0 installed, such that flake8-bugbear auto-loads but flake8-pie doesn't, I can get both plugins to load if I declare flake8-pie manually in tox.ini:

[flake8:local-plugins]
extension =
    PieC = flake8_pie:Flake8PieCheck

As follows:

$ flake8 --version
3.7.7 (flake8-bugbear: 19.3.0, flake8-colors: 0.1.0, flake8-docstrings: 1.3.0, pydocstyle: 3.0.0, flake8-pie: 0.1.0, flake8_builtins: 1.4.1, mccabe: 0.6.1, naming: 0.8.2, pycodestyle: 2.5.0, pyflakes: 2.1.0) CPython 3.6.6 on Linux

For completeness:

$ flake8 --bug-report
{
  "dependencies": [
    {
      "dependency": "entrypoints",
      "version": "0.3"
    }
  ],
  "platform": {
    "python_implementation": "CPython",
    "python_version": "3.6.6",
    "system": "Linux"
  },
  "plugins": [
    {
      "is_local": false,
      "plugin": "flake8-bugbear",
      "version": "19.3.0"
    },
    {
      "is_local": false,
      "plugin": "flake8-colors",
      "version": "0.1.0"
    },
    {
      "is_local": false,
      "plugin": "flake8-docstrings",
      "version": "1.3.0, pydocstyle: 3.0.0"
    },
    {
      "is_local": false,
      "plugin": "flake8-pie",
      "version": "0.1.0"
    },
    {
      "is_local": false,
      "plugin": "flake8_builtins",
      "version": "1.4.1"
    },
    {
      "is_local": false,
      "plugin": "mccabe",
      "version": "0.6.1"
    },
    {
      "is_local": false,
      "plugin": "naming",
      "version": "0.8.2"
    },
    {
      "is_local": false,
      "plugin": "pycodestyle",
      "version": "2.5.0"
    },
    {
      "is_local": false,
      "plugin": "pyflakes",
      "version": "2.1.0"
    }
  ],
  "version": "3.7.7"
}

Curiously, flake8-pie doesn't show as is_local, despite it being loaded due to the specification in [flake8:local-plugins].

[asottile]

In GitLab by @asottile on Apr 14, 2019, 23:06

yeah not sure if this is documented anywhere but the name has to be unique. The oddity with uninstall / upgrade / ordering is due to setuptools metadata being stored on the filesystem and so you have (~relatively undefined) ordering based on inode.

I think this is working as intended right now, though it would maybe be better if some sort of warning / error occurred in this case.

This is probably the most relevant line: https://gitlab.com/pycqa/flake8/blob/98beabdcc50ee352a8a5eded1009b0914a3645b9/src/flake8/plugins/manager.py#L276

[asottile]

In GitLab by @bskinn on Apr 14, 2019, 23:27

@asottile To be sure I understand you, is it that the B in both of these supposed to be a ~free-text name for the plugin? The 'name that has to be unique'? Because in most (all?) of the (few) plugins whose setup.pys I've looked at, that value has tended to be some representation of the error codes raised by the plugin. There may be a pervasive misunderstanding of the semantics of that identifier! This uniqueness requirement of the entry point name should probably be prominently called out in any official plugin-developers' how-to/tutorial...?

And, regardless, yeah, it seems like at minimum a warning should be emitted at execution time for plugin namespace collisions like this.

[asottile]

In GitLab by @asottile on Apr 14, 2019, 23:34

Actually yeah, now that I found the right docs page (I was looking at the using plugins page earlier!) it does seem to be prominently called out twice there:

http://flake8.pycqa.org/en/latest/plugin-development/registering-plugins.html

Flake8 requires each entry point to be unique amongst all plugins installed in the users environment. Selecting an entry point that is already used can cause plugins to be deactivated without warning!

This was part of the rationale for moving to allowing 3-letter codes in flake8 3.x

And yeah, upgrading / downgrading a package likely changes the inode ordering (try running os.listdir('.') on an arbitrary directory, the results are very unlikely to be sorted!)

[asottile]

In GitLab by @bskinn on Apr 14, 2019, 23:43

Unngh... So... This... collision behavior is intentional?

The X in checking plugins define what error codes it is going to report. So if the plugin reports only the error code X101 your entry-point would look like:

This seems like a problematically fragile semantics. Is the idea that this would avoid plugins reporting the same error codes? It...seems like a really awkward/incomplete/problematic approach, if so.

IMO the entry point name should be, e.g., the package name...MUCH more likely to be unique. If there's a desire to have some sort of machinery in place to avoid multiple things raising the same code, it would make more sense to me for plugins to register the codes they raise via some other parameter in the plugin registration handshake...then it'd be fully granular.

[asottile]

In GitLab by @bskinn on Apr 14, 2019, 23:44

And <nod>, I have noticed that listdir results are usually wildly unsorted. :-)

[asottile]

In GitLab by @asottile on Apr 14, 2019, 23:48

It's not so simple unfortunately -- that prefix is used by --enable-extension to turn on non-default plugins. It's also pretty ingrained into the plugin structure dating back to at least flake8 2.x so it's pretty unlikely to change.

I'm not sure that the behaviour is ~intentional so-to-speak, though it is at least known and documented. Ideally plugins would select unique reported error codes, though there isn't really any sort of centralized place to manage that (even if there was, that'd box out proprietary / closed-source plugins).

[asottile]

In GitLab by @bskinn on Apr 14, 2019, 23:56

Hrmmmm...so, the recommended best-practice is for plugin developers to switch to the multi-letter code prefixes? If so, a marketing push to try to coax devs of existing plugins to make that switch might be of some value...counterweighted by the users of the plugins potentially needing to update their configs, I guess.

And...as a user, I guess I just have to manually check every time I install a new plugin, that all of the plugins I expect do still show up on a flake8 --version? <shakes head> I may just put all of my desired plugins into [flake8:local-plugins], and just forget about the auto-load functionality, so that I don't have to be concerned about the whole thing. :-/

[asottile]

In GitLab by @asottile on Apr 14, 2019, 23:58

I think it's more that new plugins should pick the three-letter codes, assuming there aren't existing conflicts

[asottile]

In GitLab by @bskinn on Apr 15, 2019, 24:00

closed

[asottile]

In GitLab by @bskinn on Apr 15, 2019, 24:04

<nod>, makes sense.

In this case, which project would you recommend I approach first to switch codes, and avoid the collision? -pie is smaller and non-PyCQA, and so seems the more natural one to displace, but -bugbear might be more responsive, being PyCQA?

[asottile]

In GitLab by @asottile on Apr 15, 2019, 24:08

flake8-pie appears to be brand new so they should probably change, maybe suggest PIE as their prefix 😆?

(an aside, I'm hoping to get one of their rules directly into pyflakes here)

[asottile]

In GitLab by @bskinn on Apr 15, 2019, 24:16

Hehe, PIE was my thought, too. Ok, I'll start there. Thanks vm for your time!

Good luck with that PR...man, soooo many complex cases to consider...!