-p plugin not working, fails to early-load plugin

[nickwilliams-eventbrite]

Operating systems: Ubuntu 16.04, macOS Sierra, macOS High Sierra
Python: 3.7.1
Pytest:

$ pytest --version
This is pytest version 4.2.0, imported from /usr/local/lib/python3.7/site-packages/pytest.py
setuptools registered plugins:
  pytest-django-3.4.7 at /usr/local/lib/python3.7/site-packages/pytest_django/plugin.py
  pytest-cov-2.6.1 at /usr/local/lib/python3.7/site-packages/pytest_cov/plugin.py
  pysoa-0.56.0 at /usr/local/lib/python3.7/site-packages/pysoa/test/plugins/pytest/fixtures.py
  pysoa-0.56.0 at /usr/local/lib/python3.7/site-packages/pysoa/test/plugins/pytest/plans.py

pytest --help describes an option for early-loading plugins:

  -p name               early-load given plugin (multi-allowed). To avoid
                        loading of plugins, use the `no:` prefix, e.g.
                        `no:doctest`.

I hoped to use this to early-load the code coverage plugin pytest-cov. (The motivation is that pytest-cov should be loaded before any other plugins, but on some of our machines pytest-django is loading first, resulting in about a 15% coverage difference, which causes tests to fail on those machines because we have --fail-under set for enforcement of code coverage.) However, as discussed in #935, -p is broken and no longer works. Here are all the different approaches I tried:

# disables plugin
$ pytest -p pytest_cov
usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov-report --cov-branch --cov-fail-under=90 --cov=.......

# does not recognize it
$ pytest -p pytest-cov
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
    __import__(importspec)
ModuleNotFoundError: No module named 'pytest-cov'

# does not recognize it
$ pytest -p cov
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
    __import__(importspec)
ModuleNotFoundError: No module named 'cov'

# finally no error, but does not load it early
$ pytest -p coverage
================= test session starts =================
platform linux -- Python 3.7.1, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
Django settings: xxxx.settings.dev_local (from environment variable)
rootdir: /srv/xxxx, inifile: setup.cfg
plugins: django-3.4.6, cov-2.6.1, pysoa-0.56.0
collected 256 items
...
xxxx/settings/__init__.py                           36     36      0      0     0.0%   1-135
xxxx/settings/dev_local.py                          33     33      6      0     0.0%   1-125
xxxx/settings/jenkins_test.py                       24     24      4      0     0.0%   1-131
xxxx/settings/production_docker.py                  23     23      9      0     0.0%   1-125
...

# gives an error about its already being loaded
$ pytest -p pytest_cov.plugin
Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 11, in <module>
    sys.exit(main())
...
ValueError: Plugin already registered: pytest_cov=<module 'pytest_cov.plugin' from '/usr/local/lib/python3.7/site-packages/pytest_cov/plugin.py'>
{'140434823318496': <_pytest.config.PytestPluginManager object at 0x7fb987c20be0>, 'pytestconfig': <_pytest.config.Config object at 0x7fb98498c5c0> ......

I'm not sure if this is a regression or if this command-line argument has always been mis-documented and wasn't actually intended for early-loading plugins. Either way, it's a bug. The -p argument conflicts with the entrypoints plugin loading system and there is no way to early-load a plugin unless it doesn't have an entrypoint.

[RonnyPfannschmidt]

it never worked in tune with entry-point names

[nicoddemus]

Strange, it warrants investigation, the code seems correct:

pytest/src/_pytest/config/__init__.py

Lines 780 to 784 in a945734

	self.pluginmanager.consider_preparse(args)
	if not os.environ.get("PYTEST_DISABLE_PLUGIN_AUTOLOAD"):
	# Don't autoload from setuptools entry point. Only explicitly specified
	# plugins are going to be loaded.
	self.pluginmanager.load_setuptools_entrypoints("pytest11")

(consider_preparse loads based on -p)

[nickwilliams-eventbrite]

A full stack trace might help you here, if you haven't seen one already. I think consider_preparse is working. It does successfully load the plugin. The problem is that load_setuptools_entrypoints then fails because the plugin is already loaded:

$ pytest -p pytest_cov.plugin
Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 61, in main
    config = _prepareconfig(args, plugins)
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 196, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/usr/local/lib/python3.7/site-packages/pluggy/hooks.py", line 284, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 68, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 62, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/usr/local/lib/python3.7/site-packages/_pytest/helpconfig.py", line 93, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 652, in pytest_cmdline_parse
    self.parse(args)
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 838, in parse
    self._preparse(args, addopts=addopts)
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 784, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 276, in load_setuptools_entrypoints
    self.register(plugin, name=ep.name)
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 311, in register
    ret = super(PytestPluginManager, self).register(plugin, name)
  File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 81, in register
    % (plugin_name, plugin, self._name2plugin)
ValueError: Plugin already registered: pytest_cov=<module 'pytest_cov.plugin' from '/usr/local/lib/python3.7/site-packages/pytest_cov/plugin.py'>
{'140500909530360': <_pytest.config.PytestPluginManager object at 0x7fc8eacd9cf8>, 'pytestconfig': <_pytest.config.Config object at 0x7fc8e6ec30b8>, 'mark': <module '_pytest.mark' from '/usr/local/lib/python3.7/site-packages/_pytest/mark/__init__.py'>, 'main': <module '_pytest.main' from '/usr/local/lib/python3.7/site-packages/_pytest/main.py'>, 'terminal': <module '_pytest.terminal' from '/usr/local/lib/python3.7/site-packages/_pytest/terminal.py'>, 'runner': <module '_pytest.runner' from '/usr/local/lib/python3.7/site-packages/_pytest/runner.py'>, 'python': <module '_pytest.python' from '/usr/local/lib/python3.7/site-packages/_pytest/python.py'>, 'fixtures': <module '_pytest.fixtures' from '/usr/local/lib/python3.7/site-packages/_pytest/fixtures.py'>, 'debugging': <module '_pytest.debugging' from '/usr/local/lib/python3.7/site-packages/_pytest/debugging.py'>, 'unittest': <module '_pytest.unittest' from '/usr/local/lib/python3.7/site-packages/_pytest/unittest.py'>, 'capture': <module '_pytest.capture' from '/usr/local/lib/python3.7/site-packages/_pytest/capture.py'>, 'skipping': <module '_pytest.skipping' from '/usr/local/lib/python3.7/site-packages/_pytest/skipping.py'>, 'tmpdir': <module '_pytest.tmpdir' from '/usr/local/lib/python3.7/site-packages/_pytest/tmpdir.py'>, 'monkeypatch': <module '_pytest.monkeypatch' from '/usr/local/lib/python3.7/site-packages/_pytest/monkeypatch.py'>, 'recwarn': <module '_pytest.recwarn' from '/usr/local/lib/python3.7/site-packages/_pytest/recwarn.py'>, 'pastebin': <module '_pytest.pastebin' from '/usr/local/lib/python3.7/site-packages/_pytest/pastebin.py'>, 'helpconfig': <module '_pytest.helpconfig' from '/usr/local/lib/python3.7/site-packages/_pytest/helpconfig.py'>, 'nose': <module '_pytest.nose' from '/usr/local/lib/python3.7/site-packages/_pytest/nose.py'>, 'assertion': <module '_pytest.assertion' from '/usr/local/lib/python3.7/site-packages/_pytest/assertion/__init__.py'>, 'junitxml': <module '_pytest.junitxml' from '/usr/local/lib/python3.7/site-packages/_pytest/junitxml.py'>, 'resultlog': <module '_pytest.resultlog' from '/usr/local/lib/python3.7/site-packages/_pytest/resultlog.py'>, 'doctest': <module '_pytest.doctest' from '/usr/local/lib/python3.7/site-packages/_pytest/doctest.py'>, 'cacheprovider': <module '_pytest.cacheprovider' from '/usr/local/lib/python3.7/site-packages/_pytest/cacheprovider.py'>, 'freeze_support': <module '_pytest.freeze_support' from '/usr/local/lib/python3.7/site-packages/_pytest/freeze_support.py'>, 'setuponly': <module '_pytest.setuponly' from '/usr/local/lib/python3.7/site-packages/_pytest/setuponly.py'>, 'setupplan': <module '_pytest.setupplan' from '/usr/local/lib/python3.7/site-packages/_pytest/setupplan.py'>, 'stepwise': <module '_pytest.stepwise' from '/usr/local/lib/python3.7/site-packages/_pytest/stepwise.py'>, 'warnings': <module '_pytest.warnings' from '/usr/local/lib/python3.7/site-packages/_pytest/warnings.py'>, 'logging': <module '_pytest.logging' from '/usr/local/lib/python3.7/site-packages/_pytest/logging.py'>, 'pytest_cov.plugin': <module 'pytest_cov.plugin' from '/usr/local/lib/python3.7/site-packages/pytest_cov/plugin.py'>, 'django': <module 'pytest_django.plugin' from '/usr/local/lib/python3.7/site-packages/pytest_django/plugin.py'>}

[nicoddemus]

Oh OK, that makes sense now, thanks!

[nickwilliams-eventbrite]

👍 Glad I could help

[nicoddemus]

it never worked in tune with entry-point names

After looking at the code I see what you mean. 😅

Took a stab at a solution in #4727

[blueyed]

Fixed in features branch, closing.

[rohitkurangi]

okk

[nickwilliams-eventbrite]

Just wanted to post here and confirm that when I updated PyTest, -p is now working for me like it's supposed to. Thanks for fixing it so quickly!