[BUG]Why does adding a module produce F6401

[whg517]

Describe the bug

Can you explain the prompt 'F6401' when I run pylint pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (Can-enumerate-pytest-fixtures) is the reason?

I would like to know how it works, or why it appears, and sometimes has different outputs. The same code, sometimes two, sometimes more. I was depressed.

I did run pytest --fixtures --collect-only without any unusual hints and my tests were normal.

Description:

After I fine-tune my existing code, including running pylint, pytest, and isort, everything works. I added a new package executor with three modules, one is the abstract module of base.py, two are corresponding to different implementation modules(local.py, docker.py).

Then I run isort, and pylint works fine

Then I import the base class and two implementation classes in the module's __init__.py file, and add a factory method.

When I run pylint again, the input tells me that some of the test modules have problems with F6401.

Again, I want to emphasize that everything was fine until I added this module. But now I just added the source code of this module, this exception will appear.

What makes it even more confusing to me is that the module I'm prompted doesn't include any fixtures. I ran pylint again and found that F6401 has more test modules (several times more than last time).

I've been using PyLint for a new project to check for a mode-by-module migration, and when I migrate to this module, I can't continue.

To Reproduce

Package versions

• pylint 3.0.0a3
• pylint-pytest 1.1.2
• pyparsing 2.4.7
• pytest 6.2.3
• pytest-asyncio 0.14.0
• pytest-cov 2.11.1
• pytest-mock 3.5.1

(add any relevant pylint/pytest plugin here)

load-plugins = pylint_pytest,

Folder structure

.
├── demo.py
├── dist
├── Pipfile
├── Pipfile.lock
├── README.md
├── setup.cfg
├── setup.py
├── src
│   ├── crawlerstack_spiderkeeper
│   │   ├── alembic
│   │   │   ├── alembic.ini
│   │   │   ├── env.py
│   │   │   ├── README
│   │   │   ├── script.py.mako
│   │   │   └── versions
│   │   │       ├── 17feeb1da886_init.py
│   │   │       ├── dae2b963c44f_.py
│   │   │       └── __init__.py
│   │   ├── cmdline.py
│   │   ├── config
│   │   │   ├── __init__.py
│   │   │   ├── settings.yaml
│   │   │   └── settings.yml
│   │   ├── dao
│   │   │   ├── artifact.py
│   │   │   ├── base.py
│   │   │   ├── __init__.py
│   │   │   ├── job.py
│   │   │   ├── project.py
│   │   │   ├── server.py
│   │   │   ├── storage.py
│   │   │   └── task.py
│   │   ├── db
│   │   │   ├── __init__.py
│   │   │   ├── models.py
│   │   ├── executor
│   │   │   ├── base.py
│   │   │   ├── docker.py
│   │   │   ├── __init__.py
│   │   │   ├── local.py
│   │   │   └── subprocess.py
│   │   ├── __init__.py
│   │   ├── schedule
│   │   │   ├── __init__.py
│   │   │   └── scheduler.py
│   │   ├── schemas
│   │   │   ├── artifact.py
│   │   │   ├── audit.py
│   │   │   ├── base.py
│   │   │   ├── __init__.py
│   │   │   ├── job.py
│   │   │   ├── project.py
│   │   │   ├── schedule.py
│   │   │   ├── server.py
│   │   │   ├── storage.py
│   │   │   └── task.py
│   │   ├── signals.py
│   │   └── utils
│   │       ├── constants.py
│   │       ├── exceptions.py
│   │       ├── __init__.py
│   │       ├── log.py
│   │       ├── metadata.py
│   │       ├── states.py
│   │       ├── types.py
│   │       └── virtualenv.py
│   └── crawlerstack_spiderkeeper.egg-info
│       ├── dependency_links.txt
│       ├── entry_points.txt
│       ├── not-zip-safe
│       ├── PKG-INFO
│       ├── requires.txt
│       ├── SOURCES.txt
│       └── top_level.txt
├── tests
│   ├── conftest.py
│   ├── dao
│   │   ├── __init__.py
│   │   ├── test_artifact.py
│   │   ├── test_base.py
│   │   ├── test_job.py
│   │   ├── test_project.py
│   │   ├── test_server.py
│   │   ├── test_storage.py
│   │   └── test_task.py
│   ├── data
│   │   ├── agpl-3.0.txt
│   │   └── demo
│   │       ├── demo
│   │       │   ├── extensions.py
│   │       │   ├── __init__.py
│   │       │   ├── items.py
│   │       │   ├── middlewares.py
│   │       │   ├── mock.py
│   │       │   ├── settings.py
│   │       │   └── spiders
│   │       │       ├── example.py
│   │       │       └── __init__.py
│   │       ├── Dockerfile
│   │       ├── Pipfile
│   │       ├── Pipfile.lock
│   │       ├── requirements.txt
│   │       ├── scrapy.cfg
│   │       └── setup.py
│   ├── __init__.py
│   ├── test_db.py
│   └── utils
│       ├── artifacts
│       │   ├── artifacts
│       │   ├── virtualenvs
│       │   └── workspaces
│       ├── __init__.py
│       ├── test_log.py
│       ├── test_metadata.py
│       └── tests.py
└── tox.ini

File content :

When I start this code, I get F6401, and when I comment it out, everything is fine.

crawlerstack_spiderkeeper.executor::__init__.py

"""
Executor API
"""
from typing import Type

from crawlerstack_spiderkeeper.config import settings
from crawlerstack_spiderkeeper.executor.base import BaseExecutor
from crawlerstack_spiderkeeper.executor.docker import DockerExecutor
from crawlerstack_spiderkeeper.executor.local import LocalExecutor
from crawlerstack_spiderkeeper.utils.exceptions import SpiderkeeperError


def executor_factory() -> Type[BaseExecutor]:
    """
    Executor factory.
    Create executor from settings.EXECUTOR
    :return:
    """
    executor = str(settings.EXECUTOR).lower()
    if executor == 'local':
        executor_kls = LocalExecutor
    elif executor == 'docker':
        executor_kls = DockerExecutor
    else:
        raise SpiderkeeperError('Invalid executor.')

    return executor_kls


__all__ = [
    'DockerExecutor',
    'LocalExecutor',
    'executor_factory'
]

pylint output with the plugin :

Why is it different before and after?

❯ pylint src tests
************* Module tests.dao.test_base
tests/dao/test_base.py:1:0: F6401: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (cannot-enumerate-pytest-fixtures)

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

❯ 
❯ pylint src tests
************* Module tests.test_db
tests/test_db.py:1:0: F6401: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (cannot-enumerate-pytest-fixtures)
************* Module tests.utils.test_metadata
tests/utils/test_metadata.py:1:0: F6401: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (cannot-enumerate-pytest-fixtures)
************* Module tests.utils.test_log
tests/utils/test_log.py:1:0: F6401: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (cannot-enumerate-pytest-fixtures)
************* Module tests.dao.test_task
tests/dao/test_task.py:1:0: F6401: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (cannot-enumerate-pytest-fixtures)
************* Module tests.dao.test_artifact
tests/dao/test_artifact.py:1:0: F6401: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (cannot-enumerate-pytest-fixtures)
************* Module tests.dao.test_job
tests/dao/test_job.py:1:0: F6401: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (cannot-enumerate-pytest-fixtures)
************* Module tests.dao.test_server
tests/dao/test_server.py:1:0: F6401: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (cannot-enumerate-pytest-fixtures)
************* Module tests.dao.test_project
tests/dao/test_project.py:1:0: F6401: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (cannot-enumerate-pytest-fixtures)
************* Module tests.dao.test_storage
tests/dao/test_storage.py:1:0: F6401: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (cannot-enumerate-pytest-fixtures)
************* Module tests.dao.test_base
tests/dao/test_base.py:1:0: F6401: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (cannot-enumerate-pytest-fixtures)

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

(Optional) pytest output from fixture collection

❯ pytest --fixtures --collect-only
======================================================================================================================================================================== test session starts ========================================================================================================================================================================
platform linux -- Python 3.7.3, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: /home/kevin/workspaces/develop/python/crawlerstack/crawlerstack-spiderkeeper, configfile: setup.cfg, testpaths: tests
plugins: asyncio-0.14.0, cov-2.11.1, mock-3.5.1
collected 60 items                                                                                                                                                                                                                                                                                                                                                  

<Package tests>
  <Module test_db.py>
    <Function test_migrate>
    <Function test_db>
<Package dao>
  <Module test_artifact.py>
    <Function test_get_artifact_from_job_id>
    <Function test_get_project_of_artifacts[True]>
    <Function test_get_project_of_artifacts[False]>
  <Module test_base.py>
    <Function test_get>
    <Function test_get_not_exist>
    <Function test_get_multi_not_exist>
    <Function test_get_multi[None-None-gt]>
    <Function test_get_multi[DESC-None-gt]>
    <Function test_get_multi[ASC-None-lt]>
    <Function test_get_multi[None-datetime-gt]>
    <Function test_get_multi[None-foo-SpiderkeeperError]>
    <Function test_create>
    <Function test_update[True]>
    <Function test_update[False]>
    <Function test_update_by_id[True]>
    <Function test_update_by_id[False]>
    <Function test_delete>
    <Function test_delete_by_id[True]>
    <Function test_delete_by_id[False]>
    <Function test_delete_by_id_obj_not_exist>
    <Function test_count>
  <Module test_job.py>
    <Function test_job_state[True]>
    <Function test_job_state[False]>
  <Module test_project.py>
    <Function test_create>
  <Module test_server.py>
    <Function test_get_server_by_job_id>
  <Module test_storage.py>
    <Function test_increase_storage_count>
    <Function test_running_storage>
  <Module test_task.py>
    <Function test_get_running>
    <Function test_count_running_task>
    <Function test_increase_item_count>
<Package utils>
  <Module test_log.py>
    <Function test_verbose_formatter[True-verbose]>
    <Function test_verbose_formatter[False-simple]>
    <Function test_log_level[True-DEBUG-DEBUG]>
    <Function test_log_level[True-INFO-DEBUG]>
    <Function test_log_level[True-ERROR-DEBUG]>
    <Function test_log_level[False-DEBUG-DEBUG]>
    <Function test_log_level[False-INFO-INFO]>
  <Module test_metadata.py>
    <Function test>
    <Function test_from_project>
  <Module tests.py>
    <Function test_run_in_executor>
    <Function test_app_data[APPID-1-1-expect_value0]>
    <Function test_app_data[app_id1-expect_value1]>
    <Function test_common_query_params[None-None-None-None-expect_value0]>
    <Function test_common_query_params[None-100-None-None-expect_value1]>
    <Function test_common_query_params[90-None-None-None-expect_value2]>
    <Function test_common_query_params[90-100-None-None-expect_value3]>
    <Function test_common_query_params[50-100-None-None-expect_value4]>
    <Function test_get_host_address>
    <Function test_upload>
    <Function test_init_app_id_from_str>
    <Function test_init_app_id>
    <Function test_app_id_eq>
    <Function test_kill_proc_tree>
    <Function test_staging_path>
    <Function test_head>
    <Function test_last>
    <Function test_tail[2048-22]>
    <Function test_tail[100-4]>
cache
    Return a cache object that can persist state between testing sessions.
    
    cache.get(key, default)
    cache.set(key, value)
    
    Keys must be ``/`` separated strings, where the first part is usually the
    name of your plugin or application to avoid clashes with other cache users.
    
    Values can be any object handled by the json stdlib module.

capsys
    Enable text capturing of writes to ``sys.stdout`` and ``sys.stderr``.
    
    The captured output is made available via ``capsys.readouterr()`` method
    calls, which return a ``(out, err)`` namedtuple.
    ``out`` and ``err`` will be ``text`` objects.

capsysbinary
    Enable bytes capturing of writes to ``sys.stdout`` and ``sys.stderr``.
    
    The captured output is made available via ``capsysbinary.readouterr()``
    method calls, which return a ``(out, err)`` namedtuple.
    ``out`` and ``err`` will be ``bytes`` objects.

capfd
    Enable text capturing of writes to file descriptors ``1`` and ``2``.
    
    The captured output is made available via ``capfd.readouterr()`` method
    calls, which return a ``(out, err)`` namedtuple.
    ``out`` and ``err`` will be ``text`` objects.

capfdbinary
    Enable bytes capturing of writes to file descriptors ``1`` and ``2``.
    
    The captured output is made available via ``capfd.readouterr()`` method
    calls, which return a ``(out, err)`` namedtuple.
    ``out`` and ``err`` will be ``byte`` objects.

doctest_namespace [session scope]
    Fixture that returns a :py:class:`dict` that will be injected into the
    namespace of doctests.

pytestconfig [session scope]
    Session-scoped fixture that returns the :class:`_pytest.config.Config` object.
    
    Example::
    
        def test_foo(pytestconfig):
            if pytestconfig.getoption("verbose") > 0:
                ...

record_property
    Add extra properties to the calling test.
    
    User properties become part of the test report and are available to the
    configured reporters, like JUnit XML.
    
    The fixture is callable with ``name, value``. The value is automatically
    XML-encoded.
    
    Example::
    
        def test_function(record_property):
            record_property("example_key", 1)

record_xml_attribute
    Add extra xml attributes to the tag for the calling test.
    
    The fixture is callable with ``name, value``. The value is
    automatically XML-encoded.

record_testsuite_property [session scope]
    Record a new ``<property>`` tag as child of the root ``<testsuite>``.
    
    This is suitable to writing global information regarding the entire test
    suite, and is compatible with ``xunit2`` JUnit family.
    
    This is a ``session``-scoped fixture which is called with ``(name, value)``. Example:
    
    .. code-block:: python
    
        def test_foo(record_testsuite_property):
            record_testsuite_property("ARCH", "PPC")
            record_testsuite_property("STORAGE_TYPE", "CEPH")
    
    ``name`` must be a string, ``value`` will be converted to a string and properly xml-escaped.
    
    .. warning::
    
        Currently this fixture **does not work** with the
        `pytest-xdist <https://github.com/pytest-dev/pytest-xdist>`__ plugin. See issue
        `#7767 <https://github.com/pytest-dev/pytest/issues/7767>`__ for details.

caplog
    Access and control log capturing.
    
    Captured logs are available through the following properties/methods::
    
    * caplog.messages        -> list of format-interpolated log messages
    * caplog.text            -> string containing formatted log output
    * caplog.records         -> list of logging.LogRecord instances
    * caplog.record_tuples   -> list of (logger_name, level, message) tuples
    * caplog.clear()         -> clear captured records and formatted log output string

monkeypatch
    A convenient fixture for monkey-patching.
    
    The fixture provides these methods to modify objects, dictionaries or
    os.environ::
    
        monkeypatch.setattr(obj, name, value, raising=True)
        monkeypatch.delattr(obj, name, raising=True)
        monkeypatch.setitem(mapping, name, value)
        monkeypatch.delitem(obj, name, raising=True)
        monkeypatch.setenv(name, value, prepend=False)
        monkeypatch.delenv(name, raising=True)
        monkeypatch.syspath_prepend(path)
        monkeypatch.chdir(path)
    
    All modifications will be undone after the requesting test function or
    fixture has finished. The ``raising`` parameter determines if a KeyError
    or AttributeError will be raised if the set/deletion operation has no target.

recwarn
    Return a :class:`WarningsRecorder` instance that records all warnings emitted by test functions.
    
    See http://docs.python.org/library/warnings.html for information
    on warning categories.

tmpdir_factory [session scope]
    Return a :class:`_pytest.tmpdir.TempdirFactory` instance for the test session.

tmp_path_factory [session scope]
    Return a :class:`_pytest.tmpdir.TempPathFactory` instance for the test session.

tmpdir
    Return a temporary directory path object which is unique to each test
    function invocation, created as a sub directory of the base temporary
    directory.
    
    By default, a new base temporary directory is created each test session,
    and old bases are removed after 3 sessions, to aid in debugging. If
    ``--basetemp`` is used then it is cleared each session. See :ref:`base
    temporary directory`.
    
    The returned object is a `py.path.local`_ path object.
    
    .. _`py.path.local`: https://py.readthedocs.io/en/latest/path.html

tmp_path
    Return a temporary directory path object which is unique to each test
    function invocation, created as a sub directory of the base temporary
    directory.
    
    By default, a new base temporary directory is created each test session,
    and old bases are removed after 3 sessions, to aid in debugging. If
    ``--basetemp`` is used then it is cleared each session. See :ref:`base
    temporary directory`.
    
    The returned object is a :class:`pathlib.Path` object.


------------------------------------------------------------------------------------------------------------------------------------------------------------ fixtures defined from pytest_asyncio.plugin ------------------------------------------------------------------------------------------------------------------------------------------------------------
event_loop
    Create an instance of the default event loop for each test case.

unused_tcp_port
    /home/kevin/.local/share/virtualenvs/crawlerstack-spiderkeeper-wbKs1mAt/lib/python3.7/site-packages/pytest_asyncio/plugin.py:221: no docstring available

unused_tcp_port_factory
    A factory function, producing different unused TCP ports.


-------------------------------------------------------------------------------------------------------------------------------------------------------------- fixtures defined from pytest_cov.plugin --------------------------------------------------------------------------------------------------------------------------------------------------------------
no_cover
    A pytest fixture to disable coverage.

cov
    A pytest fixture to provide access to the underlying coverage object.


------------------------------------------------------------------------------------------------------------------------------------------------------------- fixtures defined from pytest_mock.plugin --------------------------------------------------------------------------------------------------------------------------------------------------------------
class_mocker [class scope]
    Return an object that has the same interface to the `mock` module, but
    takes care of automatically undoing all patches after each test method.

mocker
    Return an object that has the same interface to the `mock` module, but
    takes care of automatically undoing all patches after each test method.

module_mocker [module scope]
    Return an object that has the same interface to the `mock` module, but
    takes care of automatically undoing all patches after each test method.

package_mocker [package scope]
    Return an object that has the same interface to the `mock` module, but
    takes care of automatically undoing all patches after each test method.

session_mocker [session scope]
    Return an object that has the same interface to the `mock` module, but
    takes care of automatically undoing all patches after each test method.


--------------------------------------------------------------------------------------------------------------------------------------------------------------- fixtures defined from tests.conftest ----------------------------------------------------------------------------------------------------------------------------------------------------------------
session_factory [session scope]
    Session factory fixture

uploaded_file
    Uploaded file fixture.

migrate
    migrate fixture.

session
    Session fixture.

init_audit
    Init audit fixture.

init_server
    Init server fixture.

init_project
    Init project fixture.

init_artifact
    Init artifact fixture.

init_job
    Init job fixture.

init_task
    Init task fixture.

init_storage
    Init storage fixture.

temp_dir
    初始化测试目录，同时将测试目录赋值到 settings 上
    因为测试中所有内容都会放在这个目录，所以在引用 settings 时除非没有引用这个 fixture
    或者使用了这个 fixture 的其他 fixture，否则 settings.ARTIFACT_PATH 都将返回测试目录
    :return:

base_dir
    Base dir fixture.

data_dir
    Test data dir fixture.

demo_zip
    只用 test/data/demo 构建的测试数据
    压缩文件中的目录结构
    xxx-20191215152202.zip
    .
    |-- xxx
    |    |- xxx
    |    |    |- spiders
    |    |    |    |- __init__.py
    |    |    |    └── xxx.py
    |    |    |- settings.py
    |    |    |- middlewares.py
    |    |    └── items.py
    |    |- Dockerfile
    |    |- requirements.txt
    |    |- Pipfile
    |    |- Pipfile.lock
    |    |- setup.py
    |    |- setup.cfg
    |    |- scrapy.cfg
    |    └── README.md


-------------------------------------------------------------------------------------------------------------------------------------------------------------- fixtures defined from tests.utils.tests --------------------------------------------------------------------------------------------------------------------------------------------------------------
demo_file
    Fixture demo file

tail
    Fixture tail


------------------------------------------------------------------------------------------------------------------------------------------------------------- fixtures defined from tests.dao.test_base -------------------------------------------------------------------------------------------------------------------------------------------------------------
dao
    dao fixture


==================================================================================================================================================================== 60 tests collected in 0.09s ====================================================================================================================================================================

Expected behavior

Everything is ok

Additional context
Add any other context about the problem here.

[whg517]

After debugging the source code, I found out that the cause of my problems was an error in pylint-pytest when running pytest to collect fixtures from source code, and then pylint-pytest passed the error to PyLint.

My source code had a type annotation error that caused pytest to look for a fixture from that module that was wrong, and the error was passed to pylint. But why there is a different output is not clear to me.

From debugging the source code, we know that pylint-pytest registers itself with pylint, and when pylint checks all files, it passes the files to pylint-pytest's FixtureChecker.

pylint-pytest/pylint_pytest/checkers/fixture.py

Lines 92 to 142 in 6267638

	def visit_module(self, node):
	'''
	- only run once per module
	- invoke pytest session to collect available fixtures
	- create containers for the module to store args and fixtures
	'''
	# storing all fixtures discovered by pytest session
	FixtureChecker._pytest_fixtures = {} # Dict[List[_pytest.fixtures.FixtureDef]]
	# storing all used function arguments
	FixtureChecker._invoked_with_func_args = set() # Set[str]
	# storing all invoked fixtures through @pytest.mark.usefixture(...)
	FixtureChecker._invoked_with_usefixtures = set() # Set[str]
	is_test_module = False
	for pattern in FILE_NAME_PATTERNS:
	if fnmatch.fnmatch(Path(node.file).name, pattern):
	is_test_module = True
	break
	try:
	with open(os.devnull, 'w') as devnull:
	# suppress any future output from pytest
	stdout, stderr = sys.stdout, sys.stderr
	sys.stderr = sys.stdout = devnull
	# run pytest session with customized plugin to collect fixtures
	fixture_collector = FixtureCollector()
	# save and restore sys.path to prevent pytest.main from altering it
	sys_path = sys.path.copy()
	ret = pytest.main(
	[
	node.file, '--fixtures', '--collect-only',
	'--pythonwarnings=ignore:Module already imported:pytest.PytestWarning',
	],
	plugins=[fixture_collector],
	)
	# restore sys.path
	sys.path = sys_path
	FixtureChecker._pytest_fixtures = fixture_collector.fixtures
	if (ret != pytest.ExitCode.OK or fixture_collector.errors) and is_test_module:
	self.add_message('cannot-enumerate-pytest-fixtures', node=node)
	finally:
	# restore output devices
	sys.stdout, sys.stderr = stdout, stderr

The visit_module method in the FixtureChecker passes the file to pytest, running pytest <module_file> --fixtures --collect-only, At the same time load the FixtureCollector plug-in into pytest.

pylint-pytest/pylint_pytest/checkers/fixture.py

Lines 125 to 131 in 6267638

	ret = pytest.main(
	[
	node.file, '--fixtures', '--collect-only',
	'--pythonwarnings=ignore:Module already imported:pytest.PytestWarning',
	],
	plugins=[fixture_collector],
	)

In pytest_collectreport , if an error is reported by pytest, it is logged and the error information is passed to pytest.

pylint-pytest/pylint_pytest/checkers/fixture.py

Lines 24 to 34 in 6267638

	class FixtureCollector:
	fixtures = {}
	errors = set()
	def pytest_sessionfinish(self, session):
	# pylint: disable=protected-access
	self.fixtures = session._fixturemanager._arg2fixturedefs
	def pytest_collectreport(self, report):
	if report.failed:
	self.errors.add(report)

I don't think this logic makes sense. Pytest should only collect fixtures from the test modules, and instead of collecting fixtures from all modules, Pylint-Pytest should filter out the source code when PyLint checks.

Hope to fix this problem.

[miketheman]

I'm seeing this today as well.

Curious about this statement:

Pytest should only collect fixtures from the test modules, and instead of collecting fixtures from all modules, Pylint-Pytest should filter out the source code when PyLint checks.

That makes sense to me, because I don't think I'd want to put fixtures in source code, but how would pylint (and subsequently pytest) know the difference between source and test code?

[whg517]

Hi @miketheman

In pylint-pytest logic, pylint-pytest or pytest does not know the difference between the source code and the test code. Because pylint checks all files, for example pylint src tests checks all python files in both directories. So pylint-pytest simply passes the file to
pytest that pytest needs to check and lets pytest collect fixtures in that file.

If aone source file is bad small, pytest will error.I had these problems because one of the methods in the source code had the wrong return value type. However, it should be indicated exactly by pylint, but the information above obviously doesn't pinpoint exactly where the problem is and what caused it. It just tells you that the loading fixture is wrong. And it has nothing to do with the wrong file.

If we only talk about pytest, pytest will load its own configuration before running, such as from pytest.ini or setup.cfg, depending on pytest itself. For example, I configured it in setup.cfg

[tool:pytest]
testpaths = tests
python_files = tests.py test_*.py

Then pytest will only look for the test code in the tests directory. It also filters the non-test source code files in the directory.

From this point on, if I fix this problem, I have a hypothesis:

Refer to pytest's filter logic for test files and filter out any logic that does not conform to the rules before passing the files to pytest for inspection.

Since I was having conflicts with dynaconf when using the pylint plugin pylint-django, I lowered my detection requirements and switched to the less stringent flake8 detection code. And that's just one of my current alternatives.

[miketheman]

@whg517 Thanks for the excellent explanation! This appears similar to #16 and #20 - so possibly if this is fixed, all three will be resolved.

[webknjaz]

Refer to pytest's filter logic for test files and filter out any logic that does not conform to the rules before passing the files to pytest for inspection.

I did some debugging and came to a similar conclusion: if there's a way for the plugin to skip non-pytest files altogether, it'd fix the problem. Maybe even w/o taping into _pytest this plugin could introduce a wildcard setting for including/excluding files?

P.S. In my case, the underlying collection error was caused by the fact that the project uses implicit namespaces and so feeding full file path to modules confuses Python when it tries to resolve relative imports (because it doesn't know what the root package is). It is easy to reproduce once you run the proper command (in this case python -m pytest --fixtures --collect-only octomachinery/github/models/events.py):

=========================================================================== ERRORS ===========================================================================
___________________________________________________ ERROR collecting octomachinery/github/models/events.py ___________________________________________________
octomachinery/github/models/events.py:17: in <module>
    from ...utils.asynctools import aio_gather
E   ImportError: attempted relative import beyond top-level package
        Any        = typing.Any
        Iterable   = typing.Iterable
        Mapping    = typing.Mapping
        TYPE_CHECKING = False
        TextIO     = <class 'typing.TextIO'>
        Type       = typing.Type
        Union      = typing.Union
        _GidgetHubEvent = <class 'gidgethub.sansio.Event'>
        __builtins__ = <builtins>
        __cached__ = '/home/runner/work/octomachinery/octomachinery/octomachinery/github/models/__pycache__/events.cpython-39.pyc'
        __doc__    = 'Generic GitHub event containers.'        __file__   = '/home/runner/work/octomachinery/octomachinery/octomachinery/github/models/events.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7ff8f7a0ab80>
        __name__   = 'models.events'
        __package__ = 'models'
        __spec__   = ModuleSpec(name='models.events', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7ff8f7a0ab80>, origin='/home/runner/work
/octomachinery/octomachinery/octomachinery/github/models/events.py')
        annotations = _Feature((3, 7, 0, 'beta', 1), (3, 10, 0, 'alpha', 0), 16777216)
        attr       = <module 'attr' from '/home/runner/work/octomachinery/octomachinery/.tox/pre-commit/lib/python3.9/site-packages/attr/__init__.py'>
        cast       = <function cast at 0x7ff8f97e5430>
        json       = <module 'json' from '/opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/json/__init__.py'>
        pathlib    = <module 'pathlib' from '/opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/pathlib.py'>
        uuid       = <module 'uuid' from '/opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/uuid.py'>
        warnings   = <module 'warnings' from '/opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/warnings.py'>
___________________________________________________ ERROR collecting octomachinery/github/models/events.py ___________________________________________________
ImportError while importing test module '/home/runner/work/octomachinery/octomachinery/octomachinery/github/models/events.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
octomachinery/github/models/events.py:17: in <module>
    from ...utils.asynctools import aio_gather
E   ImportError: attempted relative import beyond top-level package
================================================================== short test summary info ===================================================================
ERROR octomachinery/github/models/events.py - ImportError: attempted relative import beyond top-level package
ERROR octomachinery/github/models/events.py
=========================================================== no tests collected, 2 errors in 0.09s ============================================================

P.P.S. Another problem is that the error message gives an example with a dummy file path making it hard to debug. I think the message could be improved to specifically mention what file exactly causes the problem, instead of saying it's path/to/current/module.py.

[webknjaz]

This should fix it: #22.

cc @reverbc @ssbarnea

[reverbc]

Pytest should only collect fixtures from the test modules, and instead of collecting fixtures from all modules, Pylint-Pytest should filter out the source code when PyLint checks.

I believe we have other use cases that pytest should care about non-test modules.

Similar to #2 use case, test author can use separate modules (let's call them helper modules) to organize their fixtures/helpers if they have a large and complex test framework. In that case, if there's a coding error in one of the helper modules and causing the pytest failed to enumerate the fixtures, pylint-pytest will not be able to suppress the FP. That was the main intention when I introduced this new warning, and unfortunately the fixes in #22 will not be able to prevent that from happening since it'll ignore the collection errors from helper modules.

I'm now more inclined to temporary remove this cannot-enumerate-pytest-fixtures check if it cannot correctly identify which modules actually contain pytest fixtures until we have a better solution to detect that.

[webknjaz]

I do not think so. Checking the test modules should load dependencies in the background anyway. If they are loaded properly, they'll return the fixtures. If not, they'll cause errors.
The problem with pointing pytest at random files is that those may be loaded in different ways (runpy vs scripts) and depending on whether there's __init__.py the module loader may or may not recognize the package called tests, for example.

So I'm pretty sure that pointing at the test files will identify the missing fixtures one way or another.