Incorrect number of collected items reported when specific class methods are provided

[tonyo]

• Include a detailed description of the bug or suggestion

It seems that the number of collected tests is displayed incorrectly when the "class method" notation (ModuleX::ClassY::MethodZ) is used from the command line. In a simple example (see below) when we want to collect/run only one specific test method from a class, one expects to get the message that only 1 item was collected (collected 1 item). Instead, pytest outputs collected N items, where N equals the total number of test methods collected from ClassY plus one.

• pip list of the virtual environment you are using

appdirs (1.4.3)
packaging (16.8)
pip (9.0.1)
py (1.4.33)
pyparsing (2.2.0)
pytest (3.1.2.dev13+gf826b23, /home/user/reps/pytest_fix/pytest)
setuptools (35.0.2)
six (1.10.0)
wheel (0.30.0a0)

• pytest and operating system versions

pytest version 3.1.2.dev13+gf826b23

Ubuntu 16.04.2, Linux 4.4.0-78-generic x86_64

• Minimal example if possible

# collect.py

class TestClass(object):
    def test_main(self):
        assert 1
    def test_bla(self):
        assert True

$ py.test collect.py::TestClass::test_main --collect-only

Output:

=================== test session starts ======================
platform linux2 -- Python 2.7.12, pytest-3.1.2.dev13+gf826b23, py-1.4.33, pluggy-0.4.0
rootdir: /home/user/reps/pytest_fix, inifile:
collected 3 items 
<Module 'collect.py'>
  <Class 'TestClass'>
    <Instance '()'>
      <Function 'test_main'>

Expected output:

...
collected 1 item 
<Module 'collect.py'>
...

[nicoddemus]

Just to be clear here: the problem is that it should display collected 2 items since there are 2 items in TestClass, althought it will end up running one of them.

[tonyo]

@nicoddemus Is it documented somewhere? I expected --collect-only to return the number of tests that pytest would run if started without --collect-only.

[nicoddemus]

I expected --collect-only to return the number of tests that pytest would run if started without --collect-only.

That's perfectly reasonable to assume, it it is just not how things work currently.

To discover test items from a module, pytest has to import that module. At that point it creates the test items it finds in that namespace, and only then it picks out the test id provided in the command line to execute. It does so because it has to create a test item in order to know its node id (to account for parametrization, hooks, etc, anything which in the end affects the node id), so it might as well create all test items for a module upon discovery.

[tonyo]

@nicoddemus Thank you for the explanation, it makes more sense now.