Added ability to collect test cases by supported topology

[SavchukRomanLv]

Signed-off-by: Roman Savchuk romanx.savchuk@intel.com

Description of PR

Summary:
We do have ability run TC with --topology t0,t1 etc. argument. But if we want to collect which cases support for example t0 topo without execution, option --collect-only always returns all available tests in module/folder. PR resolves this problem.

Type of change

• Bug fix
• [+] Testbed and Framework(new/improvement)
• Test case(new/improvement)

Approach

What is the motivation for this PR?

Can not collect test cases based on topology.

How did you do it?

Enhanced check_topology method.
Added pytest_collection_modifyitems hook.

How did you verify/test it?

test_mark.py

import pytest

pytestmark = [
    pytest.mark.topology('t1')
]

@pytest.fixture(scope="module", params=["param1_value1", "param1_value2"])
def param1(request):
    return request.param

def test_case1_1(param1):
    assert True

@pytest.mark.topology('t0-64')
def test_case1_2(param1):
    assert True

@pytest.mark.topology('t1')
def test_case1_3(param1):
    assert True

@pytest.mark.topology('any')
def test_case1_4(param1):
    assert True

@pytest.mark.parametrize("param2", ["param2_value1", "param2_value2"])
def test_case2_1(param2):
    assert True

@pytest.mark.topology('t0')
@pytest.mark.parametrize("param2", ["param2_value1", "param2_value2"])
def test_case2_2(param2):
    assert True

@pytest.mark.topology('t1-lag')
@pytest.mark.parametrize("param2", ["param2_value1", "param2_value2"])
def test_case2_3(param2):
    assert True

@pytest.mark.topology('any')
@pytest.mark.parametrize("param2", ["param2_value1", "param2_value2"])
def test_case2_4(param2):
    assert True

def test_case3_1():
    assert True

@pytest.mark.topology('tx')
def test_case3_2():
    assert True

@pytest.mark.topology('tx')
def test_case3_3():
    assert True

@pytest.mark.topology('tx')
def test_case3_4():
    assert True

pytest test_marks.py --topology any  -v -rA 
============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.5.2, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/rsavchuk/marks, configfile: pytest.ini
collected 20 items                                                                                                                                                                                                                          

test_marks.py::test_case1_1[param1_value1] SKIPPED                                                                                                                                                                                    [  5%]
test_marks.py::test_case1_2[param1_value1] SKIPPED                                                                                                                                                                                    [ 10%]
test_marks.py::test_case1_3[param1_value1] SKIPPED                                                                                                                                                                                    [ 15%]
test_marks.py::test_case1_4[param1_value1] PASSED                                                                                                                                                                                     [ 20%]
test_marks.py::test_case1_1[param1_value2] SKIPPED                                                                                                                                                                                    [ 25%]
test_marks.py::test_case1_2[param1_value2] SKIPPED                                                                                                                                                                                    [ 30%]
test_marks.py::test_case1_3[param1_value2] SKIPPED                                                                                                                                                                                    [ 35%]
test_marks.py::test_case1_4[param1_value2] PASSED                                                                                                                                                                                     [ 40%]
test_marks.py::test_case2_1[param2_value1] SKIPPED                                                                                                                                                                                    [ 45%]
test_marks.py::test_case2_1[param2_value2] SKIPPED                                                                                                                                                                                    [ 50%]
test_marks.py::test_case2_2[param2_value1] SKIPPED                                                                                                                                                                                    [ 55%]
test_marks.py::test_case2_2[param2_value2] SKIPPED                                                                                                                                                                                    [ 60%]
test_marks.py::test_case2_3[param2_value1] SKIPPED                                                                                                                                                                                    [ 65%]
test_marks.py::test_case2_3[param2_value2] SKIPPED                                                                                                                                                                                    [ 70%]
test_marks.py::test_case2_4[param2_value1] PASSED                                                                                                                                                                                     [ 75%]
test_marks.py::test_case2_4[param2_value2] PASSED                                                                                                                                                                                     [ 80%]
test_marks.py::test_case3_1 SKIPPED                                                                                                                                                                                                   [ 85%]
test_marks.py::test_case3_2 SKIPPED                                                                                                                                                                                                   [ 90%]
test_marks.py::test_case3_3 SKIPPED                                                                                                                                                                                                   [ 95%]
test_marks.py::test_case3_4 SKIPPED                                                                                                                                                                                                   [100%]

================================================================================================================== PASSES ===================================================================================================================
========================================================================================================== short test summary info ==========================================================================================================
PASSED test_marks.py::test_case1_4[param1_value1]
PASSED test_marks.py::test_case1_4[param1_value2]
PASSED test_marks.py::test_case2_4[param2_value1]
PASSED test_marks.py::test_case2_4[param2_value2]
SKIPPED [1] test_marks.py:48: test requires topology in Mark(name='topology', args=('tx',), kwargs={})
SKIPPED [2] test_marks.py:14: test requires topology in Mark(name='topology', args=('t0-64',), kwargs={})
SKIPPED [2] test_marks.py:26: test requires topology in Mark(name='topology', args=('t1',), kwargs={})
SKIPPED [1] test_marks.py:56: test requires topology in Mark(name='topology', args=('tx',), kwargs={})
SKIPPED [1] test_marks.py:52: test requires topology in Mark(name='topology', args=('tx',), kwargs={})
SKIPPED [2] test_marks.py:35: test requires topology in Mark(name='topology', args=('t1-lag',), kwargs={})
SKIPPED [3] test_marks.py: test requires topology in Mark(name='topology', args=('t1',), kwargs={})
SKIPPED [2] test_marks.py:18: test requires topology in Mark(name='topology', args=('t1',), kwargs={})
SKIPPED [2] test_marks.py:30: test requires topology in Mark(name='topology', args=('t0',), kwargs={})

pytest test_marks.py --topology any  -v -rA --collectonly
============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.5.2, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/rsavchuk/marks, configfile: pytest.ini
collected 20 items / 16 deselected / 4 selected                                                                                                                                                                                             

<Module test_marks.py>
  <Function test_case1_4[param1_value1]>
  <Function test_case1_4[param1_value2]>
  <Function test_case2_4[param2_value1]>
  <Function test_case2_4[param2_value2]>

========================================================================================================== 16 deselected in 0.01s ===========================================================================================================

pytest test_marks.py --topology t0,t1,any  -v -rA --collectonly
============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.5.2, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/rsavchuk/marks, configfile: pytest.ini
collected 20 items / 7 deselected / 13 selected                                                                                                                                                                                             

<Module test_marks.py>
  <Function test_case1_1[param1_value1]>
  <Function test_case1_3[param1_value1]>
  <Function test_case1_4[param1_value1]>
  <Function test_case1_1[param1_value2]>
  <Function test_case1_3[param1_value2]>
  <Function test_case1_4[param1_value2]>
  <Function test_case2_1[param2_value1]>
  <Function test_case2_1[param2_value2]>
  <Function test_case2_2[param2_value1]>
  <Function test_case2_2[param2_value2]>
  <Function test_case2_4[param2_value1]>
  <Function test_case2_4[param2_value2]>
  <Function test_case3_1>

=========================================================================================================== 7 deselected in 0.01s ===========================================================================================================

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[SavchukRomanLv]

@wangxin @yxieca can you please take a look? Thanks in advance!