Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[multi-dut] Added fixtures for selecting nodes from multi-duts based on card type and hwsku #2693

Merged
merged 3 commits into from
Jan 7, 2021
Merged

[multi-dut] Added fixtures for selecting nodes from multi-duts based on card type and hwsku #2693

merged 3 commits into from
Jan 7, 2021

Conversation

sanmalho-git
Copy link
Contributor

Description of PR

Summary:
Fixes # (issue)

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Approach

What is the motivation for this PR?

When dealing with SONiC Chassis you have cards of different type

  • linecards with front-panel ports
  • supervisor card
  • fabric cards

Also, you could have linecards with different hwsku's in the same chassis.

When using the parameterized approach for multi-dut to select test cases to run on a chassis, we need to make sure that we parameterize the tests to provide all the required coverage.

How did you do it?

Added the following fixtures that can be used for parameterizing tests for chassis:

  • rand_one_frontend_dut_hostname - returns a random dut that is a frontend node.
  • enum_supervisor_dut_hostname - returns a dut that is a supervisor node. For pizza box (single dut) this will the pizza box itself.
    • A supervisor node is defined as having 'type' variable in the inventory as 'supervisor'
    • For pizza box (single dut) the single dut itself will be the supervisor node.
  • enum_frontend_dut_hostname - returns all duts that are frontend nodes
    • A frontend node is defined as a node that is not a supervisor.
  • enum_rand_one_per_hwsku_frontend_hostname - returns 1 random frontend dut per hwsksu amongst all the frontend nodes.
    - hwsku for a node is derived from the 'hwsku' or 'sonic_hwsku' variable in the inventory for the node.

To achieve the above:

  • Added dut_utils under common/helpers to:

    • get_inventory_variable - get the value of a variable in the inventory for a given node.
    • is_supervisor_node - check if 'type' variable is defined in the inventory for the node and is 'supervisor'
    • is_frontend_node - check if node is not a supervisor node.

  • Modified SonicHost.is_supervisor_node to use dut_utils.is_supervisor_node.

How did you verify/test it?

Created a sample test case to exercise the selection of DUT's in a multi-dut testbed.

import logging

def test_one_frontend_node(rand_one_frontend_dut_hostname,):
    logging.info("Running test against " + rand_one_frontend_dut_hostname)

def test_all_frontend_nodes(enum_frontend_dut_hostname):
    logging.info("Running test against " + enum_frontend_dut_hostname)

def test_superivisor(enum_supervisor_dut_hostname):
    logging.info("Running test against " + enum_supervisor_dut_hostname)

def test_all_nodes(enum_dut_hostname):
    logging.info("Running test against " + enum_dut_hostname)

def test_one_node(rand_one_dut_hostname):
    logging.info("Running test against " + rand_one_dut_hostname)

def test_one_per_hwsku(enum_rand_one_per_hwsku_frontend_hostname):
    logging.info("Running test against " + enum_rand_one_per_hwsku_frontend_hostname)

The test were run against a Chassis testbed with the following cads:

  • board1 - frontend node (linecard) with hwsku1
  • board2 - frontend node (linecard) with hwsku1
  • board3 - frontend node (linecard) with hwsku2
  • board4 - supervisor node

The pytest results of running the sample test above against the testbed above is shown below.


=========================== short test summary info ============================
PASSED ../tests/sample_test.py::test_one_frontend_node
      21:46:20 INFO sample_test.py:test_one_frontend_node:5: Running test against board1

PASSED ../tests/sample_test.py::test_all_frontend_nodes[board1]
PASSED ../tests/sample_test.py::test_all_frontend_nodes[board2]
PASSED ../tests/sample_test.py::test_all_frontend_nodes[board3]

PASSED ../tests/sample_test.py::test_superivisor[board4]

PASSED ../tests/sample_test.py::test_all_nodes[board1]
PASSED ../tests/sample_test.py::test_all_nodes[board2]
PASSED ../tests/sample_test.py::test_all_nodes[board3]
PASSED ../tests/sample_test.py::test_all_nodes[board4]

PASSED ../tests/sample_test.py::test_one_node
      21:46:39 INFO sample_test.py:test_one_node:21: Running test against board4

PASSED ../tests/sample_test.py::test_one_per_hwsku[board1]
PASSED ../tests/sample_test.py::test_one_per_hwsku[board3]

========================== 12 passed in 82.67 seconds ==========================

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

@sanmalho-git
Added fixtures for selecting nodes from multi-duts based on card type…
2fd7394 
…/hwsku.

Added the following fixtures that can be used for parameterizing tests for chassis:
  - rand_one_frontend_dut_hostname - returns a random dut that is a frontend node.
  - enum_supervisor_dut_hostname - returns a dut that is a supervisor node. For pizza box (single dut) this will the pizza box itself.
     - A supervisor node is defined as having 'type' variable in the inventory as 'supervisor'
     - For pizza box (single dut) the single dut itself will be the supervisor node.
   - enum_frontend_dut_hostname - returns all duts that are frontend nodes
     - A frontend node is defined as a node that is not a supervisor.
   - enum_rand_one_per_hwsku_frontend_hostname - returns 1 random frontend dut per hwsksu amongst all the frontend nodes.
         - hwsku for a node is derived from the 'hwsku' or 'sonic_hwsku' variable in the inventory for the node.

To achieve the above:
  - Added dut_utils under common/helpers to:
    - get_inventory_variable - get the value of a variable in the inventory for a given node.
    - is_supervisor_node - check if 'type' variable is defined in the inventory for the node and is 'supervisor'
    - is_frontend_node - check if node is not a supervisor node.

  - Modified SonicHost.is_supervisor_node to use dut_utils.is_supervisor_node.
@sanmalho-git
Code optimization; review comments; updates based on converting tests
0abe36e 
- Pulling common code of getting duts in testbed into a function.
  - There was code to create TestbedInfo by parsing the testbed-file and getting the testbed info for a testbed.
    This code was repeated in tbinfo fixture and all the parameterization helper methods.
    Moved this into another function 'get_tbinfo' and made other functions/fixtures call this function.

- Review comments:
   - When selecting a frontend node, if we find that none of the DUTs in the testbed
     are frontend nodes, then we should fail.

- scope of parameterized fixtures
   - By default, all the 'enum_dut*' fixtures in pytest_generate_tests were created with scope 'function'.
     This made these fixtures not usable in other fixtures that have 'module' scope.
   - Added scope to be 'module' to all the 'enum_dut*' fixtures in 'pytest_generate_tests'.
@lgtm-com
Copy link

lgtm-com bot commented Jan 6, 2021

This pull request introduces 2 alerts when merging 0abe36e into 985eec2 - view on LGTM.com

new alerts:

  • 2 for Wrong number of arguments in a call

@wangxin wangxin merged commit 077f148 into sonic-net:master Jan 7, 2021
@sanmalho-git sanmalho-git deleted the fixtures branch January 7, 2021 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wangxin wangxin wangxin approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sanmalho-git @wangxin