[Fix] Fixing reboot check, IP cleanup and Execution time for multidut PFC cases #15111
Conversation
|
The pre-commit check detected issues in the files touched by this pull request. Detailed pre-commit check results: To run the pre-commit checks locally, you can follow below steps:
|
|
The pre-commit check detected issues in the files touched by this pull request. Detailed pre-commit check results: To run the pre-commit checks locally, you can follow below steps:
|
|
The pre-commit check detected issues in the files touched by this pull request. Detailed pre-commit check results: To run the pre-commit checks locally, you can follow below steps:
|
|
The pre-commit check detected issues in the files touched by this pull request. Detailed pre-commit check results: To run the pre-commit checks locally, you can follow below steps:
|
selldinesh
changed the title
Description of PR
Summary: Fixing reboot check and IP cleanup for multidut PFC cases
Fixes # (issue)
Type of change
Back port request
Approach
What is the motivation for this PR?
Previously the ip cleanup was not working as expected and safe-reboot argument was not present,
How did you do it?
Added try finally block for IP cleanup and safe_reboot=True option and instead of enum_dut_lossy added rand_one_dut_lossy_prio fixture which reduces the execution time of the script
How did you verify/test it?
Any platform specific information?
Supported testbed topology if it's a new test case?
Documentation
Output
PASSED [ 91%]
---------------------------------------------------------------------- live log teardown -----------------------------------------------------------------------
03:34:11 init.pytest_runtest_teardown L0049 INFO | collect memory after test test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-cold]
03:34:11 init.pytest_runtest_teardown L0072 INFO | After test: collected memory_values {'before_test': {'sonic-s6100-dut1': {'monit': {'memory_usage': 28.1}}}, 'after_test': {'sonic-s6100-dut1': {'monit': {'memory_usage': 27.3}}}}
snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py::test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-fast]
------------------------------------------------------------------------ live log setup ------------------------------------------------------------------------
03:34:11 init.set_default L0053 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic
03:34:11 init.check_test_completeness L0151 INFO | Test has no defined levels. Continue without test completeness checks
03:34:11 init.loganalyzer L0051 INFO | Log analyzer is disabled
03:34:11 init.memory_utilization L0091 INFO | Hostname: sonic-s6100-dut1, Hwsku: Arista-7060CX-32S-C32, Platform: x86_64-arista_7060_cx32s
03:34:11 init.store_fixture_values L0017 INFO | store memory_utilization test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-fast]
03:34:11 init.pytest_runtest_setup L0024 INFO | collect memory before test test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-fast]
03:34:12 init.pytest_runtest_setup L0044 INFO | Before test: collected memory_values {'before_test': {'sonic-s6100-dut1': {'monit': {'memory_usage': 27.3}}}, 'after_test': {'sonic-s6100-dut1': {}}}
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
03:34:12 test_multidut_pfc_pause_lossless_with_sn L0337 INFO | Running test for testbed subtype: single-dut-single-asic
03:34:12 snappi_fixtures.__intf_config_multidut L0796 INFO | Configuring Dut: sonic-s6100-dut1 with port Ethernet0 with IP 20.1.1.1/24
03:34:14 snappi_fixtures.__intf_config_multidut L0796 INFO | Configuring Dut: sonic-s6100-dut1 with port Ethernet4 with IP 20.1.2.1/24
03:34:17 test_multidut_pfc_pause_lossless_with_sn L0348 INFO | Snappi Ports : [{'ip': '10.36.78.53', 'card_id': '6', 'port_id': 0, 'peer_port': 'Ethernet0', 'peer_device': 'sonic-s6100-dut1', 'speed': '100000', 'intf_config_changed': True, 'location': '10.36.78.53;6;1', 'api_server_ip': '10.36.78.59', 'asic_type': 'broadcom', 'duthost': , 'snappi_speed_type': 'speed_100_gbps', 'asic_value': None}, {'ip': '10.36.78.53', 'card_id': '6', 'port_id': 1, 'peer_port': 'Ethernet4', 'peer_device': 'sonic-s6100-dut1', 'speed': '100000', 'intf_config_changed': True, 'location': '10.36.78.53;6;2', 'api_server_ip': '10.36.78.59', 'asic_type': 'broadcom', 'duthost': , 'snappi_speed_type': 'speed_100_gbps', 'asic_value': None}]
03:34:17 test_multidut_pfc_pause_lossless_with_sn L0354 INFO | Issuing a fast reboot on the dut sonic-s6100-dut1
03:34:17 reboot.reboot L0264 INFO | Reboot sonic-s6100-dut1: wait[120], timeout[180]
03:34:17 reboot.reboot L0266 INFO | DUT sonic-s6100-dut1 create a file /dev/shm/test_reboot before rebooting
03:34:18 reboot.wait_for_shutdown L0161 INFO | waiting for ssh to drop on sonic-s6100-dut1
03:34:18 reboot.execute_reboot_command L0201 INFO | rebooting sonic-s6100-dut1 with command "fast-reboot"
03:35:15 reboot.wait_for_startup L0182 INFO | waiting for ssh to startup on sonic-s6100-dut1
03:35:32 reboot.wait_for_startup L0193 INFO | ssh has started up on sonic-s6100-dut1
03:35:32 reboot.reboot L0281 INFO | waiting for switch sonic-s6100-dut1 to initialize
03:39:00 processes_utils.wait_critical_processes L0076 INFO | Wait until all critical processes are healthy in 300 sec
03:39:00 processes_utils.all_critical_processes L0045 INFO | Check critical processes status
03:39:10 reboot.reboot L0311 INFO | fast reboot finished on sonic-s6100-dut1
03:39:11 reboot.reboot L0314 INFO | DUT sonic-s6100-dut1 up since 2024-10-23 02:12:37.030000
03:39:11 test_multidut_pfc_pause_lossless_with_sn L0356 INFO | Wait until the system is stable
03:39:11 processes_utils.wait_critical_processes L0076 INFO | Wait until all critical processes are healthy in 300 sec
03:39:11 processes_utils.all_critical_processes L0045 INFO | Check critical processes status
03:39:20 test_multidut_pfc_pause_lossless_with_sn L0354 INFO | Issuing a fast reboot on the dut sonic-s6100-dut1
03:39:20 reboot.reboot L0264 INFO | Reboot sonic-s6100-dut1: wait[120], timeout[180]
03:39:20 reboot.reboot L0266 INFO | DUT sonic-s6100-dut1 create a file /dev/shm/test_reboot before rebooting
03:39:21 reboot.wait_for_shutdown L0161 INFO | waiting for ssh to drop on sonic-s6100-dut1
03:39:21 reboot.execute_reboot_command L0201 INFO | rebooting sonic-s6100-dut1 with command "fast-reboot"
03:40:17 reboot.wait_for_startup L0182 INFO | waiting for ssh to startup on sonic-s6100-dut1
03:40:34 reboot.wait_for_startup L0193 INFO | ssh has started up on sonic-s6100-dut1
03:40:34 reboot.reboot L0281 INFO | waiting for switch sonic-s6100-dut1 to initialize
03:44:02 processes_utils.wait_critical_processes L0076 INFO | Wait until all critical processes are healthy in 300 sec
03:44:02 processes_utils.all_critical_processes L0045 INFO | Check critical processes status
03:44:12 reboot.reboot L0311 INFO | fast reboot finished on sonic-s6100-dut1
03:44:13 reboot.reboot L0314 INFO | DUT sonic-s6100-dut1 up since 2024-10-23 02:17:37.570000
03:44:13 test_multidut_pfc_pause_lossless_with_sn L0356 INFO | Wait until the system is stable
03:44:13 processes_utils.wait_critical_processes L0076 INFO | Wait until all critical processes are healthy in 300 sec
03:44:13 processes_utils.all_critical_processes L0045 INFO | Check critical processes status
03:44:32 snappi_api.info L1242 INFO | Config validation 0.022s
03:44:33 snappi_api.info L1242 INFO | Ports configuration 0.244s
03:44:34 snappi_api.info L1242 INFO | Captures configuration 0.159s
03:44:40 snappi_api.info L1242 INFO | Location hosts ready [10.36.78.53] 2.082s
03:44:40 snappi_api.info L1242 INFO | Speed change not require due to redundant Layer1 config
03:44:40 snappi_api.info L1242 INFO | Aggregation mode speed change 0.013s
03:44:42 snappi_api.info L1242 INFO | Location preemption [10.36.78.53;6;2] 0.087s
03:45:03 snappi_api.info L1242 INFO | Location connect [Port 1] 20.785s
03:45:03 snappi_api.info L1242 INFO | Location state check [Port 1] 0.229s
03:45:03 snappi_api.info L1242 INFO | Location configuration 29.758s
03:45:11 snappi_api.info L1242 INFO | Layer1 configuration 7.547s
03:45:11 snappi_api.info L1242 INFO | Lag Configuration 0.070s
03:45:12 snappi_api.info L1242 INFO | Convert device config : 0.446s
03:45:12 snappi_api.info L1242 INFO | Create IxNetwork device config : 0.000s
03:45:12 snappi_api.info L1242 INFO | Push IxNetwork device config : 0.179s
03:45:12 snappi_api.info L1242 INFO | Devices configuration 0.701s
03:45:18 snappi_api.info L1242 INFO | Flows configuration 5.866s
03:45:25 snappi_api.info L1242 INFO | Start interfaces 6.855s
03:45:25 snappi_api.info L1242 INFO | IxNet - The Traffic Item was modified. Please perform a Traffic Generate to update the associated traffic Flow Groups
03:45:25 traffic_generation.run_traffic L0321 INFO | Wait for Arp to Resolve ...
03:45:30 traffic_generation.run_traffic L0342 INFO | Starting transmit on all flows ...
03:45:33 snappi_api.info L1242 INFO | Flows generate/apply 1.852s
03:45:45 snappi_api.info L1242 INFO | Flows clear statistics 12.401s
03:45:45 snappi_api.info L1242 INFO | Captures start 0.000s
03:45:49 snappi_api.info L1242 INFO | Flows start 3.557s
03:45:49 traffic_generation.run_traffic L0349 INFO | Polling DUT for traffic statistics for 67 seconds ...
03:47:27 traffic_generation.run_traffic L0368 INFO | Polling TGEN for in-flight traffic statistics...
03:47:28 traffic_generation.run_traffic L0373 INFO | In-flight traffic statistics for flows: ['Test Flow Prio 3', 'Test Flow Prio 4', 'Background Flow Prio 1', 'Background Flow Prio 5', 'Background Flow Prio 6', 'Background Flow Prio 2', 'Background Flow Prio 0']
03:47:28 traffic_generation.run_traffic L0374 INFO | In-flight TX frames: [729, 729, 67887931, 67887931, 67887931, 67887931, 67887931]
03:47:28 traffic_generation.run_traffic L0375 INFO | In-flight RX frames: [0, 0, 67887931, 67887931, 67887931, 67887931, 67887931]
03:48:33 traffic_generation.run_traffic L0376 INFO | DUT polling complete
03:48:33 traffic_generation.run_traffic L0387 INFO | Checking if all flows have stopped. Attempt #1
03:48:34 traffic_generation.run_traffic L0394 INFO | All test and background traffic flows stopped
03:48:36 traffic_generation.run_traffic L0417 INFO | Dumping per-flow statistics
03:48:38 traffic_generation.run_traffic L0419 INFO | Stopping transmit on all remaining flows
03:48:44 snappi_api.info L1242 INFO | Flows stop 6.372s
03:48:57 snappi_fixtures.cleanup_config L0952 INFO | Removing Configuration on Dut: sonic-s6100-dut1 with port Ethernet0 with ip :20.1.1.1/24
03:48:59 snappi_fixtures.cleanup_config L0952 INFO | Removing Configuration on Dut: sonic-s6100-dut1 with port Ethernet4 with ip :20.1.2.1/24
PASSED [100%]
---------------------------------------------------------------------- live log teardown -----------------------------------------------------------------------
03:49:01 init.pytest_runtest_teardown L0049 INFO | collect memory after test test_pfc_pause_multi_lossless_prio_reboot[multidut_port_info0-fast]
03:49:02 init.pytest_runtest_teardown L0072 INFO | After test: collected memory_values {'before_test': {'sonic-s6100-dut1': {'monit': {'memory_usage': 27.3}}}, 'after_test': {'sonic-s6100-dut1': {'monit': {'memory_usage': 28.0}}}}
03:49:02 init._fixture_generator_decorator L0093 INFO | -------------------- fixture snappi_api teardown starts --------------------
03:49:31 init._fixture_generator_decorator L0102 INFO | -------------------- fixture snappi_api teardown ends --------------------
03:49:31 init._fixture_generator_decorator L0093 INFO | -------------------- fixture start_pfcwd_after_test teardown starts --------------------
03:49:32 init._fixture_generator_decorator L0102 INFO | -------------------- fixture start_pfcwd_after_test teardown ends --------------------
03:49:32 init._fixture_generator_decorator L0093 INFO | -------------------- fixture rand_lossy_prio teardown starts --------------------
03:49:32 init._fixture_generator_decorator L0102 INFO | -------------------- fixture rand_lossy_prio teardown ends --------------------
03:49:32 init._fixture_generator_decorator L0093 INFO | -------------------- fixture rand_lossless_prio teardown starts --------------------
03:49:32 init._fixture_generator_decorator L0102 INFO | -------------------- fixture rand_lossless_prio teardown ends --------------------
03:49:32 init._fixture_generator_decorator L0093 INFO | -------------------- fixture enable_packet_aging_after_test teardown starts --------------------
03:49:32 init._fixture_generator_decorator L0102 INFO | -------------------- fixture enable_packet_aging_after_test teardown ends --------------------
03:49:35 conftest.core_dump_and_config_check L2203 INFO | Dumping Disk and Memory Space informataion after test on sonic-s6100-dut1
03:49:36 conftest.core_dump_and_config_check L2207 INFO | Collecting core dumps after test on sonic-s6100-dut1
03:49:36 conftest.core_dump_and_config_check L2224 INFO | Collecting running config after test on sonic-s6100-dut1
03:49:38 conftest.core_dump_and_config_check L2381 INFO | Executing config reload of config_db_bgp.json
03:50:08 conftest.core_dump_and_config_check L2383 INFO | Core dump and config check passed for test_multidut_pfc_pause_lossless_with_snappi.py
======================================================================= warnings summary =======================================================================
../../../../usr/local/lib/python3.8/dist-packages/_pytest/config/init.py:755
/usr/local/lib/python3.8/dist-packages/_pytest/config/init.py:755: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: tests.common.plugins.loganalyzer
self.import_plugin(import_spec)
../../../../usr/local/lib/python3.8/dist-packages/_pytest/config/init.py:755
/usr/local/lib/python3.8/dist-packages/_pytest/config/init.py:755: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: tests.common.plugins.sanity_check
self.import_plugin(import_spec)
../../../../usr/local/lib/python3.8/dist-packages/_pytest/config/init.py:755
/usr/local/lib/python3.8/dist-packages/_pytest/config/init.py:755: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: tests.common.plugins.test_completeness
self.import_plugin(import_spec)
../../../../usr/local/lib/python3.8/dist-packages/_pytest/config/init.py:755
/usr/local/lib/python3.8/dist-packages/_pytest/config/init.py:755: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: tests.common.dualtor
self.import_plugin(import_spec)
../../../../usr/local/lib/python3.8/dist-packages/paramiko/transport.py:236
/usr/local/lib/python3.8/dist-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
"class": algorithms.Blowfish,
../../../../usr/local/lib/python3.8/dist-packages/scapy/layers/ipsec.py:471
/usr/local/lib/python3.8/dist-packages/scapy/layers/ipsec.py:471: CryptographyDeprecationWarning: Blowfish has been deprecated
cipher=algorithms.Blowfish,
../../../../usr/local/lib/python3.8/dist-packages/scapy/layers/ipsec.py:485
/usr/local/lib/python3.8/dist-packages/scapy/layers/ipsec.py:485: CryptographyDeprecationWarning: CAST5 has been deprecated
cipher=algorithms.CAST5,
snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py: 17 warnings
/usr/local/lib/python3.8/dist-packages/pytest_ansible/module_dispatcher/v213.py:100: UserWarning: provided hosts list is empty, only localhost is available
warnings.warn("provided hosts list is empty, only localhost is available")
snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py::test_pfc_pause_single_lossless_prio[sonic-s6100-dut1|3-multidut_port_info0]
snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py::test_pfc_pause_single_lossless_prio[sonic-s6100-dut1|3-multidut_port_info0]
/usr/local/lib/python3.8/dist-packages/ixnetwork_restpy/testplatform/sessions/sessions.py:59: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
elif LooseVersion(build_number) < LooseVersion('8.52'):
snappi_tests/multidut/pfc/test_multidut_pfc_pause_lossless_with_snappi.py: 11 warnings
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.36.78.59'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
-------------------------------------------------------------------- live log sessionfinish --------------------------------------------------------------------
03:50:08 init.pytest_terminal_summary L0067 INFO | Can not get Allure report URL. Please check logs
========================================================= 12 passed, 37 warnings in 8129.33s (2:15:29) =========================================================
INFO:root:Can not get Allure report URL. Please check logs