[test_bgp_gr_helper] | fails over KeyError: 'admin_status'

[slutati1536]

Description

test is broken by PR #7748, failed over key error

Steps to reproduce the issue:

1. run bgp/test_bgp_gr_helper.py

Describe the results you received:
if test_interface.startswith("PortChannel"):
for member in list(portchannels[test_interface].keys()):

          nbr_ports.append(dev_nbrs[member]['port'])

E KeyError: 'admin_status'

_find_test_bgp_neighbors = <function test_bgp_gr_helper_routes_perserved.._find_test_bgp_neighbors at 0x7fded0cbdf70>
_get_learned_bgp_routes_from_neighbor = <function test_bgp_gr_helper_routes_perserved.._get_learned_bgp_routes_from_neighbor at 0x7fded0e21430>
_get_prefix_counters = <function test_bgp_gr_helper_routes_perserved.._get_prefix_counters at 0x7fded0e21d30>
_get_rib = <function test_bgp_gr_helper_routes_perserved.._get_rib at 0x7fded0e21dc0>
_verify_bgp_neighbor_routes_during_graceful_restart = <function test_bgp_gr_helper_routes_perserved.._verify_bgp_neighbor_routes_during_graceful_restart at 0x7fded16a0820>
_verify_prefix_counters_from_neighbor_after_graceful_restart = <function test_bgp_gr_helper_routes_perserved.._verify_prefix_counters_from_neighbor_after_graceful_restart at 0x7fded271c820>
_verify_prefix_counters_from_neighbor_during_graceful_restart = <function test_bgp_gr_helper_routes_perserved.._verify_prefix_counters_from_neighbor_during_graceful_restart at 0x7fded69afc10>
bgp_neighbors = {'10.0.0.1': {'admin_status': 'up', 'asn': '64802', 'holdtime': '10', 'keepalive': '3', ...}, '10.0.0.13': {'admin_sta...eepalive': '3', ...}, '10.0.0.9': {'admin_status': 'up', 'asn': '64802', 'holdtime': '10', 'keepalive': '3', ...}, ...}
config_facts = {'ACL_TABLE': {'DATAACL': {'policy_desc': 'DATAACL', 'ports': ['PortChannel101', 'PortChannel102', 'PortChannel103', '...'rate_limit_interval': '600', 'state': 'enabled'}, ...}, 'BGP_DEVICE_GLOBAL': {'STATE': {'tsa_enabled': 'false'}}, ...}
configurations = {'common': {'dut_asn': 64601, 'dut_type': 'ToRRouter', 'failure_rate': 0, 'leaf_asn_start': 64802, ...}}
dev_nbrs = {'Ethernet0': {'name': 'ARISTA01T1', 'port': 'Ethernet1'}, 'Ethernet100': {'name': 'Servers13', 'port': 'eth0'}, 'Ethernet104': {'name': 'Servers14', 'port': 'eth0'}, 'Ethernet108': {'name': 'Servers15', 'port': 'eth0'}, ...}
duthost =
duthosts = []
exabgp_ips = ['10.10.246.254', 'FC0A::FF']
exabgp_sessions = ['exabgp_v4', 'exabgp_v6']
ifnames_common = ['PortChannel101', 'PortChannel104', 'PortChannel103', 'PortChannel102']
ifnames_v4 = ['PortChannel101', 'PortChannel104', 'PortChannel103', 'PortChannel102']
ifnames_v6 = ['PortChannel101', 'PortChannel104', 'PortChannel103', 'PortChannel102']
member = 'admin_status'
nbr_ports = []
nbrhosts = {'ARISTA01T1': , 'ARISTA02T1': , 'ARISTA03T1': , 'ARISTA04T1': }
portchannels = {'PortChannel101': {'admin_status': 'up', 'lacp_key': 'auto', 'members': ['Ethernet0', 'Ethernet4'], 'min_links': '2',...hannel104': {'admin_status': 'up', 'lacp_key': 'auto', 'members': ['Ethernet80', 'Ethernet84'], 'min_links': '2', ...}}
portchannels_memebers = {'PortChannel101': {'Ethernet0': {}, 'Ethernet4': {}}, 'PortChannel102': {'Ethernet16': {}, 'Ethernet20': {}}, 'PortChannel103': {'Ethernet64': {}, 'Ethernet68': {}}, 'PortChannel104': {'Ethernet80': {}, 'Ethernet84': {}}}
rand_one_dut_hostname = 'r-tigon-20'
rtinfo_v4 = {'nexthops': [(IPv4Address('10.0.0.1'), 'PortChannel101'), (IPv4Address('10.0.0.13'), 'PortChannel104'), (IPv4Address('10.0.0.9'), 'PortChannel103'), (IPv4Address('10.0.0.5'), 'PortChannel102')], 'set_src': IPv4Address('10.1.0.32')}
rtinfo_v6 = {'nexthops': [(IPv6Address('fc00::2'), 'PortChannel101'), (IPv6Address('fc00::1a'), 'PortChannel104'), (IPv6Address('fc00::12'), 'PortChannel103'), (IPv6Address('fc00::a'), 'PortChannel102')], 'set_src': IPv6Address('fc00:1::32')}
setup_bgp_graceful_restart = None
tbinfo = {'comment': 'Mellanox MTR testbed', 'conf-name': 'r-tigon-20-t0-64', 'duts': ['r-tigon-20'], 'duts_map': {'r-tigon-20': 0}, ...}
test_interface = 'PortChannel101'

bgp/test_bgp_gr_helper.py:135: KeyError

for member in list(portchannels[test_interface].keys()): should be fixed to
for member in list(portchannels[test_interface]['members'].keys())

Describe the results you expected:

test should not fail over test issue

Additional information you deem important:
Output of show version:

SONiC Software Version: SONiC.202211_RC8.3-696a6a43b_Internal
Distribution: Debian 11.6
Kernel: 5.10.0-18-2-amd64
Build commit: 696a6a43b
Build date: Mon Mar 27 10:50:51 UTC 2023
Built by: sw-r2d2-bot@r-build-sonic-ci03-244

Platform: x86_64-mlnx_msn4600c-r0
HwSKU: Mellanox-SN4600C-C64
ASIC: mellanox
ASIC Count: 1
Serial Number: MT2131X10295
Model Number: MSN4600-CS2FO
Hardware Revision: A1
Uptime: 13:16:06 up 13:25,  2 users,  load average: 1.37, 1.34, 1.30
Date: Tue 28 Mar 2023 13:16:06

Docker images:
REPOSITORY                                         TAG                               IMAGE ID       SIZE
docker-orchagent                                   202211_RC8.3-696a6a43b_Internal   84055ebb4c77   539MB
docker-orchagent                                   latest                            84055ebb4c77   539MB
docker-fpm-frr                                     202211_RC8.3-696a6a43b_Internal   83d5f57abddb   550MB
docker-fpm-frr                                     latest                            83d5f57abddb   550MB
docker-teamd                                       202211_RC8.3-696a6a43b_Internal   e6b58409a5c6   520MB
docker-teamd                                       latest                            e6b58409a5c6   520MB
docker-macsec                                      latest                            d76f633dda63   522MB
docker-syncd-mlnx                                  202211_RC8.3-696a6a43b_Internal   2bd090589266   939MB
docker-syncd-mlnx                                  latest                            2bd090589266   939MB
docker-platform-monitor                            202211_RC8.3-696a6a43b_Internal   4fcfc27b9bbe   942MB
docker-platform-monitor                            latest                            4fcfc27b9bbe   942MB
docker-sonic-telemetry                             202211_RC8.3-696a6a43b_Internal   be9f9e750657   802MB
docker-sonic-telemetry                             latest                            be9f9e750657   802MB
docker-snmp                                        202211_RC8.3-696a6a43b_Internal   62432cf85f40   549MB
docker-snmp                                        latest                            62432cf85f40   549MB
docker-eventd                                      202211_RC8.3-696a6a43b_Internal   855b7c3d7218   503MB
docker-eventd                                      latest                            855b7c3d7218   503MB
docker-dhcp-relay                                  latest                            c90d1c5967eb   513MB
docker-lldp                                        202211_RC8.3-696a6a43b_Internal   e33be345d3f2   546MB
docker-lldp                                        latest                            e33be345d3f2   546MB
docker-mux                                         202211_RC8.3-696a6a43b_Internal   e9f6fa7ef277   552MB
docker-mux                                         latest                            e9f6fa7ef277   552MB
docker-database                                    202211_RC8.3-696a6a43b_Internal   9cae58961a25   503MB
docker-database                                    latest                            9cae58961a25   503MB
docker-sonic-p4rt                                  202211_RC8.3-696a6a43b_Internal   77b20e5b758d   586MB
docker-sonic-p4rt                                  latest                            77b20e5b758d   586MB
docker-router-advertiser                           202211_RC8.3-696a6a43b_Internal   f3a99688e9df   503MB
docker-router-advertiser                           latest                            f3a99688e9df   503MB
docker-sflow                                       202211_RC8.3-696a6a43b_Internal   780973343beb   479MB
docker-sflow                                       latest                            780973343beb   479MB
docker-nat                                         202211_RC8.3-696a6a43b_Internal   83454bb6925c   481MB
docker-nat                                         latest                            83454bb6925c   481MB
docker-sonic-mgmt-framework                        202211_RC8.3-696a6a43b_Internal   20cfc12ab3c5   612MB
docker-sonic-mgmt-framework                        latest                            20cfc12ab3c5   612MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/doroce      1.1.0-202211-internal-5           078ba366e374   200MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/sonic-wjh   1.4.0-202211-internal-5           b4f92852d9ca   323MB

[wsycqyz]

This test case passed in my testbed. Need to discuss with case owner.

[wsycqyz]

Not a Python3 migration issue.
In @slutati1536 's traceback,

portchannels = {'PortChannel101': {'admin_status': 'up', 'lacp_key': 'auto', 'members': ['Ethernet0', 'Ethernet4'], 'min_links': '2',...hannel104': {'admin_status': 'up', 'lacp_key': 'auto', 'members': ['Ethernet80', 'Ethernet84'], 'min_links': '2', ...}}
portchannels_memebers = {'PortChannel101': {'Ethernet0': {}, 'Ethernet4': {}}, 'PortChannel102': {'Ethernet16': {}, 'Ethernet20': {}}, 'PortChannel103': {'Ethernet64': {}, 'Ethernet68': {}}, 'PortChannel104': {'Ethernet80': {}, 'Ethernet84': {}}}

In my internal code,

portchannels = {'PortChannel101': {'Ethernet24': {}}, 'PortChannel102': {'Ethernet26': {}}, 'PortChannel103': {'Ethernet28': {}}, 'PortChannel104': {'Ethernet30': {}}, 'PortChannel105': {'Ethernet96': {}}, 'PortChannel106': {'Ethernet98': {}}, 'PortChannel107': {'Ethernet100': {}}, 'PortChannel108': {'Ethernet102': {}}}

Suspect #7568 has something to do with this. Will contact #7568 's owner.

[wsycqyz]

@slutati1536 FYI

[saiarcot895]

@slutati1536 What version of sonic-mgmt are you using? If you are using a newer image, you will need to update sonic-mgmt as well, since there were some test-breaking changes that went in.