[UT][YANG] Test_SonicYang.test_xlate_rev_xlate JSON comparison issue #7733

nazariig · 2021-05-27T12:38:27Z

Description

There are some cases when comparison logic in Test_SonicYang.test_xlate_rev_xlate could end up in unexpected failures.
Example:

def test_xlate_rev_xlate(self, sonic_yang_data):
    # In this test, xlation and revXlation is tested with latest Sonic
    # YANG model.
    test_file = sonic_yang_data['test_file']
    syc = sonic_yang_data['syc']

    jIn = self.readIjsonInput(test_file, 'SAMPLE_CONFIG_DB_JSON')
    jIn = json.loads(jIn)
    numTables = len(jIn)

    syc.loadData(jIn, debug=True)
    # check all tables are loaded and no tables is without Yang Models
    assert len(syc.jIn) == numTables
    assert len(syc.tablesWithOutYang) == 0

    syc.getData(debug=True)

    if syc.jIn and syc.jIn == syc.revXlateJson:
        print("Xlate and Rev Xlate Passed")
    else:
        print("Xlate and Rev Xlate failed")
        # print for better debugging, in case of failure.
        print("syc.jIn: {}".format({t:syc.jIn[t].keys() \
            for t in syc.jIn.keys()}))
        print("syc.revXlateJson: {}".format({t:syc.revXlateJson[t].keys() \
            for t in syc.revXlateJson.keys()}))
        # make it fail
        assert False == True

    return

The current implementation assumes simple dict comparison:

if syc.jIn and syc.jIn == syc.revXlateJson:
    print("Xlate and Rev Xlate Passed")

This will end up in failures when input sample config data has inappropriate values formatting:
Input:

    "PBH_HASH": {
        "inner_dst_ipv6": {
            "hash_field": "INNER_DST_IPV6",
            "ipv6_mask": "FFFF::",
            "sequence_id": "4"
        },
        "inner_src_ipv6": {
            "hash_field": "INNER_SRC_IPV6",
            "ipv6_mask": "::FFFF",
            "sequence_id": "4"
        }
    },

Output:

    "PBH_HASH": {
        "inner_dst_ipv6": {
            "hash_field": "INNER_DST_IPV6",
            "ipv6_mask": "ffff::",
            "sequence_id": "4"
        },
        "inner_src_ipv6": {
            "hash_field": "INNER_SRC_IPV6",
            "ipv6_mask": "::ffff",
            "sequence_id": "4"
        }
    },

Such issues are hard to debug.
Suggest using deepdiff to get more user friendly results.

Steps to reproduce the issue:

Define IPv6 as FFFF::
Run UT

Describe the results you received:

nazariig@664912530dcf:/sonic/src/sonic-yang-mgmt$ python3 setup.py test
running pytest
Searching for ijson==2.6.1
Best match: ijson 2.6.1
Processing ijson-2.6.1-py3.7-linux-x86_64.egg

Using /sonic/src/sonic-yang-mgmt/.eggs/ijson-2.6.1-py3.7-linux-x86_64.egg
Searching for xmltodict==0.12.0
Best match: xmltodict 0.12.0
Processing xmltodict-0.12.0-py3.7.egg

Using /sonic/src/sonic-yang-mgmt/.eggs/xmltodict-0.12.0-py3.7.egg
running egg_info
writing sonic_yang_mgmt.egg-info/PKG-INFO
writing dependency_links to sonic_yang_mgmt.egg-info/dependency_links.txt
writing requirements to sonic_yang_mgmt.egg-info/requires.txt
writing top-level names to sonic_yang_mgmt.egg-info/top_level.txt
reading manifest file 'sonic_yang_mgmt.egg-info/SOURCES.txt'
writing manifest file 'sonic_yang_mgmt.egg-info/SOURCES.txt'
running build_ext
============================================================================================= test session starts =============================================================================================
platform linux -- Python 3.7.3, pytest-3.10.1, py-1.7.0, pluggy-0.8.0
rootdir: /sonic/src/sonic-yang-mgmt, inifile:
plugins: cov-2.6.0
collected 26 items

tests/test_sonic_yang_mgmt.py .                                                                                                                                                                         [  3%]
tests/libyang-python-tests/test_sonic_yang.py .......................F.                                                                                                                                 [100%]

================================================================================================== FAILURES ===================================================================================================
_____________________________________________________________________________________ Test_SonicYang.test_xlate_rev_xlate _____________________________________________________________________________________

self = <test_sonic_yang.Test_SonicYang object at 0x7fee45537fd0>
sonic_yang_data = {'syc': <sonic_yang.SonicYang object at 0x7fee45d466d8>, 'test_file': '../sonic-yang-models/tests/files/sample_config_db.json', 'yang_dir': '../sonic-yang-models/yang-models/'}

    def test_xlate_rev_xlate(self, sonic_yang_data):
        import pdb
        # In this test, xlation and revXlation is tested with latest Sonic
        # YANG model.
        test_file = sonic_yang_data['test_file']
        syc = sonic_yang_data['syc']

        jIn = self.readIjsonInput(test_file, 'SAMPLE_CONFIG_DB_JSON')
        jIn = json.loads(jIn)
        numTables = len(jIn)

        syc.loadData(jIn, debug=True)
        # check all tables are loaded and no tables is without Yang Models
        assert len(syc.jIn) == numTables
        assert len(syc.tablesWithOutYang) == 0

        syc.getData(debug=True)

        #pdb.set_trace()
        if syc.jIn and syc.jIn == syc.revXlateJson:
            print("Xlate and Rev Xlate Passed")
        else:
            print("Xlate and Rev Xlate failed")
            # print for better debugging, in case of failure.
            print("syc.jIn: {}".format({t:syc.jIn[t].keys() \
                for t in syc.jIn.keys()}))
            print("syc.revXlateJson: {}".format({t:syc.revXlateJson[t].keys() \
                for t in syc.revXlateJson.keys()}))
            # make it fail
>           assert False == True
E           assert False == True

tests/libyang-python-tests/test_sonic_yang.py:351: AssertionError
-------------------------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------------------------
 Read JSON Section: SAMPLE_CONFIG_DB_JSON
Xlate and Rev Xlate failed
syc.jIn: {'VRF': dict_keys(['Vrf_blue']), 'PORTCHANNEL': dict_keys(['PortChannel0003', 'PortChannel0004']), 'PORTCHANNEL_INTERFACE': dict_keys(['PortChannel0003', 'PortChannel0004']), 'PORTCHANNEL_MEMBER': dict_keys(['PortChannel0003|Ethernet1', 'PortChannel0004|Ethernet2']), 'VLAN_INTERFACE': dict_keys(['Vlan111', 'Vlan777', 'Vlan111|2a04:5555:45:6709::1/64', 'Vlan111|10.222.10.65/26', 'Vlan111|fe80::1/10', 'Vlan777|2a04:5555:41:4e9::1/64', 'Vlan777|10.111.58.65/26', 'Vlan777|fe80::1/10']), 'ACL_RULE': dict_keys(['V4-ACL-TABLE|DEFAULT_DENY', 'V4-ACL-TABLE|Rule_20', 'V4-ACL-TABLE|Rule_40', 'V4-ACL-TABLE|Rule_60', 'V4-ACL-TABLE|Rule_80', 'V4-ACL-TABLE|Rule_111', 'V4-ACL-TABLE|Rule_120', 'V4-ACL-TABLE|Rule_140', 'V4-ACL-TABLE|Rule_160', 'V4-ACL-TABLE|Rule_180', 'V4-ACL-TABLE|Rule_9000', 'V4-ACL-TABLE|Rule_11100', 'V6-ACL-TBLE|DEFAULT_DENY', 'V6-ACL-TBLE|Rule_20', 'V6-ACL-TBLE|Rule_40', 'V6-ACL-TBLE|Rule_60', 'V6-ACL-TBLE|Rule_80', 'V6-ACL-TBLE|Rule_111', 'V6-ACL-TBLE|Rule_9000', 'V6-ACL-TBLE|Rule_11100']), 'DEVICE_METADATA': dict_keys(['localhost']), 'VLAN': dict_keys(['Vlan111', 'Vlan777']), 'DEVICE_NEIGHBOR': dict_keys(['Ethernet112', 'Ethernet114', 'Ethernet116', 'Ethernet118']), 'PORT': dict_keys(['Ethernet0', 'Ethernet1', 'Ethernet2', 'Ethernet3', 'Ethernet4', 'Ethernet5', 'Ethernet6', 'Ethernet7', 'Ethernet8', 'Ethernet9', 'Ethernet10', 'Ethernet11', 'Ethernet12', 'Ethernet13', 'Ethernet14', 'Ethernet15', 'Ethernet16', 'Ethernet17', 'Ethernet18', 'Ethernet19', 'Ethernet20', 'Ethernet21', 'Ethernet22', 'Ethernet23', 'Ethernet24', 'Ethernet25', 'Ethernet26', 'Ethernet27', 'Ethernet28', 'Ethernet29', 'Ethernet30', 'Ethernet31', 'Ethernet32', 'Ethernet33', 'Ethernet34', 'Ethernet35', 'Ethernet36', 'Ethernet112']), 'ACL_TABLE': dict_keys(['V4-ACL-TABLE', 'V6-ACL-TBLE']), 'PBH_HASH': dict_keys(['inner_ip_proto', 'inner_l4_dst_port', 'inner_l4_src_port', 'inner_dst_ipv4', 'inner_src_ipv4', 'inner_dst_ipv6', 'inner_src_ipv6']), 'PBH_RULE': dict_keys(['pbh_table|nvgre', 'pbh_table|vxlan']), 'PBH_TABLE': dict_keys(['pbh_table']), 'INTERFACE': dict_keys(['Ethernet112', 'Ethernet14', 'Ethernet16', 'Ethernet18', 'Ethernet112|2a04:5555:40:a709::2/126', 'Ethernet112|10.184.228.211/31', 'Ethernet14|2a04:5555:40:a749::2/126', 'Ethernet14|10.184.229.211/31', 'Ethernet16|2a04:5555:40:a789::2/126', 'Ethernet16|10.184.230.211/31', 'Ethernet18|2a04:5555:40:a7c9::2/126', 'Ethernet18|10.184.231.211/31']), 'VLAN_MEMBER': dict_keys(['Vlan111|Ethernet0', 'Vlan111|Ethernet1', 'Vlan111|Ethernet2', 'Vlan111|Ethernet3', 'Vlan111|Ethernet4', 'Vlan111|Ethernet5', 'Vlan111|Ethernet6', 'Vlan111|Ethernet29', 'Vlan111|Ethernet30', 'Vlan111|Ethernet31', 'Vlan111|Ethernet32', 'Vlan111|Ethernet33', 'Vlan111|Ethernet34', 'Vlan111|Ethernet35', 'Vlan111|Ethernet36', 'Vlan111|PortChannel0003']), 'LOOPBACK_INTERFACE': dict_keys(['Loopback0', 'Loopback0|2a04:5555:40:4::4e9/128', 'Loopback0|10.184.8.233/32']), 'BREAKOUT_CFG': dict_keys(['Ethernet0', 'Ethernet4', 'Ethernet8']), 'VERSIONS': dict_keys(['DATABASE']), 'FLEX_COUNTER_TABLE': dict_keys(['PFCWD', 'PG_WATERMARK', 'PORT', 'PORT_RATES', 'PORT_BUFFER_DROP', 'BUFFER_POOL_WATERMARK', 'QUEUE', 'QUEUE_WATERMARK', 'DEBUG_COUNTER']), 'CRM': dict_keys(['Config'])}
syc.revXlateJson: {'VRF': dict_keys(['Vrf_blue']), 'PORTCHANNEL': dict_keys(['PortChannel0003', 'PortChannel0004']), 'PORTCHANNEL_INTERFACE': dict_keys(['PortChannel0003', 'PortChannel0004']), 'PORTCHANNEL_MEMBER': dict_keys(['PortChannel0003|Ethernet1', 'PortChannel0004|Ethernet2']), 'VLAN_INTERFACE': dict_keys(['Vlan111', 'Vlan777', 'Vlan111|2a04:5555:45:6709::1/64', 'Vlan111|10.222.10.65/26', 'Vlan111|fe80::1/10', 'Vlan777|2a04:5555:41:4e9::1/64', 'Vlan777|10.111.58.65/26', 'Vlan777|fe80::1/10']), 'VLAN': dict_keys(['Vlan111', 'Vlan777']), 'VLAN_MEMBER': dict_keys(['Vlan111|Ethernet0', 'Vlan111|Ethernet1', 'Vlan111|Ethernet2', 'Vlan111|Ethernet3', 'Vlan111|Ethernet4', 'Vlan111|Ethernet5', 'Vlan111|Ethernet6', 'Vlan111|Ethernet29', 'Vlan111|Ethernet30', 'Vlan111|Ethernet31', 'Vlan111|Ethernet32', 'Vlan111|Ethernet33', 'Vlan111|Ethernet34', 'Vlan111|Ethernet35', 'Vlan111|Ethernet36', 'Vlan111|PortChannel0003']), 'ACL_RULE': dict_keys(['V4-ACL-TABLE|DEFAULT_DENY', 'V4-ACL-TABLE|Rule_20', 'V4-ACL-TABLE|Rule_40', 'V4-ACL-TABLE|Rule_60', 'V4-ACL-TABLE|Rule_80', 'V4-ACL-TABLE|Rule_111', 'V4-ACL-TABLE|Rule_120', 'V4-ACL-TABLE|Rule_140', 'V4-ACL-TABLE|Rule_160', 'V4-ACL-TABLE|Rule_180', 'V4-ACL-TABLE|Rule_9000', 'V4-ACL-TABLE|Rule_11100', 'V6-ACL-TBLE|DEFAULT_DENY', 'V6-ACL-TBLE|Rule_20', 'V6-ACL-TBLE|Rule_40', 'V6-ACL-TBLE|Rule_60', 'V6-ACL-TBLE|Rule_80', 'V6-ACL-TBLE|Rule_111', 'V6-ACL-TBLE|Rule_9000', 'V6-ACL-TBLE|Rule_11100']), 'ACL_TABLE': dict_keys(['V4-ACL-TABLE', 'V6-ACL-TBLE']), 'DEVICE_METADATA': dict_keys(['localhost']), 'DEVICE_NEIGHBOR': dict_keys(['Ethernet112', 'Ethernet114', 'Ethernet116', 'Ethernet118']), 'PORT': dict_keys(['Ethernet0', 'Ethernet1', 'Ethernet2', 'Ethernet3', 'Ethernet4', 'Ethernet5', 'Ethernet6', 'Ethernet7', 'Ethernet8', 'Ethernet9', 'Ethernet10', 'Ethernet11', 'Ethernet12', 'Ethernet13', 'Ethernet14', 'Ethernet15', 'Ethernet16', 'Ethernet17', 'Ethernet18', 'Ethernet19', 'Ethernet20', 'Ethernet21', 'Ethernet22', 'Ethernet23', 'Ethernet24', 'Ethernet25', 'Ethernet26', 'Ethernet27', 'Ethernet28', 'Ethernet29', 'Ethernet30', 'Ethernet31', 'Ethernet32', 'Ethernet33', 'Ethernet34', 'Ethernet35', 'Ethernet36', 'Ethernet112']), 'PBH_HASH': dict_keys(['inner_ip_proto', 'inner_l4_dst_port', 'inner_l4_src_port', 'inner_dst_ipv4', 'inner_src_ipv4', 'inner_dst_ipv6', 'inner_src_ipv6']), 'PBH_RULE': dict_keys(['pbh_table|nvgre', 'pbh_table|vxlan']), 'PBH_TABLE': dict_keys(['pbh_table']), 'INTERFACE': dict_keys(['Ethernet112', 'Ethernet14', 'Ethernet16', 'Ethernet18', 'Ethernet112|2a04:5555:40:a709::2/126', 'Ethernet112|10.184.228.211/31', 'Ethernet14|2a04:5555:40:a749::2/126', 'Ethernet14|10.184.229.211/31', 'Ethernet16|2a04:5555:40:a789::2/126', 'Ethernet16|10.184.230.211/31', 'Ethernet18|2a04:5555:40:a7c9::2/126', 'Ethernet18|10.184.231.211/31']), 'LOOPBACK_INTERFACE': dict_keys(['Loopback0', 'Loopback0|2a04:5555:40:4::4e9/128', 'Loopback0|10.184.8.233/32']), 'BREAKOUT_CFG': dict_keys(['Ethernet0', 'Ethernet4', 'Ethernet8']), 'VERSIONS': dict_keys(['DATABASE']), 'FLEX_COUNTER_TABLE': dict_keys(['BUFFER_POOL_WATERMARK', 'DEBUG_COUNTER', 'PFCWD', 'PG_WATERMARK', 'PORT', 'PORT_RATES', 'PORT_BUFFER_DROP', 'QUEUE', 'QUEUE_WATERMARK']), 'CRM': dict_keys(['Config'])}
===================================================================================== 1 failed, 25 passed in 0.37 seconds =====================================================================================

Describe the results you expected:

nazariig@664912530dcf:/sonic/src/sonic-yang-mgmt$ python3 setup.py test
running pytest
Searching for ijson==2.6.1
Best match: ijson 2.6.1
Processing ijson-2.6.1-py3.7-linux-x86_64.egg

Using /sonic/src/sonic-yang-mgmt/.eggs/ijson-2.6.1-py3.7-linux-x86_64.egg
Searching for xmltodict==0.12.0
Best match: xmltodict 0.12.0
Processing xmltodict-0.12.0-py3.7.egg

Using /sonic/src/sonic-yang-mgmt/.eggs/xmltodict-0.12.0-py3.7.egg
running egg_info
writing sonic_yang_mgmt.egg-info/PKG-INFO
writing dependency_links to sonic_yang_mgmt.egg-info/dependency_links.txt
writing requirements to sonic_yang_mgmt.egg-info/requires.txt
writing top-level names to sonic_yang_mgmt.egg-info/top_level.txt
reading manifest file 'sonic_yang_mgmt.egg-info/SOURCES.txt'
writing manifest file 'sonic_yang_mgmt.egg-info/SOURCES.txt'
running build_ext
============================================================================================= test session starts =============================================================================================
platform linux -- Python 3.7.3, pytest-3.10.1, py-1.7.0, pluggy-0.8.0
rootdir: /sonic/src/sonic-yang-mgmt, inifile:
plugins: cov-2.6.0
collected 26 items

tests/test_sonic_yang_mgmt.py .                                                                                                                                                                         [  3%]
tests/libyang-python-tests/test_sonic_yang.py .........................                                                                                                                                 [100%]

========================================================================================== 26 passed in 0.29 seconds ==========================================================================================

Output of `show version`:

(paste your output here)

Output of `show techsupport`:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

N/A

The text was updated successfully, but these errors were encountered:

zhangyanzhao · 2021-11-04T17:54:55Z

@praveen-li can you please help to take a look? Thanks.

praveen-li · 2021-11-05T23:02:03Z

Suggest using deepdiff to get more user friendly results.

@nazariig : Sure, if you feel, it will help, Please go ahead and make the changes. Thx.

zhangyanzhao · 2022-01-27T18:18:53Z

@nazariig please check with @praveen-li between 8-10 AM PST time if you have any question.

praveen-li · 2022-02-14T23:50:05Z

@zhangyanzhao
fixed in https://github.com/Azure/sonic-buildimage/pull/8318/files. We can close this issue.

nazariig mentioned this issue May 27, 2021

[pbh]: Add YANG model #7461

Merged

1 task

nazariig added Help Wanted 🆘 Unit Tests YANG YANG model related changes labels May 27, 2021

zhangyanzhao assigned nazariig Nov 18, 2021

nazariig closed this as completed Feb 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[UT][YANG] Test_SonicYang.test_xlate_rev_xlate JSON comparison issue #7733

[UT][YANG] Test_SonicYang.test_xlate_rev_xlate JSON comparison issue #7733

nazariig commented May 27, 2021

zhangyanzhao commented Nov 4, 2021

praveen-li commented Nov 5, 2021

zhangyanzhao commented Jan 27, 2022

praveen-li commented Feb 14, 2022

[UT][YANG] Test_SonicYang.test_xlate_rev_xlate JSON comparison issue #7733

[UT][YANG] Test_SonicYang.test_xlate_rev_xlate JSON comparison issue #7733

Comments

nazariig commented May 27, 2021

Description

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Output of show version:

Output of show techsupport:

Additional information you deem important (e.g. issue happens only occasionally):

zhangyanzhao commented Nov 4, 2021

praveen-li commented Nov 5, 2021

zhangyanzhao commented Jan 27, 2022

praveen-li commented Feb 14, 2022

Output of `show version`:

Output of `show techsupport`: