Change the log messages in addKernelNeigh/Route from ERROR to INFO.

[judyjoseph]

What I did
Change the Error reporting for API's addKernelRoute, addKernelNeigh from ERR to INFO as the code logic already deletes and retries the creation of Neigh/Route

For the test tests/test_virtual_chassis.py::test_chassis_system_neigh the following change was made

1. Grouped the neighbor creation, validation of the DB entry present in local and remote LC into a new function chassis_system_neigh_create()
2. Call this function first with mac_address = "00:01:02:03:04:77", action as "add" to create a new neighbor entry.
3. Call this function again with mac_address = "00:01:02:03:04:05", action as "change" to update the mac_addres of that neighbor entry already created before.

Why I did it
There was sonic-mgmt test failures in multi-asic devices where the Neighbor and routes learnt in in one ASIC is synced with the nearby ASIC, and in case of mac move there is possibility for an existing entry to be updated with a new mac address ( using the command /sbin/ip neigh add lladdr dev ) . This causes error as ip neigh/route add command will fail (error : nbrmgrd RTNETLINK answers: File exists)

E               Failed: Processes "['analyze_logs--<MultiAsicSonicHost str2--lc1-1>']" failed with exit code "1"
E               Exception:
E               expected_match: 0
E               expected_missing_match: 0
E               match: 1
E               
E               Match Messages:
E               Aug 16 16:39:19.939498 str2--lc1-1 ERR swss1#nbrmgrd: :- doStateSystemNeighTask: Route entry add on dev Ethernet-IB1 failed for 'str2--lc1-1|asic0|PortChannel102|29.0.0.2'
E               
E               Traceback:
E               Traceback (most recent call last):
E                 File "/home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/common/helpers/parallel.py", line 31, in run
E                   Process.run(self)
E                 File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
E                   self._target(*self._args, **self._kwargs)
E                 File "/home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/common/helpers/parallel.py", line 226, in wrapper
E                   target(*args, **kwargs)
E                 File "/home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/common/plugins/loganalyzer/__init__.py", line 39, in analyze_logs
E                   dut_analyzer.analyze(markers[node.hostname])
E                 File "/home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/common/plugins/loganalyzer/loganalyzer.py", line 369, in analyze
E                   self._verify_log(analyzer_summary)
E                 File "/home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/common/plugins/loganalyzer/loganalyzer.py", line 133, in _verify_log
E                   raise LogAnalyzerError(result_str)
E               LogAnalyzerError: expected_match: 0
E               expected_missing_match: 0
E               match: 1
E               
E               Match Messages:
E               Aug 16 16:39:19.939498 str2--lc1-1 ERR swss1#nbrmgrd: :- doStateSystemNeighTask: Route entry add on dev Ethernet-IB1 failed for 'str2--lc1-1|asic0|PortChannel102|29.0.0.2'

---------------------------------------------------- generated xml file: /home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/logs/tr.xml ------------------------------------------------------
=================================================================================== short test summary info ====================================================================================
ERROR arp/test_neighbor_mac_noptf.py::TestNeighborMacNoPtf::testNeighborMacNoPtf[4-str2--lc1-1-0] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost str2--lc1-1>']" failed wit...
============================================================================= 1 passed, 1 error in 316.96 seconds ==============================================================================

How I verified it
With this fix the tests passed

jujoseph@c3e1277334f8:~/CHASSIS/CHASSIS/sonic-mgmt-int/tests$ ./run_tests.sh -c arp/test_neighbor_mac_noptf.py::TestNeighborMacNoPtf::testNeighborMacNoPtf[4-str2--lc1-1-0]  -n vms29-t2--1 -d str2--lc1-1 -i "../ansible/str2,../ansible/veos" -t "t2,any" -S "dualtor_io" -e "--skip_sanity" -u
=== Running tests in groups ===
/usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.exceptions import InvalidSignature
===================================================================================== test session starts ======================================================================================
platform linux2 -- Python 2.7.17, pytest-4.6.5, py-1.11.0, pluggy-0.13.1
ansible: 2.8.12
rootdir: /home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests, inifile: pytest.ini
plugins: forked-1.3.0, metadata-1.11.0, xdist-1.28.0, html-1.22.1, allure-pytest-2.8.22, repeat-0.9.1, ansible-2.2.2
collected 1 item                                                                                                                                                                               

arp/test_neighbor_mac_noptf.py::TestNeighborMacNoPtf::testNeighborMacNoPtf[4-str2--lc1-1-0] PASSED                                                                                   [100%]

----------------------------------------------------- generated xml file: /home/jujoseph/CHASSIS/CHASSIS/sonic-mgmt-int/tests/logs/tr.xml ------------------------------------------------------
================================================================================== 1 passed in 317.49 seconds ==================================================================================
INFO:root:Can not get Allure report URL. Please check logs
jujoseph@c3e1277334f8:~/CHASSIS/CHASSIS/sonic-mgmt-int/tests$

Details if related

[abdosi]

what is test case change about ? Can you add that in PR description.

[lgtm-com]

This pull request introduces 3 alerts when merging 9ddbd65 into dec4570 - view on LGTM.com

new alerts:

• 3 for Unused local variable

[judyjoseph]

what is test case change about ? Can you add that in PR description.

I will update. Basic idea was to add a case where there is a mac address change for neighbor, and get code coverage.

The changes here are mainly regrouping the existing code into a function which can be called multiple times to change the mac address of a neighbor.

[lgtm-com]

This pull request introduces 3 alerts when merging 88ff407 into dec4570 - view on LGTM.com

new alerts:

• 3 for Variable defined multiple times

[lgtm-com]

This pull request introduces 2 alerts when merging 33f765e into 4aaeec9 - view on LGTM.com

new alerts:

• 1 for Mismatch in multiple assignment
• 1 for Non-callable called

[lgtm-com]

This pull request introduces 1 alert when merging ee3eef0 into a4b8992 - view on LGTM.com

new alerts:

• 1 for Non-callable called

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: judyjoseph (18573f9, 065b9e4, 9ddbd65, 88ff407, 2d5d5e8, 33f765e, ee3eef0, 44aae08)

[judyjoseph]

/easycla

[judyjoseph]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[judyjoseph]

Closing this PR in favor of PR : #2437. For some reason the PR build/tests get stuck here.