loading minigraph from sonic-mgmt get crashed with orchagent in latest builds

[mfriyas]

Device dell s6100 was loaded with latest 790 build. When am trying to load T0 minigraph from sonic-mgmt device gets crashed with orchagnet core. Pasted here the core logs and syslog output.

Core

root@sonic-s6100-01:/var/core# ls -lrt
total 544
-rw-rw-rw- 1 root root 554860 Nov 30 09:51 orchagent.1543571489.45.core.gz

Syslog output during crash:-

Nov 30 09:51:02.024379 sonic-s6100-01 INFO syncd#supervisord: syncd rc: Port
Nov 30 09:51:02.024504 sonic-s6100-01 INFO syncd#supervisord: syncd  modes initialized#015
Nov 30 09:51:02.277481 sonic-s6100-01 INFO syncd#supervisord: syncd Common SDK init completed#015#015
Nov 30 09:51:17.086336 sonic-s6100-01 INFO syncd#supervisord: syncd rc: platform SDK init complete#015
Nov 30 09:51:19.086603 sonic-s6100-01 ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#012  File "/usr/local/lib/python3.6/dist-packages/ax_interface/mib.py", line 40, in start#012    self.reinit_data()#012  File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/ieee802_1ab.py", line 540, in reinit_data#012    _, _, _, _, self.oid_name_map = mibs.init_sync_d_interface_tables(self.db_conn)#012  File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/__init__.py", line 178, in init_sync_d_interface_tables#012    if_name_map, if_id_map = port_util.get_interface_oid_map(db_conn)#012  File "/usr/local/lib/python3.6/dist-packages/swsssdk/port_util.py", line 51, in get_interface_oid_map#012    if_name_map = db.get_all('COUNTERS_DB', 'COUNTERS_PORT_NAME_MAP', blocking=True)#012  File "/usr/local/lib/python3.6/dist-packages/swsssdk/interface.py", line 38, in wrapped#012    ret_data = f(inst, db_name, *args, **kwargs)#012  File "/usr/local/lib/python3.6/dist-packages/swsssdk/interface.py", line 324, in get_all#012    raise UnavailableDataError(message, _hash)#012swsssdk.exceptions.UnavailableDataError: Key 'COUNTERS_PORT_NAME_MAP' unavailable in database 'COUNTERS_DB'
Nov 30 09:51:20.578577 sonic-s6100-01 INFO swss.sh[15524]: 2018-11-30 09:51:20,578 INFO exited: enable_counters (exit status 0; expected)
Nov 30 09:51:21.391964 sonic-s6100-01 WARNING syncd#syncd: :- saiGetHardwareInfo: failed to get switc

[stcheng]

Hello, snmp related logs are not relevant, could you paste more logs here? orchagent related logs.

[mfriyas]

orchagent.1543571489.45.core.gz

LATEST crash code below:-
orchagent.1544078561.44.core.gz

Syslog files:-
syslog.txt

[stcheng]

@mfriyas the syslog was still not complete. it is better to tar all the syslog files. you could send to me via email.

[mfriyas]

After doing this , the issue was not seen.

There’s a file inside docker swss needs to be updated:
Docker exec -it swss bash
File /etc/sonic/share/templates/ipinip.json.j2

https://github.com/Azure/sonic-buildimage/blob/master/dockers/docker-orchagent/ipinip.json.j2
remove line 58 – 77 and restart and the device will survive.

This is the issue with Broadcom SAI 3.1 and the current master branch.
Once we move to a newer version of Broadcom SAI this will be fixed.

hence closing this issue