[Mellanox MSN2100] Config reload with port breakout fails to complete successfully

[vivekvijaykumar]

Description

With a port configuration as follows on Mellanox MSN2100, config reload does not successfully complete. Orchagent seems to have terminated with error pointing to the creation of Router interfaces followed by switch shutdown -
"NOTICE swss#orchagent: :- handle_switch_shutdown_request: switch shutdown request"

The ports are created in the kernel and are actually UP. But SONiC is unable to reflect the oper status due to orchagent termination.

A reboot of the switch with the configs + port_breakout has no issues. A subsequent config reload causes the failure and happens with every config reload.

show platform sum
Platform: x86_64-mlnx_msn2100-r0
HwSKU: ACS-MSN2100
ASIC: mellanox

show platform syseeprom
TlvInfo Header:
Id String: TlvInfo
Version: 1
Total Length: 527
TLV Name Code Len Value
Product Name 0x21 64 MSN2100
Part Number 0x22 20 MSN2100-CB2FO
Serial Number 0x23 24 MT1734X00415
Base MAC Address 0x24 6 EC:0D:9A:4E:5A:00
Manufacture Date 0x25 19 08/21/2017 14:32:04
Device Version 0x26 1 0
MAC Addresses 0x2A 2 128
Manufacturer 0x2B 8 Mellanox
Vendor Extension 0xFD 36
Vendor Extension 0xFD 164
Vendor Extension 0xFD 36
Vendor Extension 0xFD 36
Vendor Extension 0xFD 36
Platform Name 0x28 18 x86_64-mlnx_x86-r0
ONIE Version 0x29 21 2016.11-5.1.0012-9600
CRC-32 0xFE 4 0x03683E96

(checksum valid)

Steps to reproduce the issue:

1. Edit port_config.ini under /usr/share/sonic/device/x86-64-mlnx_msn21000-r0/ACS-MSN2100/port_config.ini to reflect the port breakout
2. generate config_db.json using "sonic-cfggen -H -k ACS-MSN2100 --preset t1"
3. config reload with the new config_db.json

Describe the results you received:

Orchanget terminates and the ports are in partially configured state.

Interface Lanes Speed MTU Alias Oper Admin
Ethernet0 0,1,2,3 40G N/A Ethernet0 down up
Ethernet4 4,5,6,7 40G N/A Ethernet4 down up
Ethernet8 8,9,10,11 40G N/A Ethernet8 down up
Ethernet12 12,13,14,15 40G N/A Ethernet12 down up
Ethernet16 16,17,18,19 100G N/A Ethernet16 down up
Ethernet20 20,21,22,23 100G N/A Ethernet20 down up
Ethernet24 24,25,26,27 100G N/A Ethernet24 down up
Ethernet28 28,29,30,31 100G N/A Ethernet28 down up
Ethernet32 32,33,34,35 100G N/A Ethernet32 down up
Ethernet36 36,37,38,39 100G N/A Ethernet36 down up
Ethernet40 40,41,42,43 100G N/A Ethernet40 down up
Ethernet44 44,45,46,47 100G N/A Ethernet44 down up
Ethernet48 48,49,50,51 100G N/A Ethernet48 down up
Ethernet52 52,53,54,55 100G N/A Ethernet52 down up
Ethernet56 56,57,58,59 100G N/A Ethernet56 down up
Ethernet60 60,61,62,63 100G N/A Ethernet60 down up

Describe the results you expected:
All interfaces should be programmed correctly and valid cabled ports should be operationally UP.

Additional information you deem important (e.g. issue happens only occasionally):

SONiC Software Version: SONiC.master.0-dirty-20181203.155003
Distribution: Debian 9.6
Kernel: 4.9.0-7-amd64
Build commit: d608a83
Build date: Tue Dec 4 01:49:44 UTC 2018
Built by: apstrktr@bs101-01r
Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-orchagent-mlnx latest 121d531ec304 287.6 MB
docker-orchagent-mlnx master.0-dirty-20181203.155003 121d531ec304 287.6 MB
docker-syncd-mlnx latest f10489708ed1 365.9 MB
docker-syncd-mlnx master.0-dirty-20181203.155003 f10489708ed1 365.9 MB
docker-dhcp-relay latest d08c97525f88 258 MB
docker-dhcp-relay master.0-dirty-20181203.155003 d08c97525f88 258 MB
docker-database latest 59233aa0628f 256.6 MB
docker-database master.0-dirty-20181203.155003 59233aa0628f 256.6 MB
docker-snmp-sv2 latest a8d8d784a576 295.5 MB
docker-snmp-sv2 master.0-dirty-20181203.155003 a8d8d784a576 295.5 MB
docker-teamd latest f725975f8e93 275.8 MB
docker-teamd master.0-dirty-20181203.155003 f725975f8e93 275.8 MB
docker-router-advertiser latest 2fd04eda5885 254.3 MB
docker-router-advertiser master.0-dirty-20181203.155003 2fd04eda5885 254.3 MB
docker-platform-monitor latest 55714f39a4bf 288.2 MB
docker-platform-monitor master.0-dirty-20181203.155003 55714f39a4bf 288.2 MB
docker-lldp-sv2 latest 098e8da513ee 275.8 MB
docker-lldp-sv2 master.0-dirty-20181203.155003 098e8da513ee 275.8 MB
docker-fpm-frr latest 15efcf836e0a 282.7 MB
docker-fpm-frr master.0-dirty-20181203.155003 15efcf836e0a 282.7 MB

docker exec -it syncd sx_sdk --version
SX-SDK ETH 4.2.9108, SX-API 1.0.0, SXD 2.00

ip link show | grep Ethernet
27: Ethernet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
28: Ethernet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
29: Ethernet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
30: Ethernet12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
31: Ethernet16: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9100 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
32: Ethernet20: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9100 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
33: Ethernet24: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9100 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
34: Ethernet28: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9100 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
35: Ethernet32: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9100 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
36: Ethernet36: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9100 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
37: Ethernet40: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9100 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
38: Ethernet44: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9100 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
39: Ethernet48: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9100 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
40: Ethernet52: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9100 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
41: Ethernet56: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9100 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
42: Ethernet60: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9100 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000

port_cfg.ini
# name lanes speed index alias
Ethernet0 0,1,2,3 40000 0 Ethernet0
Ethernet4 4,5,6,7 40000 1 Ethernet4
Ethernet8 8,9,10,11 40000 2 Ethernet8
Ethernet12 12,13,14,15 40000 3 Ethernet12
Ethernet16 16,17,18,19 100000 4 Ethernet16
Ethernet20 20,21,22,23 100000 5 Ethernet20
Ethernet24 24,25,26,27 100000 6 Ethernet24
Ethernet28 28,29,30,31 100000 7 Ethernet28
Ethernet32 32,33,34,35 100000 8 Ethernet32
Ethernet36 36,37,38,39 100000 9 Ethernet36
Ethernet40 40,41,42,43 100000 10 Ethernet40
Ethernet44 44,45,46,47 100000 11 Ethernet44
Ethernet48 48,49,50,51 100000 12 Ethernet48
Ethernet52 52,53,54,55 100000 13 Ethernet52
Ethernet56 56,57,58,59 100000 14 Ethernet56
Ethernet60 60,61,62,63 100000 15 Ethernet60

docker exec -it swss ps axf
PID TTY STAT TIME COMMAND
199 ? Rs+ 0:00 ps axf
1 ? Ss+ 0:00 /usr/bin/python /usr/bin/supervisord
38 ? Sl 0:00 /usr/sbin/rsyslogd -n
66 ? Sl 0:01 /usr/bin/portsyncd -p /usr/share/sonic/hwsku/port_con
71 ? Sl 0:00 /usr/bin/intfsyncd
74 ? Sl 0:00 /usr/bin/neighsyncd
82 ? Sl 0:00 /usr/bin/vlanmgrd
100 ? Sl 0:00 /usr/bin/intfmgrd
105 ? Sl 0:00 /usr/bin/portmgrd
110 ? Sl 0:00 /usr/bin/buffermgrd -l /usr/share/sonic/hwsku/pg_prof
189 ? Sl 0:00 /usr/bin/vrfmgrd
194 ? Sl 0:00 /usr/bin/nbrmgrd

Dec 6 22:32:19.276506 spine-1 INFO syncd#supervisord: syncd Dec 06 22:32:19 ERROR EMAD_TRANSPORT: EMAD [27000058D13FD1] completed with status : [BAD PARAMETER]
Dec 6 22:32:19.277120 spine-1 INFO syncd#supervisord: syncd Dec 06 22:32:19 ERROR REG_BULK: sx_reg_layer_sxd_reg_call - Error at access reg call reg_id 0x8002, error: FW RETURN VALUE != SUCCESS
Dec 6 22:32:19.277120 spine-1 INFO syncd#supervisord: syncd Dec 06 22:32:19 ERROR REG_BULK: __reg_bulk_post_completion_cb Error got in completion data sxd_status [15]-[FW RETURN VALUE != SUCCESS].
Dec 6 22:32:19.277120 spine-1 INFO syncd#supervisord: syncd Dec 06 22:32:19 ERROR ROUTER: Failed to RITR set: [FW RETURN VALUE != SUCCESS].
Dec 6 22:32:19.277167 spine-1 INFO syncd#supervisord: syncd Dec 06 22:32:19 ERROR ROUTER: Failed to create router interface RITR Failed , err = Driver's Return Status is Non-Zero
Dec 6 22:32:19.277752 spine-1 INFO syncd#supervisord: syncd Dec 06 22:32:19 ERROR ROUTER: Failed to create router interface in HWD layer, err= Driver's Return Status is Non-Zero.
Dec 6 22:32:19.280441 spine-1 ERR syncd#syncd: :- processEvent: attr: SAI_ROUTER_INTERFACE_ATTR_VIRTUAL_ROUTER_ID: oid:0x3000000000002
Dec 6 22:32:19.280560 spine-1 ERR syncd#syncd: :- processEvent: attr: SAI_ROUTER_INTERFACE_ATTR_SRC_MAC_ADDRESS: EC:0D:9A:4E:5A:00
Dec 6 22:32:19.281235 spine-1 NOTICE swss#orchagent: :- handle_switch_shutdown_request: switch shutdown request
Dec 6 22:32:19.281335 spine-1 ERR syncd#syncd: :- processEvent: attr: SAI_ROUTER_INTERFACE_ATTR_TYPE: SAI_ROUTER_INTERFACE_TYPE_PORT
Dec 6 22:32:19.281387 spine-1 ERR syncd#syncd: :- processEvent: attr: SAI_ROUTER_INTERFACE_ATTR_PORT_ID: oid:0x1000000000170
Dec 6 22:32:19.281437 spine-1 ERR syncd#syncd: :- processEvent: attr: SAI_ROUTER_INTERFACE_ATTR_MTU: 1492
Dec 6 22:32:19.281487 spine-1 ERR syncd#syncd: :- processEvent: failed to execute api: create, key: SAI_OBJECT_TYPE_ROUTER_INTERFACE:oid:0x60000000002d0, status: SAI_STATUS_FAILURE
Dec 6 22:32:19.281536 spine-1 INFO syncd#supervisord: syncd Dec 06 22:32:19 ERROR ROUTER: Failed to create rif Port Vlan, err = Driver's Return Status is Non-Zero
Dec 6 22:32:19.281586 spine-1 INFO syncd#supervisord: syncd Dec 06 22:32:19 ERROR SAI_RIF: mlnx_sai_rif.c[422]- mlnx_rif_sx_init: Failed to create router interface - Driver's Return Status is Non-Zero.
Dec 6 22:32:19.281657 spine-1 ERR syncd#syncd: :- syncd_main: Runtime error: :- processEvent: failed to execute api: create, key: SAI_OBJECT_TYPE_ROUTER_INTERFACE:oid:0x60000000002d0, status: SAI_STATUS_FAILURE
Dec 6 22:32:19.281706 spine-1 NOTICE syncd#syncd: :- exit_and_notify: sending switch_shutdown_request notification to OA
Dec 6 22:32:19.281885 spine-1 NOTICE syncd#syncd: :- exit_and_notify: notification send successfull

Config_db.json
{
"DEVICE_METADATA": {
"localhost": {
"bgp_asn": "",
"docker_routing_config_mode": "split",
"hostname": "spine-1",
"hwsku": "ACS-MSN2100",
"mac": "ec:0d:9a:4e:5a:00",
"platform": "x86_64-mlnx_msn2100-r0",
"type": "LeafRouter"
}
},
"INTERFACE": {
"Ethernet0|172.16.0.2/31": {},
"Ethernet12|172.16.0.8/31": {},
"Ethernet4|172.16.0.4/31": {},
"Ethernet8|172.16.0.6/31": {}
},
"LOOPBACK_INTERFACE": {
"Loopback0|172.16.0.18/32": {}
},
"PORT": {
"Ethernet0": {
"admin_status": "up",
"description": "facing_leaf-1:Ethernet1/1",
"lanes": "0,1,2,3",
"alias": "Ethernet0",
"speed": "40000"
},
"Ethernet12": {
"admin_status": "up",
"description": "facing_leaf-2:Ethernet50/1",
"lanes": "12,13,14,15",
"alias": "Ethernet12",
"speed": "40000"
},
"Ethernet16": {
"admin_status": "up",
"description": "",
"lanes": "16,17,18,19",
"alias": "Ethernet16",
"speed": "100000"
},
"Ethernet20": {
"admin_status": "up",
"description": "",
"lanes": "20,21,22,23",
"alias": "Ethernet20",
"speed": "100000"
},
"Ethernet24": {
"admin_status": "up",
"description": "",
"lanes": "24,25,26,27",
"alias": "Ethernet24",
"speed": "100000"
},
"Ethernet28": {
"admin_status": "up",
"description": "",
"lanes": "28,29,30,31",
"alias": "Ethernet28",
"speed": "100000"
},
"Ethernet32": {
"admin_status": "up",
"description": "",
"lanes": "32,33,34,35",
"alias": "Ethernet32",
"speed": "100000"
},
"Ethernet36": {
"admin_status": "up",
"description": "",
"lanes": "36,37,38,39",
"alias": "Ethernet36",
"speed": "100000"
},
"Ethernet4": {
"admin_status": "up",
"description": "facing_leaf-1:Ethernet1/2",
"lanes": "4,5,6,7",
"alias": "Ethernet4",
"speed": "40000"
},
"Ethernet40": {
"admin_status": "up",
"description": "",
"lanes": "40,41,42,43",
"alias": "Ethernet40",
"speed": "100000"
},
"Ethernet44": {
"admin_status": "up",
"description": "",
"lanes": "44,45,46,47",
"alias": "Ethernet44",
"speed": "100000"
},
"Ethernet48": {
"admin_status": "up",
"description": "",
"lanes": "48,49,50,51",
"alias": "Ethernet48",
"speed": "100000"
},
"Ethernet52": {
"admin_status": "up",
"description": "",
"lanes": "52,53,54,55",
"alias": "Ethernet52",
"speed": "100000"
},
"Ethernet56": {
"admin_status": "up",
"description": "",
"lanes": "56,57,58,59",
"alias": "Ethernet56",
"speed": "100000"
},
"Ethernet60": {
"admin_status": "up",
"description": "",
"lanes": "60,61,62,63",
"alias": "Ethernet60",
"speed": "100000"
},
"Ethernet8": {
"admin_status": "up",
"description": "facing_leaf-2:Ethernet49/1",
"lanes": "8,9,10,11",
"alias": "Ethernet8",
"speed": "40000"
}
},
"PORTCHANNEL": {},
"PORTCHANNEL_INTERFACE": {},
"VLAN": {},
"VLAN_INTERFACE": {},
"VLAN_MEMBER": {}
}
Detailed logs and output attached.

sonic_port_breakout.txt
syslog.txt

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

[yxieca]

@andriymoroz-mlnx is this still an issue?