Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DPB] [portsorch] Orchagent missing DELETE update from CONFIG_DB during DPB #10005

Open
alexrallen opened this issue Feb 16, 2022 · 11 comments
Open

[DPB] [portsorch] Orchagent missing DELETE update from CONFIG_DB during DPB #10005

alexrallen opened this issue Feb 16, 2022 · 11 comments
Assignees
@praveen-li
Labels
Help Wanted 🆘 Issue for 202111 LINKEDIN Port Breakout 🔱 Triaged this issue has been triaged

Comments

@alexrallen
Copy link
Contributor

Description

When multiple ports are configured with a LAG which is belong to a VLAN which is assigned multiple IP addresses and these ports are broken out via DPB with the force option (-f) there is a chance (~20%) that orchagent will miss one of the delete updates and will not attempt to delete the port which will result in the following error from the config breakout command.

Ports are present in ASIC DB after 60 secs
Ports are present in ASIC DB after 60 secs
[ERROR] Port breakout Failed!!! Opting Out
Failed to break out Port. Error: Critical Failure, Ports  are not Deleted from ASIC DB, Bail Out  !!!

Steps to reproduce the issue:

# configure dependencies on ports
sudo config portchannel add PortChannel0001
sudo config portchannel member add PortChannel0001 Ethernet28
sudo config portchannel add PortChannel0002
sudo config portchannel member add PortChannel0002 Ethernet32
sudo config portchannel add PortChannel0003
sudo config portchannel member add PortChannel0003 Ethernet76
sudo config portchannel add PortChannel0004
sudo config portchannel member add PortChannel0004 Ethernet80
sudo config vlan add 3362
sudo config vlan member add 3362 PortChannel0001
sudo config vlan member add 3362 PortChannel0002
sudo config vlan member add 3362 PortChannel0003
sudo config vlan member add 3362 PortChannel0004
sudo config interface ip add Vlan3362 10.0.0.1/24
sudo config interface ip add Vlan3362 10.0.0.2/24
sudo config interface ip add Vlan3362 10.0.0.3/24
sudo config interface ip add Vlan3362 10.0.0.4/24

# configure with force flag (should work on all ports)
sudo config interface breakout Ethernet28 4x50G[25G,10G,1G] -y -f
sudo config interface breakout Ethernet32 4x50G[25G,10G,1G] -y -f
sudo config interface breakout Ethernet76 4x50G[25G,10G,1G] -y -f
sudo config interface breakout Ethernet80 4x50G[25G,10G,1G] -y -f

Describe the results you received:

2022-02-03 10:34:31 [    INFO] (proxy_ssh_engine.py:224) sudo config interface breakout Ethernet80 2x100G[50G,25G,10G,1G] -y -f


Running Breakout Mode : 1x200G[100G,50G,40G,25G,10G,1G] 
Target Breakout Mode : 2x100G[50G,25G,10G,1G]

Ports to be deleted : 
 {
    "Ethernet80": "200000"
}
Ports to be added : 
 {
    "Ethernet80": "100000",
    "Ethernet82": "100000"
}
�[4m
After running Logic to limit the impact

Final list of ports to be deleted : 
 {
    "Ethernet80": "200000"
} 
Final list of ports to be added :  
 {
    "Ethernet80": "100000",
    "Ethernet82": "100000"
}
sonic_yang(6):Note: Below table(s) have no YANG models: KDUMP, SNMP, SNMP_COMMUNITY, XCVRD_LOG
Below Config can not be verified, It may cause harm to the system
 {}
Do you wish to Continue? [y/N]: !!!  Critical Failure, Ports                     are not Deleted from ASIC DB, Bail Out  !!!
Ports are present in ASIC DB after 60 secs
Ports are present in ASIC DB after 60 secs
[ERROR] Port breakout Failed!!! Opting Out
Failed to break out Port. Error: 
�admin@r-anaconda-51:~$

Describe the results you expected:

Successful port breakout.

Output of show version:

Latest master

Output of show techsupport:

sonic_dump_r-anaconda-51_20220203_083601.tar.gz

Additional information you deem important (e.g. issue happens only occasionally):

It looks like something is causing orchagent doPortTask to miss the update from ConfigDB that contains the port deletion.

Orchagent successfully recieves the update to remove the LAG membership and successfully completes the removal

Feb  3 08:33:14.022886 r-anaconda-51 NOTICE syncd#SDK: [SAI_LAG.NOTICE] mlnx_sai_lag.c[1632]- mlnx_remove_lag_member: Remove LAG member (10011,3)
Feb  3 08:33:14.354794 r-anaconda-51 NOTICE teamd#teammgrd: :- removeLagMember: Remove Ethernet80 from port channel PortChannel0004
Feb  3 08:33:14.355020 r-anaconda-51 NOTICE syncd#SDK: [SAI_LAG.NOTICE] mlnx_sai_lag.c[1721]- mlnx_remove_lag_member: Removed SAI LAG member
Feb  3 08:33:14.356589 r-anaconda-51 NOTICE swss#orchagent: :- removeLagMember: Remove member Ethernet80 from LAG PortChannel0004 lid:200000000079a lmid:1b00000000079b
Feb  3 08:33:14.357580 r-anaconda-51 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] mlnx_sai_utils.c[1833]- set_dispatch_attrib_handler: Set VLAN_TAG, key:host interface 28, val:STRIP
Feb  3 08:33:14.358113 r-anaconda-51 NOTICE swss#orchagent: :- setHostIntfsStripTag: Set SAI_HOSTIF_VLAN_TAG_STRIP to host interface: Ethernet80
Feb  3 08:33:14.359090 r-anaconda-51 NOTICE swss#orchagent: :- updatePortOperStatus: Port PortChannel0004 oper state set from down to down

Orchagent also successfully recieves the admin down update

Feb  3 08:33:13.925279 r-anaconda-51 NOTICE swss#orchagent: :- doPortTask: Set port Ethernet80 admin status to down
Feb  3 08:33:14.018877 r-anaconda-51 INFO kernel: [  757.466514] device Ethernet80 left promiscuous mode
Feb  3 08:33:14.019235 r-anaconda-51 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet80(ok:down) to state db
Feb  3 08:33:14.023142 r-anaconda-51 NOTICE swss#portmgrd: :- doTask: Configure Ethernet80 admin status to down

portmgrd sucessfully recieves the delete port update

Feb  3 08:33:14.023160 r-anaconda-51 NOTICE swss#portmgrd: :- doTask: Delete Port: Ethernet80

However nowhere in the logs does orchagent attempt to handle the port deletion... here is an example of what we might expect from the previous deleted ports...

Feb  3 08:33:11.486691 r-anaconda-51 WARNING swss#orchagent: :- doPortTask: Unable to remove port Ethernet76: ref count 1
Feb  3 08:33:11.486691 r-anaconda-51 WARNING swss#orchagent: :- doPortTask: Unable to remove port Ethernet76: ref count 1
Feb  3 08:33:11.486759 r-anaconda-51 WARNING swss#orchagent: :- doPortTask: Unable to remove port Ethernet76: ref count 1
Feb  3 08:33:11.486811 r-anaconda-51 WARNING swss#orchagent: :- doPortTask: Unable to remove port Ethernet76: ref count 1
Feb  3 08:33:11.543623 r-anaconda-51 NOTICE swss#orchagent: :- setHostIntfsStripTag: Set SAI_HOSTIF_VLAN_TAG_STRIP to host interface: Ethernet76
Feb  3 08:33:11.543633 r-anaconda-51 NOTICE swss#orchagent: :- doPortTask: Deleting Port Ethernet76
Feb  3 08:33:11.544408 r-anaconda-51 NOTICE swss#orchagent: :- deInitPort: De-Initialized port Ethernet76
Feb  3 08:33:11.544418 r-anaconda-51 NOTICE swss#orchagent: :- doPortTask: Removing hostif d000000000775 for Port Ethernet76

The apparent lack of an attempt to delete the port (as we don't see any errors related to port deletion) seems to indicate that orchagent missed this update. However we know the update was sent as portmgrd was able to consume it and process it, and the CONFIG_DB from the techsupport dump also shows the port as removed.
@dgsudharsan
Copy link
Collaborator

@praveen-li @zhenggen-xu Can you please investigate?

@praveen-li
Copy link
Member

Can you provide
-- Config DB before DPB.
-- Writes done to config DB during DPB. This should be in syslog.
-- full snapshot of relevant logs during one round of DPB.
-- dump of entire redis_db once error happened.

@alexrallen
Copy link
Contributor Author

All this information should be available in the techsupport (it contains all the syslogs and the redis dump). I do not have the ConfigDB prior to writing but that should be pretty easy to reverse engineer from the config_db dump in the techsupport and the syslogs from DPB.

@zhangyanzhao zhangyanzhao added the Triaged this issue has been triaged label Mar 2, 2022
@zhangyanzhao
Copy link
Collaborator

Praveen, please help to take a look. Thanks.

@alexrallen
Copy link
Contributor Author

@praveen-li I do not see frequent reproductions of this, however I will check internally at NVIDIA to see if we are still seeing this in our testbeds.

At the very least I would like someone familiar with DPB to take a look at this issue and see if they come to the same conclusion that I did and see if anyone has any thoughts on why we might see this behavior.

This is directly related to the DPB workflow, we don't see this issue (and wouldn't ever see this issue) except in a case where a port is being dynamically removed so I believe it is best to start triaging this bug with DPB in mind and trace out the established flows we expect to occur, and if we find another module likely to be at fault then to defer to the relevant owner.

@zhenggen-xu
Copy link
Collaborator

I looked into the swss.rec, and it did seem to miss the DEL op for Ethernet80. To be certain it is an issue in Redis lib or orchagent issue, can you record the redis events with "redis-cli monitor" during the DPB operations.

@zhangyanzhao
Copy link
Collaborator

Need capture more logs to determine which component missed the deletion operation in the invoke chain.

@puffc
Copy link
Contributor

puffc commented Nov 27, 2023

@alexrallen @praveen-li I'm able to see this problem with the latest vs image.

@dgsudharsan
Copy link
Collaborator

@puffc This issue has not been fixed yet and its currently help wanted by the community.

@puffc
Copy link
Contributor

puffc commented Nov 28, 2023

@dgsudharsan @praveen-li @zhenggen-xu Please find command and redis-cli monitor output below:

admin@sonic:~$ sudo config interface breakout Ethernet0 4x10G[10G,1G] -fy
 
Running Breakout Mode : 1x40G[40G,10G,1G]
Target Breakout Mode : 4x10G[10G,1G]
 
Ports to be deleted :
{
    "Ethernet0": "40000"
}
Ports to be added :
{
    "Ethernet0": "10000",
    "Ethernet1": "10000",
    "Ethernet2": "10000",
    "Ethernet3": "10000"
}
!!!  Critical Failure, Ports                     are not Deleted from ASIC DB, Bail Out  !!!
Ports are present in ASIC DB after 60 secs
Ports are present in ASIC DB after 60 secs
[ERROR] Port breakout Failed!!! Opting Out
Failed to break out Port. Error:

redis-cli monitor:

...
...
1701164133.829037 [4 127.0.0.1:54828] "HSET" "PORT|Ethernet0" "admin_status" "down"
1701164133.830566 [4 unix:/var/run/redis/redis.sock] "HGETALL" "PORT|Ethernet0"
1701164133.830656 [4 unix:/var/run/redis/redis.sock] "HGETALL" "PORT|Ethernet0"
1701164133.830723 [0 127.0.0.1:54830] "SELECT" "4"
1701164133.830752 [6 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE|Ethernet0"
1701164133.830937 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "6875900592cdd1621c6191fe038ec3b29775aa13" "6" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "_PORT_TABLE:Ethernet0" "_PORT_TABLE:Ethernet0" "_PORT_TABLE:Ethernet0" "G" "Ethernet0" "alias" "fortyGigE0/0" "index" "0" "lanes" "25,26,27,28" "speed" "40000"
1701164133.830967 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1701164133.830989 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "alias" "fortyGigE0/0"
1701164133.831005 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "index" "0"
1701164133.831018 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "lanes" "25,26,27,28"
1701164133.831031 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "speed" "40000"
1701164133.831044 [0 lua] "PUBLISH" "PORT_TABLE_CHANNEL@0" "G"
1701164133.831728 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88270a7c5c90583e56425aca8af8a4b8c39fe757" "3" "PORT_TABLE_KEY_SET" "PORT_TABLE:" "PORT_TABLE_DEL_SET" "1024" "_"
1701164133.831743 [0 lua] "SPOP" "PORT_TABLE_KEY_SET" "1024"
1701164133.831773 [0 lua] "SREM" "PORT_TABLE_DEL_SET" "Ethernet0"
1701164133.831778 [0 lua] "HGETALL" "_PORT_TABLE:Ethernet0"
1701164133.831786 [0 lua] "HSET" "PORT_TABLE:Ethernet0" "alias" "fortyGigE0/0"
1701164133.831802 [0 lua] "HSET" "PORT_TABLE:Ethernet0" "index" "0"
1701164133.831815 [0 lua] "HSET" "PORT_TABLE:Ethernet0" "lanes" "25,26,27,28"
1701164133.831828 [0 lua] "HSET" "PORT_TABLE:Ethernet0" "speed" "40000"
1701164133.831840 [0 lua] "DEL" "_PORT_TABLE:Ethernet0"
1701164133.831927 [4 127.0.0.1:54830] "DEL" "PORT|Ethernet0"
1701164133.832368 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88270a7c5c90583e56425aca8af8a4b8c39fe757" "3" "PORT_TABLE_KEY_SET" "PORT_TABLE:" "PORT_TABLE_DEL_SET" "1024" "_"
1701164133.832380 [0 lua] "SPOP" "PORT_TABLE_KEY_SET" "1024"
1701164133.832412 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.832443 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.832808 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.832845 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.832861 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE_CHANNEL@1" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_NUMBER_OF_INGRESS_PRIORITY_GROUPS\",\"0\"]" "Sget" "G"
1701164133.832878 [1 lua] "LPUSH" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_NUMBER_OF_INGRESS_PRIORITY_GROUPS\",\"0\"]" "Sget"
1701164133.832901 [1 lua] "PUBLISH" "ASIC_STATE_CHANNEL@1" "G"
1701164133.833039 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.833055 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE" "128" "0"
1701164133.833072 [1 lua] "LRANGE" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.833081 [1 lua] "LTRIM" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.833269 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.833298 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.833315 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE_CHANNEL@1" "SAI_STATUS_SUCCESS" "[\"SAI_PORT_ATTR_NUMBER_OF_INGRESS_PRIORITY_GROUPS\",\"8\"]" "Sgetresponse" "G"
1701164133.833329 [1 lua] "LPUSH" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "SAI_STATUS_SUCCESS" "[\"SAI_PORT_ATTR_NUMBER_OF_INGRESS_PRIORITY_GROUPS\",\"8\"]" "Sgetresponse"
1701164133.833349 [1 lua] "PUBLISH" "GETRESPONSE_CHANNEL@1" "G"
1701164133.833749 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE" "128" "1"
1701164133.833762 [1 lua] "LRANGE" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.833770 [1 lua] "LTRIM" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.833851 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.833967 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE_CHANNEL@1" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_INGRESS_PRIORITY_GROUP_LIST\",\"8:oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0\"]" "Sget" "G"
1701164133.833985 [1 lua] "LPUSH" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_INGRESS_PRIORITY_GROUP_LIST\",\"8:oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0\"]" "Sget"
1701164133.834008 [1 lua] "PUBLISH" "ASIC_STATE_CHANNEL@1" "G"
1701164133.834131 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE" "128" "0"
1701164133.834143 [1 lua] "LRANGE" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.834153 [1 lua] "LTRIM" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.834576 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_INGRESS_PRIORITY_GROUP:oid:0x1a000000000204" "NULL" "NULL"
1701164133.834621 [6 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE|Ethernet0"
1701164133.834681 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_INGRESS_PRIORITY_GROUP:oid:0x1a000000000205" "NULL" "NULL"
1701164133.834729 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_INGRESS_PRIORITY_GROUP:oid:0x1a000000000206" "NULL" "NULL"
1701164133.834771 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_INGRESS_PRIORITY_GROUP:oid:0x1a000000000207" "NULL" "NULL"
1701164133.834809 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_INGRESS_PRIORITY_GROUP:oid:0x1a000000000208" "NULL" "NULL"
1701164133.834848 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_INGRESS_PRIORITY_GROUP:oid:0x1a000000000209" "NULL" "NULL"
1701164133.834887 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_INGRESS_PRIORITY_GROUP:oid:0x1a00000000020a" "NULL" "NULL"
1701164133.834926 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_INGRESS_PRIORITY_GROUP:oid:0x1a00000000020b" "NULL" "NULL"
1701164133.834972 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE_CHANNEL@1" "SAI_STATUS_SUCCESS" "[\"SAI_PORT_ATTR_INGRESS_PRIORITY_GROUP_LIST\",\"8:oid:0x1a000000000204,oid:0x1a000000000205,oid:0x1a000000000206,oid:0x1a000000000207,oid:0x1a000000000208,oid:0x1a000000000209,oid:0x1a00000000020a,oid:0x1a00000000020b\"]" "Sgetresponse" "G"
1701164133.834992 [1 lua] "LPUSH" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "SAI_STATUS_SUCCESS" "[\"SAI_PORT_ATTR_INGRESS_PRIORITY_GROUP_LIST\",\"8:oid:0x1a000000000204,oid:0x1a000000000205,oid:0x1a000000000206,oid:0x1a000000000207,oid:0x1a000000000208,oid:0x1a000000000209,oid:0x1a00000000020a,oid:0x1a00000000020b\"]" "Sgetresponse"
1701164133.835016 [1 lua] "PUBLISH" "GETRESPONSE_CHANNEL@1" "G"
1701164133.835843 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "6875900592cdd1621c6191fe038ec3b29775aa13" "3" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "G" "Ethernet0" "mtu" "9100"
1701164133.835855 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1701164133.835870 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "mtu" "9100"
1701164133.835882 [0 lua] "PUBLISH" "PORT_TABLE_CHANNEL@0" "G"
1701164133.835909 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE" "128" "1"
1701164133.835921 [1 lua] "LRANGE" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.835928 [1 lua] "LTRIM" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.836143 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE_CHANNEL@1" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_QOS_NUMBER_OF_QUEUES\",\"8\"]" "Sget" "G"
1701164133.836156 [1 lua] "LPUSH" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_QOS_NUMBER_OF_QUEUES\",\"8\"]" "Sget"
1701164133.836238 [1 lua] "PUBLISH" "ASIC_STATE_CHANNEL@1" "G"
1701164133.836317 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE" "128" "0"
1701164133.836327 [1 lua] "LRANGE" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.836334 [1 lua] "LTRIM" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.836407 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE_CHANNEL@1" "SAI_STATUS_SUCCESS" "[\"SAI_PORT_ATTR_QOS_NUMBER_OF_QUEUES\",\"20\"]" "Sgetresponse" "G"
1701164133.836418 [1 lua] "LPUSH" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "SAI_STATUS_SUCCESS" "[\"SAI_PORT_ATTR_QOS_NUMBER_OF_QUEUES\",\"20\"]" "Sgetresponse"
1701164133.836433 [1 lua] "PUBLISH" "GETRESPONSE_CHANNEL@1" "G"
1701164133.836816 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE" "128" "1"
1701164133.836848 [1 lua] "LRANGE" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.836915 [1 lua] "LTRIM" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.837259 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE_CHANNEL@1" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_QOS_QUEUE_LIST\",\"20:oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0\"]" "Sget" "G"
1701164133.837278 [1 lua] "LPUSH" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_QOS_QUEUE_LIST\",\"20:oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0\"]" "Sget"
1701164133.837301 [1 lua] "PUBLISH" "ASIC_STATE_CHANNEL@1" "G"
1701164133.837376 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE" "128" "0"
1701164133.837447 [1 lua] "LRANGE" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.837455 [1 lua] "LTRIM" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.837609 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001e3" "NULL" "NULL"
1701164133.837631 [4 127.0.0.1:54830] "DEL" "PORTCHANNEL_MEMBER|PortChannel01|Ethernet0"
1701164133.838397 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001e4" "NULL" "NULL"
1701164133.838437 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001e5" "NULL" "NULL"
1701164133.838474 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001e6" "NULL" "NULL"
1701164133.838515 [4 unix:/var/run/redis/redis.sock] "HGETALL" "PORT|Ethernet0"
1701164133.838549 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001e7" "NULL" "NULL"
1701164133.838587 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001e8" "NULL" "NULL"
1701164133.838621 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001e9" "NULL" "NULL"
1701164133.838656 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001ea" "NULL" "NULL"
1701164133.838689 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001eb" "NULL" "NULL"
1701164133.838725 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001ec" "NULL" "NULL"
1701164133.838760 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001ed" "NULL" "NULL"
1701164133.838794 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001ee" "NULL" "NULL"
1701164133.838828 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001ef" "NULL" "NULL"
1701164133.838862 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001f0" "NULL" "NULL"
1701164133.838895 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001f1" "NULL" "NULL"
1701164133.838930 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001f2" "NULL" "NULL"
1701164133.838963 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001f3" "NULL" "NULL"
1701164133.838997 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001f4" "NULL" "NULL"
1701164133.839030 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001f5" "NULL" "NULL"
1701164133.839064 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000001f6" "NULL" "NULL"
1701164133.839106 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE_CHANNEL@1" "SAI_STATUS_SUCCESS" "[\"SAI_PORT_ATTR_QOS_QUEUE_LIST\",\"20:oid:0x150000000001e3,oid:0x150000000001e4,oid:0x150000000001e5,oid:0x150000000001e6,oid:0x150000000001e7,oid:0x150000000001e8,oid:0x150000000001e9,oid:0x150000000001ea,oid:0x150000000001eb,oid:0x150000000001ec,oid:0x150000000001ed,oid:0x150000000001ee,oid:0x150000000001ef,oid:0x150000000001f0,oid:0x150000000001f1,oid:0x150000000001f2,oid:0x150000000001f3,oid:0x150000000001f4,oid:0x150000000001f5,oid:0x150000000001f6\"]" "Sgetresponse" "G"
1701164133.839133 [1 lua] "LPUSH" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "SAI_STATUS_SUCCESS" "[\"SAI_PORT_ATTR_QOS_QUEUE_LIST\",\"20:oid:0x150000000001e3,oid:0x150000000001e4,oid:0x150000000001e5,oid:0x150000000001e6,oid:0x150000000001e7,oid:0x150000000001e8,oid:0x150000000001e9,oid:0x150000000001ea,oid:0x150000000001eb,oid:0x150000000001ec,oid:0x150000000001ed,oid:0x150000000001ee,oid:0x150000000001ef,oid:0x150000000001f0,oid:0x150000000001f1,oid:0x150000000001f2,oid:0x150000000001f3,oid:0x150000000001f4,oid:0x150000000001f5,oid:0x150000000001f6\"]" "Sgetresponse"
1701164133.839163 [1 lua] "PUBLISH" "GETRESPONSE_CHANNEL@1" "G"
1701164133.841455 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE" "128" "1"
1701164133.841509 [1 lua] "LRANGE" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.841523 [1 lua] "LTRIM" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.841622 [6 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE|Ethernet0"
1701164133.841705 [0 127.0.0.1:54838] "SELECT" "1"
1701164133.841908 [1 127.0.0.1:54838] "EXISTS" "ASIC_STATE:SAI_OBJECT_TYPE_PORT:oid:0x1000000000002"
1701164133.844460 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE_CHANNEL@1" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_QOS_NUMBER_OF_SCHEDULER_GROUPS\",\"0\"]" "Sget" "G"
1701164133.844483 [1 lua] "LPUSH" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_QOS_NUMBER_OF_SCHEDULER_GROUPS\",\"0\"]" "Sget"
1701164133.844591 [1 lua] "PUBLISH" "ASIC_STATE_CHANNEL@1" "G"
1701164133.844849 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE" "128" "0"
1701164133.844897 [1 lua] "LRANGE" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.844906 [1 lua] "LTRIM" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.845060 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE_CHANNEL@1" "SAI_STATUS_SUCCESS" "[\"SAI_PORT_ATTR_QOS_NUMBER_OF_SCHEDULER_GROUPS\",\"13\"]" "Sgetresponse" "G"
1701164133.845114 [1 lua] "LPUSH" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "SAI_STATUS_SUCCESS" "[\"SAI_PORT_ATTR_QOS_NUMBER_OF_SCHEDULER_GROUPS\",\"13\"]" "Sgetresponse"
1701164133.845132 [1 lua] "PUBLISH" "GETRESPONSE_CHANNEL@1" "G"
1701164133.845561 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE" "128" "1"
1701164133.845574 [1 lua] "LRANGE" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.845582 [1 lua] "LTRIM" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.847779 [6 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE|Ethernet0"
1701164133.847809 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE_CHANNEL@1" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_QOS_SCHEDULER_GROUP_LIST\",\"13:oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0\"]" "Sget" "G"
1701164133.847830 [1 lua] "LPUSH" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_QOS_SCHEDULER_GROUP_LIST\",\"13:oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0,oid:0x0\"]" "Sget"
1701164133.847875 [1 lua] "PUBLISH" "ASIC_STATE_CHANNEL@1" "G"
1701164133.847905 [4 unix:/var/run/redis/redis.sock] "KEYS" "PEER_SWITCH|*"
1701164133.848379 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE" "128" "0"
1701164133.848393 [1 lua] "LRANGE" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.848401 [1 lua] "LTRIM" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.848455 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.848518 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "6875900592cdd1621c6191fe038ec3b29775aa13" "3" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "G" "Ethernet0" "admin_status" "down"
1701164133.848531 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1701164133.848537 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "admin_status" "down"
1701164133.848575 [6 unix:/var/run/redis/redis.sock] "HSET" "PORT_TABLE|Ethernet0" "state" "ok" "netdev_oper_status" "down" "admin_status" "down" "mtu" "9100"
1701164133.848784 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x170000000001f7" "NULL" "NULL"
1701164133.849005 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.849062 [6 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE|Ethernet0"
1701164133.849152 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88ba6312b8de850b3506966425174d8899aadd93" "4" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "PORT_TABLE_DEL_SET" "G" "Ethernet0" "''" "''"
1701164133.849164 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1701164133.849169 [0 lua] "SADD" "PORT_TABLE_DEL_SET" "Ethernet0"
1701164133.849182 [0 lua] "DEL" "_PORT_TABLE:Ethernet0"
1701164133.849214 [6 unix:/var/run/redis/redis.sock] "HSET" "PORT_TABLE|Ethernet0" "state" "ok" "netdev_oper_status" "down" "admin_status" "down" "mtu" "9100"
1701164133.849306 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x170000000001f8" "NULL" "NULL"
1701164133.849507 [6 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE|Ethernet0"
1701164133.849563 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x170000000001f9" "NULL" "NULL"
1701164133.849685 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x170000000001fa" "NULL" "NULL"
1701164133.849776 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x170000000001fb" "NULL" "NULL"
1701164133.849871 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x170000000001fc" "NULL" "NULL"
1701164133.849967 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x170000000001fd" "NULL" "NULL"
1701164133.850062 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x170000000001fe" "NULL" "NULL"
1701164133.850158 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x170000000001ff" "NULL" "NULL"
1701164133.850260 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x17000000000200" "NULL" "NULL"
1701164133.850356 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x17000000000201" "NULL" "NULL"
1701164133.850486 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x17000000000202" "NULL" "NULL"
1701164133.850552 [1 unix:/var/run/redis/redis.sock] "HSET" "ASIC_STATE:SAI_OBJECT_TYPE_SCHEDULER_GROUP:oid:0x17000000000203" "NULL" "NULL"
1701164133.850687 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE_CHANNEL@1" "SAI_STATUS_SUCCESS" "[\"SAI_PORT_ATTR_QOS_SCHEDULER_GROUP_LIST\",\"13:oid:0x170000000001f7,oid:0x170000000001f8,oid:0x170000000001f9,oid:0x170000000001fa,oid:0x170000000001fb,oid:0x170000000001fc,oid:0x170000000001fd,oid:0x170000000001fe,oid:0x170000000001ff,oid:0x17000000000200,oid:0x17000000000201,oid:0x17000000000202,oid:0x17000000000203\"]" "Sgetresponse" "G"
1701164133.850808 [1 lua] "LPUSH" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "SAI_STATUS_SUCCESS" "[\"SAI_PORT_ATTR_QOS_SCHEDULER_GROUP_LIST\",\"13:oid:0x170000000001f7,oid:0x170000000001f8,oid:0x170000000001f9,oid:0x170000000001fa,oid:0x170000000001fb,oid:0x170000000001fc,oid:0x170000000001fd,oid:0x170000000001fe,oid:0x170000000001ff,oid:0x17000000000200,oid:0x17000000000201,oid:0x17000000000202,oid:0x17000000000203\"]" "Sgetresponse"
1701164133.850837 [1 lua] "PUBLISH" "GETRESPONSE_CHANNEL@1" "G"
1701164133.851050 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE" "128" "1"
1701164133.851062 [1 lua] "LRANGE" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.851070 [1 lua] "LTRIM" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.851437 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE_CHANNEL@1" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_QOS_MAXIMUM_HEADROOM_SIZE\",\"13\"]" "Sget" "G"
1701164133.851451 [1 lua] "LPUSH" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "SAI_OBJECT_TYPE_PORT:oid:0x10000000001cc" "[\"SAI_PORT_ATTR_QOS_MAXIMUM_HEADROOM_SIZE\",\"13\"]" "Sget"
1701164133.851519 [1 lua] "PUBLISH" "ASIC_STATE_CHANNEL@1" "G"
1701164133.851884 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE" "128" "0"
1701164133.851895 [1 lua] "LRANGE" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.851902 [1 lua] "LTRIM" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.852495 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE_CHANNEL@1" "SAI_STATUS_NOT_IMPLEMENTED" "[]" "Sgetresponse" "G"
1701164133.852507 [1 lua] "LPUSH" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "SAI_STATUS_NOT_IMPLEMENTED" "[]" "Sgetresponse"
1701164133.852575 [1 lua] "PUBLISH" "GETRESPONSE_CHANNEL@1" "G"
1701164133.852705 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88ba6312b8de850b3506966425174d8899aadd93" "4" "LAG_MEMBER_TABLE_CHANNEL@0" "LAG_MEMBER_TABLE_KEY_SET" "_LAG_MEMBER_TABLE:PortChannel01:Ethernet0" "LAG_MEMBER_TABLE_DEL_SET" "G" "PortChannel01:Ethernet0" "''" "''"
1701164133.852720 [0 lua] "SADD" "LAG_MEMBER_TABLE_KEY_SET" "PortChannel01:Ethernet0"
1701164133.852736 [0 lua] "SADD" "LAG_MEMBER_TABLE_DEL_SET" "PortChannel01:Ethernet0"
1701164133.852799 [0 lua] "DEL" "_LAG_MEMBER_TABLE:PortChannel01:Ethernet0"
1701164133.852805 [0 lua] "PUBLISH" "LAG_MEMBER_TABLE_CHANNEL@0" "G"
1701164133.852935 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE" "128" "1"
1701164133.852946 [1 lua] "LRANGE" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.852953 [1 lua] "LTRIM" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.853062 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "6875900592cdd1621c6191fe038ec3b29775aa13" "5" "LAG_TABLE_CHANNEL@0" "LAG_TABLE_KEY_SET" "_LAG_TABLE:PortChannel01" "_LAG_TABLE:PortChannel01" "_LAG_TABLE:PortChannel01" "G" "PortChannel01" "admin_status" "up" "oper_status" "down" "mtu" "9100"
1701164133.853078 [0 lua] "SADD" "LAG_TABLE_KEY_SET" "PortChannel01"
1701164133.853213 [0 lua] "HSET" "_LAG_TABLE:PortChannel01" "admin_status" "up"
1701164133.853305 [0 lua] "HSET" "_LAG_TABLE:PortChannel01" "oper_status" "down"
1701164133.853317 [0 lua] "HSET" "_LAG_TABLE:PortChannel01" "mtu" "9100"
1701164133.853449 [0 lua] "PUBLISH" "LAG_TABLE_CHANNEL@0" "G"
1701164133.853647 [6 unix:/var/run/redis/redis.sock] "HSET" "BUFFER_MAX_PARAM_TABLE|Ethernet36" "max_priority_groups" "8" "max_queues" "20"
1701164133.853858 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.854009 [6 unix:/var/run/redis/redis.sock] "HSET" "PORT_TABLE|Ethernet0" "state" "ok" "netdev_oper_status" "down" "admin_status" "down" "mtu" "9100"
1701164133.854263 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE_CHANNEL@1" "oid:0x21000000000000" "[\"OBJECT_TYPE\",\"SAI_OBJECT_TYPE_HOSTIF\",\"ATTR_ID\",\"SAI_HOSTIF_ATTR_QUEUE\"]" "Sattribute_capability_query" "G"
1701164133.854278 [1 lua] "LPUSH" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "oid:0x21000000000000" "[\"OBJECT_TYPE\",\"SAI_OBJECT_TYPE_HOSTIF\",\"ATTR_ID\",\"SAI_HOSTIF_ATTR_QUEUE\"]" "Sattribute_capability_query"
1701164133.854397 [1 lua] "PUBLISH" "ASIC_STATE_CHANNEL@1" "G"
1701164133.854512 [6 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE|Ethernet0"
1701164133.854622 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.854723 [6 unix:/var/run/redis/redis.sock] "HSET" "PORT_TABLE|Ethernet0" "state" "ok" "netdev_oper_status" "down" "admin_status" "down" "mtu" "9100"
1701164133.854854 [6 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE|Ethernet0"
1701164133.855113 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.855203 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE" "128" "0"
1701164133.855215 [1 lua] "LRANGE" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.855221 [1 lua] "LTRIM" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.855818 [6 unix:/var/run/redis/redis.sock] "HSET" "PORT_TABLE|Ethernet0" "state" "ok" "netdev_oper_status" "down" "admin_status" "down" "mtu" "1500"
1701164133.855950 [6 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE|Ethernet0"
1701164133.856273 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE_CHANNEL@1" "SAI_STATUS_SUCCESS" "[\"CREATE_IMPLEMENTED\",\"true\",\"SET_IMPLEMENTED\",\"true\",\"GET_IMPLEMENTED\",\"true\"]" "Sattribute_capability_response" "G"
1701164133.856339 [1 lua] "LPUSH" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "SAI_STATUS_SUCCESS" "[\"CREATE_IMPLEMENTED\",\"true\",\"SET_IMPLEMENTED\",\"true\",\"GET_IMPLEMENTED\",\"true\"]" "Sattribute_capability_response"
1701164133.856359 [1 lua] "PUBLISH" "GETRESPONSE_CHANNEL@1" "G"
1701164133.856465 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "4437a4e435b1d0b82a9900c9b796a0aaeb8463ea" "2" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "GETRESPONSE" "128" "1"
1701164133.856477 [1 lua] "LRANGE" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "-384" "-1"
1701164133.856484 [1 lua] "LTRIM" "GETRESPONSE_KEY_VALUE_OP_QUEUE" "0" "-385"
1701164133.856728 [1 unix:/var/run/redis/redis.sock] "INCR" "VIDCOUNTER"
1701164133.857308 [0 unix:/var/run/redis/redis.sock] "HGETALL" "PORT_TABLE:Ethernet0"
1701164133.861473 [6 unix:/var/run/redis/redis.sock] "HSET" "PORT_TABLE|Ethernet0" "state" "ok" "netdev_oper_status" "down" "admin_status" "down" "mtu" "9100"
1701164133.861517 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88ba6312b8de850b3506966425174d8899aadd93" "4" "ROUTE_TABLE_CHANNEL@0" "ROUTE_TABLE_KEY_SET" "_ROUTE_TABLE:fe80::/64" "ROUTE_TABLE_DEL_SET" "G" "fe80::/64" "''" "''"
1701164133.861534 [0 lua] "SADD" "ROUTE_TABLE_KEY_SET" "fe80::/64"
1701164133.861550 [0 lua] "SADD" "ROUTE_TABLE_DEL_SET" "fe80::/64"
1701164133.861562 [0 lua] "DEL" "_ROUTE_TABLE:fe80::/64"
1701164133.861566 [0 lua] "PUBLISH" "ROUTE_TABLE_CHANNEL@0" "G"
1701164133.861574 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88ba6312b8de850b3506966425174d8899aadd93" "4" "ROUTE_TABLE_CHANNEL@0" "ROUTE_TABLE_KEY_SET" "_ROUTE_TABLE:0.0.0.0/0" "ROUTE_TABLE_DEL_SET" "G" "0.0.0.0/0" "''" "''"
1701164133.861584 [0 lua] "SADD" "ROUTE_TABLE_KEY_SET" "0.0.0.0/0"
1701164133.861596 [0 lua] "SADD" "ROUTE_TABLE_DEL_SET" "0.0.0.0/0"
1701164133.861606 [0 lua] "DEL" "_ROUTE_TABLE:0.0.0.0/0"
1701164133.861609 [0 lua] "PUBLISH" "ROUTE_TABLE_CHANNEL@0" "G"
1701164133.861621 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88ba6312b8de850b3506966425174d8899aadd93" "4" "ROUTE_TABLE_CHANNEL@0" "ROUTE_TABLE_KEY_SET" "_ROUTE_TABLE:::/0" "ROUTE_TABLE_DEL_SET" "G" "::/0" "''" "''"
1701164133.861630 [0 lua] "SADD" "ROUTE_TABLE_KEY_SET" "::/0"
1701164133.861642 [0 lua] "SADD" "ROUTE_TABLE_DEL_SET" "::/0"
1701164133.861652 [0 lua] "DEL" "_ROUTE_TABLE:::/0"
1701164133.861656 [0 lua] "PUBLISH" "ROUTE_TABLE_CHANNEL@0" "G"
1701164133.861661 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88ba6312b8de850b3506966425174d8899aadd93" "4" "ROUTE_TABLE_CHANNEL@0" "ROUTE_TABLE_KEY_SET" "_ROUTE_TABLE:fd00::/80" "ROUTE_TABLE_DEL_SET" "G" "fd00::/80" "''" "''"
1701164133.861670 [0 lua] "SADD" "ROUTE_TABLE_KEY_SET" "fd00::/80"
1701164133.861681 [0 lua] "SADD" "ROUTE_TABLE_DEL_SET" "fd00::/80"
1701164133.861692 [0 lua] "DEL" "_ROUTE_TABLE:fd00::/80"
1701164133.861695 [0 lua] "PUBLISH" "ROUTE_TABLE_CHANNEL@0" "G"
1701164133.861726 [1 unix:/var/run/redis/redis.sock] "EVALSHA" "d171e04fd79e95ca2287f3b067c46ae76a82208b" "2" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "ASIC_STATE_CHANNEL@1" "SAI_OBJECT_TYPE_HOSTIF:oid:0xd000000000ba6" "[\"SAI_HOSTIF_ATTR_TYPE\",\"SAI_HOSTIF_TYPE_NETDEV\",\"SAI_HOSTIF_ATTR_OBJ_ID\",\"oid:0x10000000001cc\",\"SAI_HOSTIF_ATTR_NAME\",\"Ethernet36\",\"SAI_HOSTIF_ATTR_QUEUE\",\"7\"]" "Screate" "G"
1701164133.861748 [1 lua] "LPUSH" "ASIC_STATE_KEY_VALUE_OP_QUEUE" "SAI_OBJECT_TYPE_HOSTIF:oid:0xd000000000ba6" "[\"SAI_HOSTIF_ATTR_TYPE\",\"SAI_HOSTIF_TYPE_NETDEV\",\"SAI_HOSTIF_ATTR_OBJ_ID\",\"oid:0x10000000001cc\",\"SAI_HOSTIF_ATTR_NAME\",\"Ethernet36\",\"SAI_HOSTIF_ATTR_QUEUE\",\"7\"]" "Screate"
1701164133.861771 [1 lua] "PUBLISH" "ASIC_STATE_CHANNEL@1" "G"
1701164133.861805 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "6875900592cdd1621c6191fe038ec3b29775aa13" "4" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "_PORT_TABLE:Ethernet0" "G" "Ethernet0" "admin_status" "down" "mtu" "9100"
1701164133.861820 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1701164133.861826 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "admin_status" "down"
1701164133.861839 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "mtu" "9100"
...
...

@sandyw777
Copy link

redis-cli monitor:

1701164134.544183 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88270a7c5c90583e56425aca8af8a4b8c39fe757" "3" "PORT_TABLE_KEY_SET" "PORT_TABLE:" "PORT_TABLE_DEL_SET" "1024" "_"
1701164134.544194 [0 lua] "SPOP" "PORT_TABLE_KEY_SET" "1024"
1701164134.544220 [0 lua] "SREM" "PORT_TABLE_DEL_SET" "Ethernet0"
1701164134.544240 [0 lua] "DEL" "PORT_TABLE:Ethernet0"
1701164134.544253 [0 lua] "HGETALL" "_PORT_TABLE:Ethernet0"
1701164134.544260 [0 lua] "HSET" "PORT_TABLE:Ethernet0" "admin_status" "down"
1701164134.544274 [0 lua] "HSET" "PORT_TABLE:Ethernet0" "mtu" "9100"
1701164134.544286 [0 lua] "DEL" "_PORT_TABLE:Ethernet0"

We found "DEL" is executed before "HSET", and causes Ethernet0 to be created in PORT_TABLE again, and dynamic port breakout failure.

The commit about "DEL" is executed before "HSET" at "consumer_state_table_pops.lua" is added at sonic-net/sonic-swss-common#240 to fix sonic-net/sonic-swss-common#236.
In issue#236, the produceStateTable operation sequence is "delete" and "set".

But in the dynamic port breakout case, the produceStateTable operation sequence is "set" and "delete". The #240 fix will cause Ethernet0 to be created in PORT_TABLE again.

@jipanyang @qiluo-msft @dgsudharsan @praveen-li @zhenggen-xu Could you help to check if sonic-net/sonic-swss-common#240 causes this issue?
Thank you!

This was referenced Dec 28, 2023
Closed
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@praveen-li praveen-li

Labels
Help Wanted 🆘 Issue for 202111 LINKEDIN Port Breakout 🔱 Triaged this issue has been triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@zhangyanzhao @alexrallen @puffc @dgsudharsan @zhenggen-xu @praveen-li @liat-grozovik @sandyw777