Collect ICMP packet loss information #14

zjswhhh · 2022-01-03T19:22:40Z

Description of PR

Summary:
Fixes # (issue)

This PR is to collect ICMP packet loss information.

sign-off: Jing Zhang zhangjing@microsoft.com

Type of change

Bug fix
New feature
Doc/Design
Unit test

Approach

What is the motivation for this PR?

When ICMP heartbeat loss happens, we want to know how long it lasts. We also want to collect the packet loss ratio information.

How did you do it?

Post link prober state change events and time stamps to state db metrics, when get in or out link prober unknown state;
Post link prober pck loss ratio every 300 ms to state db.

How did you verify/test it?

Tested on dual testbed.

Table entries created as expected:

$ redis-cli -n 6 
127.0.0.1:6379[6]> KEYS *LINK_PROBE*
1) "LINK_PROBE_STATS|Ethernet48"
2) "LINK_PROBE_STATS|Ethernet84"
      ... ...

Before ICMP responder was turned on:

~$ redis-cli -n 6 HGETALL "LINK_PROBE_STATS|Ethernet44" 
1) "pck_loss_count"
2) "24"
3) "pck_expected_count"
4) "24"
5) "link_prober_unknown_start"
6) "2022-Jan-26 02:59:58.138248"

After running a link_failure test case, link_prober_unknown_start and link_prober_unknown_end are updated:

~$ redis-cli -n 6 HGETALL "LINK_PROBE_STATS|Ethernet44"
1) "pck_loss_count"
2) "612"
3) "pck_expected_count"
4) "840"
5) "link_prober_unknown_start"
6) "2022-Jan-26 03:13:05.366900"
7) "link_prober_unknown_end"
8) "2022-Jan-26 03:17:35.446580"

After resetting the packet loss counts:

~$ redis-cli -n 4 HSET "MUX_CABLE|Ethernet44" pck_loss_data_reset reset 
(integer) 0

~$ redis-cli -n 6 HGETALL "LINK_PROBE_STATS|Ethernet44"
1) "pck_loss_count"
2) "0"
3) "pck_expected_count"
4) "0"
5) "link_prober_unknown_start"
6) "2022-Jan-26 03:13:05.366900"
7) "link_prober_unknown_end"
8) "2022-Jan-26 03:17:35.446580"

~$ redis-cli -n 6 HGETALL "LINK_PROBE_STATS|Ethernet44"
1) "pck_loss_count"
2) "0"
3) "pck_expected_count"
4) "6"
5) "link_prober_unknown_start"
6) "2022-Jan-26 03:13:05.366900"
7) "link_prober_unknown_end"
8) "2022-Jan-26 03:17:35.446580"

Any platform specific information?

Documentation

src/link_prober/LinkProber.cpp

lolyu

Do we need a method to reset the packet loss ratio other than restarting the linkmgrd?

src/link_prober/LinkProber.cpp

zjswhhh · 2022-01-06T19:12:29Z

Do we need a method to reset the packet loss ratio other than restarting the linkmgrd?

Good question. I think we do? Once the ping is back working, we can simply reset so that the ratio number is more meaningful.

Signed-off-by: Longxiang Lyu <lolv@microsoft.com>

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

Signed-off-by: Longxiang Lyu <lolv@microsoft.com>

Summary: Fixes # (issue) This PR is to make linkmgrd subscribe events from ROUTE_TABLE in State DB, and react accordingly: - If any of the two default route state appears to be 'na', linkmgrd should switch to standby. - If both are 'ok', there will be a mux state probing and what happens next depends on linkmgrd state machine and the probing response. Type of change: New feature Motivation for this PR: To make linkmgrd subscribe state DB route event, and handle the switchovers. Documentation: Related PR: sonic-net/sonic-swss#2009

…er is off

…c-linkmgrd into collectIcmpLossInfo

src/link_prober/LinkProber.cpp

… ICMP packet loss data (#2046) Stemming from sonic-net/sonic-linkmgrd#14 sign-off: Jing Zhang zhangjing@microsoft.com #### What I did Added support to retrieve and reset ICMP packet loss data in state db for muxcable. #### How I did it Changes made in show/muxcable.py and config/muxcable.py #### How to verify it - Added unit tests. - Tested the command lines on testbeds. - Ran dualtor_io/test_link_failure.py. #### Previous command output (if the output of a command-line utility has changed) #### New command output (if the output of a command-line utility has changed) ``` show muxcable pckloss <port_name>``` ``` admin@str2-7050cx3-acs-07:~$ show muxcable packetloss Ethernet96 PORT COUNT VALUE ---------- ------------------ ------- Ethernet96 pck_loss_count 10439 Ethernet96 pck_expected_count 11406 PORT EVENT TIME ---------- ------------------------- --------------------------- Ethernet96 link_prober_unknown_start 2022-Jan-27 19:47:17.819699 Ethernet96 link_prober_unknown_end 2022-Jan-27 22:28:36.736928 ``` ```config muxcable pckloss reset <port_name>``` ``` admin@str2-7050cx3-acs-07:~$ sudo config muxcable packetloss reset Ethernet96 admin@str2-7050cx3-acs-07:~$ show muxcable packetloss Ethernet96 PORT COUNT VALUE ---------- ------------------ ------- Ethernet96 pck_expected_count 0 Ethernet96 pck_loss_count 0 PORT EVENT TIME ---------- ------------------------- --------------------------- Ethernet96 link_prober_unknown_start 2022-Jan-27 19:47:17.819699 Ethernet96 link_prober_unknown_end 2022-Jan-27 22:28:36.736928 ``` ```config muxcable pckloss reset all``` ``` admin@str2-7050cx3-acs-07:~$ sudo config muxcable packetloss reset all admin@str2-7050cx3-acs-07:~$ show muxcable packetloss Ethernet68 PORT COUNT VALUE ---------- ------------------ ------- Ethernet68 pck_loss_count 0 Ethernet68 pck_expected_count 3 PORT EVENT TIME ---------- ------------------------- --------------------------- Ethernet68 link_prober_unknown_start 2022-Jan-27 19:47:17.702760 Ethernet68 link_prober_unknown_end 2022-Jan-27 22:28:36.756113 ```

… ICMP packet loss data (sonic-net#2046) Stemming from sonic-net/sonic-linkmgrd#14 sign-off: Jing Zhang zhangjing@microsoft.com Added support to retrieve and reset ICMP packet loss data in state db for muxcable. Changes made in show/muxcable.py and config/muxcable.py - Added unit tests. - Tested the command lines on testbeds. - Ran dualtor_io/test_link_failure.py. ``` show muxcable pckloss <port_name>``` ``` admin@str2-7050cx3-acs-07:~$ show muxcable packetloss Ethernet96 PORT COUNT VALUE ---------- ------------------ ------- Ethernet96 pck_loss_count 10439 Ethernet96 pck_expected_count 11406 PORT EVENT TIME ---------- ------------------------- --------------------------- Ethernet96 link_prober_unknown_start 2022-Jan-27 19:47:17.819699 Ethernet96 link_prober_unknown_end 2022-Jan-27 22:28:36.736928 ``` ```config muxcable pckloss reset <port_name>``` ``` admin@str2-7050cx3-acs-07:~$ sudo config muxcable packetloss reset Ethernet96 admin@str2-7050cx3-acs-07:~$ show muxcable packetloss Ethernet96 PORT COUNT VALUE ---------- ------------------ ------- Ethernet96 pck_expected_count 0 Ethernet96 pck_loss_count 0 PORT EVENT TIME ---------- ------------------------- --------------------------- Ethernet96 link_prober_unknown_start 2022-Jan-27 19:47:17.819699 Ethernet96 link_prober_unknown_end 2022-Jan-27 22:28:36.736928 ``` ```config muxcable pckloss reset all``` ``` admin@str2-7050cx3-acs-07:~$ sudo config muxcable packetloss reset all admin@str2-7050cx3-acs-07:~$ show muxcable packetloss Ethernet68 PORT COUNT VALUE ---------- ------------------ ------- Ethernet68 pck_loss_count 0 Ethernet68 pck_expected_count 3 PORT EVENT TIME ---------- ------------------------- --------------------------- Ethernet68 link_prober_unknown_start 2022-Jan-27 19:47:17.702760 Ethernet68 link_prober_unknown_end 2022-Jan-27 22:28:36.756113 ```

… ICMP packet loss data (#2046) (#2094) Stemming from sonic-net/sonic-linkmgrd#14 sign-off: Jing Zhang zhangjing@microsoft.com Added support to retrieve and reset ICMP packet loss data in state db for muxcable. Changes made in show/muxcable.py and config/muxcable.py - Added unit tests. - Tested the command lines on testbeds. - Ran dualtor_io/test_link_failure.py. ``` show muxcable pckloss <port_name>``` ``` admin@str2-7050cx3-acs-07:~$ show muxcable packetloss Ethernet96 PORT COUNT VALUE ---------- ------------------ ------- Ethernet96 pck_loss_count 10439 Ethernet96 pck_expected_count 11406 PORT EVENT TIME ---------- ------------------------- --------------------------- Ethernet96 link_prober_unknown_start 2022-Jan-27 19:47:17.819699 Ethernet96 link_prober_unknown_end 2022-Jan-27 22:28:36.736928 ``` ```config muxcable pckloss reset <port_name>``` ``` admin@str2-7050cx3-acs-07:~$ sudo config muxcable packetloss reset Ethernet96 admin@str2-7050cx3-acs-07:~$ show muxcable packetloss Ethernet96 PORT COUNT VALUE ---------- ------------------ ------- Ethernet96 pck_expected_count 0 Ethernet96 pck_loss_count 0 PORT EVENT TIME ---------- ------------------------- --------------------------- Ethernet96 link_prober_unknown_start 2022-Jan-27 19:47:17.819699 Ethernet96 link_prober_unknown_end 2022-Jan-27 22:28:36.736928 ``` ```config muxcable pckloss reset all``` ``` admin@str2-7050cx3-acs-07:~$ sudo config muxcable packetloss reset all admin@str2-7050cx3-acs-07:~$ show muxcable packetloss Ethernet68 PORT COUNT VALUE ---------- ------------------ ------- Ethernet68 pck_loss_count 0 Ethernet68 pck_expected_count 3 PORT EVENT TIME ---------- ------------------------- --------------------------- Ethernet68 link_prober_unknown_start 2022-Jan-27 19:47:17.702760 Ethernet68 link_prober_unknown_end 2022-Jan-27 22:28:36.756113 ```

… ICMP packet loss data (#2046) Stemming from sonic-net/sonic-linkmgrd#14 sign-off: Jing Zhang zhangjing@microsoft.com #### What I did Added support to retrieve and reset ICMP packet loss data in state db for muxcable. #### How I did it Changes made in show/muxcable.py and config/muxcable.py #### How to verify it - Added unit tests. - Tested the command lines on testbeds. - Ran dualtor_io/test_link_failure.py. #### Previous command output (if the output of a command-line utility has changed) #### New command output (if the output of a command-line utility has changed) ``` show muxcable pckloss <port_name>``` ``` admin@str2-7050cx3-acs-07:~$ show muxcable packetloss Ethernet96 PORT COUNT VALUE ---------- ------------------ ------- Ethernet96 pck_loss_count 10439 Ethernet96 pck_expected_count 11406 PORT EVENT TIME ---------- ------------------------- --------------------------- Ethernet96 link_prober_unknown_start 2022-Jan-27 19:47:17.819699 Ethernet96 link_prober_unknown_end 2022-Jan-27 22:28:36.736928 ``` ```config muxcable pckloss reset <port_name>``` ``` admin@str2-7050cx3-acs-07:~$ sudo config muxcable packetloss reset Ethernet96 admin@str2-7050cx3-acs-07:~$ show muxcable packetloss Ethernet96 PORT COUNT VALUE ---------- ------------------ ------- Ethernet96 pck_expected_count 0 Ethernet96 pck_loss_count 0 PORT EVENT TIME ---------- ------------------------- --------------------------- Ethernet96 link_prober_unknown_start 2022-Jan-27 19:47:17.819699 Ethernet96 link_prober_unknown_end 2022-Jan-27 22:28:36.736928 ``` ```config muxcable pckloss reset all``` ``` admin@str2-7050cx3-acs-07:~$ sudo config muxcable packetloss reset all admin@str2-7050cx3-acs-07:~$ show muxcable packetloss Ethernet68 PORT COUNT VALUE ---------- ------------------ ------- Ethernet68 pck_loss_count 0 Ethernet68 pck_expected_count 3 PORT EVENT TIME ---------- ------------------------- --------------------------- Ethernet68 link_prober_unknown_start 2022-Jan-27 19:47:17.702760 Ethernet68 link_prober_unknown_end 2022-Jan-27 22:28:36.756113 ```

zjswhhh added 2 commits January 3, 2022 03:05

Count ICMP Unknown events

919a0aa

Calculate pck loss rate.

c974691

yxieca reviewed Jan 4, 2022

View reviewed changes

src/link_prober/LinkProber.cpp Outdated Show resolved Hide resolved

lolyu reviewed Jan 6, 2022

View reviewed changes

src/link_prober/LinkProber.cpp Outdated Show resolved Hide resolved

zjswhhh marked this pull request as draft January 6, 2022 20:49

yxieca force-pushed the master branch from 0ba431f to 12b9951 Compare January 14, 2022 17:33

lolyu and others added 8 commits January 24, 2022 08:44

Improve PR template (sonic-net#16)

1fd40a8

Signed-off-by: Longxiang Lyu <lolv@microsoft.com>

[linkmgrd] update README, set coding style guidance (sonic-net#15)

442cac7

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

Add missing intermediate states (sonic-net#16)

c9a3ccc

Signed-off-by: Longxiang Lyu <lolv@microsoft.com>

Add TLV support to ICMP payload (sonic-net#11)

aa03765

Signed-off-by: Longxiang Lyu <lolv@microsoft.com>

Add DbInterface related handlers

e2c4cd1

Add handlers for pck loss updates

d12b7fd

update tests

f3b5966

zjswhhh force-pushed the collectIcmpLossInfo branch from 4a2405c to f3b5966 Compare January 24, 2022 08:54

fix conflicts

0c25d49

zjswhhh marked this pull request as ready for review January 24, 2022 09:07

zjswhhh added 2 commits January 24, 2022 09:33

fix table name typo

b1b5926

add unit test

bd885ee

zjswhhh force-pushed the collectIcmpLossInfo branch from 2121f60 to bd885ee Compare January 24, 2022 09:49

zjswhhh added 6 commits January 24, 2022 17:01

add unit test

815839f

add flag to hanlde edge case: link probe stays wait when icmp_respond…

4416b0b

…er is off

fix conflicts

69a0daf

add unit test

1dbc400

add more unit tests

913cd1f

Merge branch 'collectIcmpLossInfo' of https://github.com/zjswhhh/soni…

9e913fd

…c-linkmgrd into collectIcmpLossInfo

yxieca reviewed Jan 24, 2022

View reviewed changes

src/link_prober/LinkProber.cpp Outdated Show resolved Hide resolved

Post packet loss counts instead of ratio

22b205f

zjswhhh added 2 commits January 25, 2022 19:22

Update log levels

17c989b

Add handlers for packet loss count resetting

14aea6a

zjswhhh force-pushed the collectIcmpLossInfo branch from 1559298 to 14aea6a Compare January 26, 2022 03:28

Add unit tests for resetting

0733034

zjswhhh requested review from yxieca and lolyu January 26, 2022 19:40

yxieca approved these changes Jan 27, 2022

View reviewed changes

zjswhhh mentioned this pull request Jan 27, 2022

[show][config] add muxcable command line support for retrieve / reset ICMP packet loss data sonic-net/sonic-utilities#2046

Merged

lolyu approved these changes Jan 28, 2022

View reviewed changes

zjswhhh merged commit bcd74b4 into sonic-net:master Feb 1, 2022

zjswhhh deleted the collectIcmpLossInfo branch February 1, 2022 17:52

zjswhhh added the Included in 202012 Branch label Mar 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Collect ICMP packet loss information #14

Collect ICMP packet loss information #14

zjswhhh commented Jan 3, 2022 •

edited

Loading

lolyu left a comment

zjswhhh commented Jan 6, 2022

Collect ICMP packet loss information #14

Collect ICMP packet loss information #14

Conversation

zjswhhh commented Jan 3, 2022 • edited Loading

Description of PR

Type of change

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Documentation

lolyu left a comment

Choose a reason for hiding this comment

zjswhhh commented Jan 6, 2022

zjswhhh commented Jan 3, 2022 •

edited

Loading