[ycabled] fix exception-handling logic for ycabled #306

vdahiya12 · 2022-10-19T23:14:17Z

Signed-off-by: vaibhav-dahiya vdahiya@microsoft.com
This PR is changing how exception handling is handled in python threading.
In python3.7 if child thread gets an exception, there is no way of knowing an exception occured, unless a join is called. The problem is resolved in this PR
Also task classes are structured this way

class YcableInfoUpdateTask(threading.Thread):

    def __init__(self):
        threading.Thread.__init__(self)
        self.exc = None
        
   def run(self):
        if self.task_stopping_event.is_set():
            return

        self.task_thread = threading.Thread(target=self.task_worker, args=(y_cable_presence,))
        self.task_thread.start()
        try:
            self.task_worker(self.y_cable_presence)
        except Exception as e:
            helper_logger.log_error("Exception occured at child thread YcableInfoUpdateTask due to {} {}".format(repr(e), traceback.format_exc()))

            self.exc = e

    def join(self):
        threading.Thread.join(self)

        if self.exc:
            raise self.exc

This allows helping catching the exception thrown by a child thread get caught along with traceback.
Main thread has a while loop, which monitors each thread, if not running will kill the whole process and supervisord will bring back the ycabled up.

Description

Motivation and Context

How Has This Been Tested?

Unit-tests and deploying changes on testbed

Additional Information (Optional)

sonic-mgmt tests which pass with this change

/var/src/sonic-mgmt-int/tests/logs/dualtor/test_ipinip.py::test_decap_active_tor
/var/src/sonic-mgmt-int/tests/logs/dualtor/test_ipinip.py::test_decap_standby_tor
/var/src/sonic-mgmt-int/tests/logs/dualtor/test_toggle_mux
/var/src/sonic-mgmt-int/dualtor_io/test_link_failure.py::test_standby_tor_downlink_down_downstream_active
/var/src/sonic-mgmt-int/tests/logs/platform_tests/api/test_chassis.py
In addition ran this script for testing the change on some Gemini devices
as well as switchover regression
time cost distribution

key: time cost

value: counts

i.e. 10:272 means 272 toggles took 0~10ms

{10: 272, 20: 217, 30: 3, 50: 3, 100: 3, 101: 3}

average (ms)

11.204035928143714

sudo cp <fw> /usr/share/sonic/firmware/.

declare -a PORTS='Ethernet0 Ethernet4 Ethernet8 Ethernet12 Ethernet16 Ethernet20 Ethernet40 Ethernet44 Ethernet48 Ethernet52 Ethernet56 Ethernet60 Ethernet64 Ethernet68 Ethernet72 Ethernet76 Ethernet80 Ethernet84 Ethernet104 Ethernet108 Ethernet112 Ethernet116 Ethernet120 Ethernet124'

declare -a PORTS1='Ethernet4'

for port in ${PORTS[@]}
do
  echo -e "\n\nname: $port"
  sudo show muxcable firmware version $port
  sleep 1
  sudo config muxcable firmware download <fw> $port
  sleep 10
  sudo show muxcable firmware version $port

  sudo config muxcable firmware activate $port
  sleep 10
  sudo show muxcable firmware version $port

  show mux status
  sleep 1
  sudo config mux mode auto $port
  sudo config mux mode active $port
  sleep 1
  show mux metrics $port
  sudo config mux mode auto $port
  sudo config mux mode standby $port
  sleep 1
  show mux metrics $port
  sleep 1
  show mux cableinfo $port
  sleep 1
  show mux alivecablestatus $port
  sleep 1
  show mux eyeinfo $port NIC
  show mux eyeinfo $port TORA
  show mux eyeinfo $port TORB
  sleep 1
  show mux pcsstatistics $port NIC
  sleep 1
  show mux pcsstatistics $port TORA
  sleep 1
  show mux pcsstatistics $port TORB
  sleep 1
  show mux fecstatistics $port NIC
  sleep 1
  show mux fecstatistics $port TORA
  sleep 1
  show mux fecstatistics $port TORB
  sleep 1
  show mux get-fec-anlt-speed $port
  sleep 1
  show mux hwmode mux $port
  sleep 1
  show mux hwmode switch $port
  sleep 1
  redis-cli -n 6 hgetall "HW_MUX_CABLE_TABLE|$port"
  sleep 1
  redis-cli -n 6 hgetall "MUX_CABLE_INFO|$port"
  sleep 1
  redis-cli -n 6 hgetall "MUX_CABLE_STATIC_INFO|$port"
  sleep 1
  redis-cli -n 6 hset "TRANSCEIVER_STATUS|$port" "status" "0"
  sleep 1
  sudo config mux mode auto $port
  sudo config mux mode active $port
  sleep 1
  show mux metrics $port
  sudo config mux mode auto $port
  sudo config mux mode standby $port
  sleep 1
  show mux metrics $port
  sleep 1
  show mux cableinfo $port
  sleep 1
  show mux alivecablestatus $port
  sleep 1
  show mux eyeinfo $port NIC
  show mux eyeinfo $port TORA
  show mux eyeinfo $port TORB
  sleep 1
  show mux pcsstatistics $port NIC
  sleep 1
  show mux pcsstatistics $port TORA
  sleep 1
  show mux fecstatistics $port NIC
  sleep 1
  show mux fecstatistics $port TORA
  sleep 1
  show mux fecstatistics $port TORB
  sleep 1
  show mux get-fec-anlt-speed $port
  sleep 1
  show mux hwmode mux $port
  sleep 1
  show mux hwmode switch $port
  sleep 1
  redis-cli -n 6 hgetall "HW_MUX_CABLE_TABLE|$port"
  sleep 1
  redis-cli -n 6 hgetall "MUX_CABLE_INFO|$port"
  sleep 1
  redis-cli -n 6 hgetall "MUX_CABLE_STATIC_INFO|$port"
  sleep 1
  redis-cli -n 6 hset "TRANSCEIVER_STATUS|$port" "status" "1"
  sleep 1
  show mux status
  sleep 1
  sudo config mux mode auto $port
  sudo config mux mode active $port
  sleep 1
  show mux metrics $port
  sudo config mux mode auto $port
  sudo config mux mode standby $port
  sleep 1
  show mux metrics $port
  sleep 1
  show mux cableinfo $port
  sleep 1
  show mux alivecablestatus $port
  sleep 1
  show mux eyeinfo $port NIC
  show mux eyeinfo $port TORA
  show mux eyeinfo $port TORB
  sleep 1
  show mux pcsstatistics $port NIC
  sleep 1
  show mux pcsstatistics $port TORA
  sleep 1
  show mux pcsstatistics $port TORB
  sleep 1
  show mux fecstatistics $port NIC
  sleep 1
  show mux fecstatistics $port TORA
  sleep 1
  show mux fecstatistics $port TORB

done

No exceptions observed, ycabled seems healthy with true thread count

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-10-19T23:19:59Z

This pull request introduces 2 alerts when merging e3cd1a0 into 23b6970 - view on LGTM.com

new alerts:

1 for Too few arguments in formatting call
1 for Syntax error

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-10-19T23:34:24Z

This pull request introduces 1 alert when merging 68e9412 into 23b6970 - view on LGTM.com

new alerts:

1 for Too few arguments in formatting call

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-10-20T22:54:57Z

This pull request introduces 2 alerts when merging cc96144 into 4ea12cf - view on LGTM.com

new alerts:

2 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-10-21T01:13:39Z

This pull request introduces 3 alerts when merging 81cf7ca into 4ea12cf - view on LGTM.com

new alerts:

3 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-10-21T01:58:07Z

This pull request introduces 4 alerts when merging b898ffa into 4ea12cf - view on LGTM.com

new alerts:

4 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-10-21T02:22:22Z

This pull request introduces 4 alerts when merging 2807278 into 4ea12cf - view on LGTM.com

new alerts:

4 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-10-21T02:58:23Z

This pull request introduces 4 alerts when merging 3cfc033 into 4ea12cf - view on LGTM.com

new alerts:

4 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-10-21T18:20:09Z

This pull request introduces 4 alerts when merging 2c14791 into 4ea12cf - view on LGTM.com

new alerts:

4 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-11-01T23:36:52Z

This pull request introduces 4 alerts when merging a71b11b into 9ae551f - view on LGTM.com

new alerts:

4 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-11-01T23:53:54Z

This pull request introduces 4 alerts when merging e92c74f into 9ae551f - view on LGTM.com

new alerts:

4 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-11-02T22:54:38Z

This pull request introduces 4 alerts when merging 9745e6a into 9ae551f - view on LGTM.com

new alerts:

4 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

prgeor · 2022-11-07T18:55:49Z

@vdahiya12 there are 4 new LGTM alerts. could you fix?

prgeor · 2022-11-07T20:45:03Z

sonic-ycabled/tests/test_y_cable_helper.py

        logical_port_name = "Ethernet0"
-        mux_tbl = Table("STATE_DB", "Y_CABLE_INFO_TABLE")
-        rc = post_port_mux_info_to_db(logical_port_name, mux_tbl,'active-standby')


can we add a test to crash at known location and see if we print backtrace ?

UT is added, fixed

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-11-07T22:38:01Z

This pull request introduces 4 alerts when merging 1ac2a2b into 905874d - view on LGTM.com

new alerts:

4 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-11-08T22:50:36Z

This pull request introduces 4 alerts when merging 56c48ef into 905874d - view on LGTM.com

new alerts:

4 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-11-08T23:24:54Z

This pull request introduces 4 alerts when merging 2e19b16 into 905874d - view on LGTM.com

new alerts:

4 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-11-08T23:43:21Z

This pull request introduces 4 alerts when merging e23540c into 905874d - view on LGTM.com

new alerts:

4 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-11-09T00:18:41Z

This pull request introduces 4 alerts when merging 0bd7cfb into 905874d - view on LGTM.com

new alerts:

4 for Illegal raise

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-11-09T00:35:45Z

This pull request introduces 4 alerts when merging 2061ab1 into 905874d - view on LGTM.com

new alerts:

4 for Illegal raise

prgeor · 2022-11-09T22:49:55Z

sonic-ycabled/tests/test_ycable.py

@@ -32,6 +33,20 @@

 os.environ["YCABLE_UNIT_TESTING"] = "1"

+exception_output_expected= """\


will this work if python version changes in future?

this would not be backwards compatible, thats why I had added some string checks only,
Better to omit this check I think.

prgeor · 2022-11-09T22:51:05Z

sonic-ycabled/tests/test_ycable.py

+        assert(expected_exception_start.args == expected_exception_join.args)
+        assert("NotImplementedError" in str(trace) and "effect" in str(trace))
+        assert("sonic-ycabled/ycable/ycable_utilities/y_cable_helper.py" in str(trace))
+        assert("line 3655" in str(trace))


can the line no change if the code changes?

yes line number will change and assert will have to be updated.
Maybe we need to just focus on the trace lines

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

lgtm-com · 2022-11-10T22:08:35Z

This pull request introduces 4 alerts when merging 419e742 into 905874d - view on LGTM.com

new alerts:

4 for Illegal raise

Cherry-pick conflict #306 Signed-off-by: vaibhav-dahiya vdahiya@microsoft.com This PR is changing how exception handling is handled in python threading. In python3.7 if child thread gets an exception, there is no way of knowing an exception occured, unless a join is called. The problem is resolved in this PR Also task classes are structured this way class YcableInfoUpdateTask(threading.Thread): def __init__(self): threading.Thread.__init__(self) self.exc = None def run(self): if self.task_stopping_event.is_set(): return self.task_thread = threading.Thread(target=self.task_worker, args=(y_cable_presence,)) self.task_thread.start() try: self.task_worker(self.y_cable_presence) except Exception as e: helper_logger.log_error("Exception occured at child thread YcableInfoUpdateTask due to {} {}".format(repr(e), traceback.format_exc())) self.exc = e def join(self): threading.Thread.join(self) if self.exc: raise self.exc This allows helping catching the exception thrown by a child thread get caught along with traceback. Main thread has a while loop, which monitors each thread, if not running will kill the whole process and supervisord will bring back the ycabled up. Description Motivation and Context How Has This Been Tested? Unit-tests and deploying changes on testbed Additional Information (Optional) sonic-mgmt tests which pass with this change /var/src/sonic-mgmt-int/tests/logs/dualtor/test_ipinip.py::test_decap_active_tor /var/src/sonic-mgmt-int/tests/logs/dualtor/test_ipinip.py::test_decap_standby_tor /var/src/sonic-mgmt-int/tests/logs/dualtor/test_toggle_mux /var/src/sonic-mgmt-int/dualtor_io/test_link_failure.py::test_standby_tor_downlink_down_downstream_active /var/src/sonic-mgmt-int/tests/logs/platform_tests/api/test_chassis.py

[ycabled] fix exception-handling logic for ycabled

e3cd1a0

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix all changes

68e9412

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

vdahiya12 added 3 commits October 19, 2022 23:44

fix some typo

cdcdc2e

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix UT

aa70dcf

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix typo

cc96144

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix tests

81cf7ca

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

vdahiya12 added 3 commits October 21, 2022 01:17

fix

6421d9e

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix more UT

5320c94

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

add coverage

b898ffa

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix UT

2807278

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

try LGTM fix

3cfc033

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix LGTM

2c14791

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

vdahiya12 requested a review from prgeor October 31, 2022 17:10

vdahiya12 added the Y-Cable label Oct 31, 2022

realign

a71b11b

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

vdahiya12 added 2 commits November 1, 2022 23:45

fix spaces

bfc979c

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix UT

e92c74f

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

add all

9745e6a

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix missing changes

03c43d1

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

prgeor reviewed Nov 7, 2022

View reviewed changes

Merge branch 'master' into exception-changes

1ac2a2b

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

add exception handling UT

56c48ef

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix UT

2e19b16

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix string

e23540c

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix some exceptions

0bd7cfb

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

fix UT

2061ab1

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

vdahiya12 requested a review from prgeor November 9, 2022 00:56

prgeor reviewed Nov 9, 2022

View reviewed changes

fix comment

419e742

Signed-off-by: vaibhav-dahiya <vdahiya@microsoft.com>

prgeor approved these changes Nov 11, 2022

View reviewed changes

prgeor merged commit e8c5657 into sonic-net:master Nov 11, 2022

vdahiya12 added Request for 202012 Branch Request for 202205 Branch Cherry Pick Conflict_202012 and removed Request for 202012 Branch labels Nov 11, 2022

vdahiya12 mentioned this pull request Nov 11, 2022

[ycabled] fix exception-handling logic for ycabled #312

Merged

keboliu mentioned this pull request Nov 11, 2022

[submodule] Advance sonic-platform-daemons pointer sonic-net/sonic-buildimage#12674

Closed

7 tasks

stephenxs mentioned this pull request Nov 22, 2022

[submodule] Advance sonic-utilities and sonic-platform-common pointers sonic-net/sonic-buildimage#12791

Merged

7 tasks

yxieca pushed a commit that referenced this pull request Nov 29, 2022

[ycabled] fix exception-handling logic for ycabled (#306)

8324c26

yxieca added the Included in 202205 Branch label Nov 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ycabled] fix exception-handling logic for ycabled #306

[ycabled] fix exception-handling logic for ycabled #306

vdahiya12 commented Oct 19, 2022 •

edited

Loading

lgtm-com bot commented Oct 19, 2022

lgtm-com bot commented Oct 19, 2022

lgtm-com bot commented Oct 20, 2022

lgtm-com bot commented Oct 21, 2022

lgtm-com bot commented Oct 21, 2022

lgtm-com bot commented Oct 21, 2022

lgtm-com bot commented Oct 21, 2022

lgtm-com bot commented Oct 21, 2022

lgtm-com bot commented Nov 1, 2022

lgtm-com bot commented Nov 1, 2022

lgtm-com bot commented Nov 2, 2022

prgeor commented Nov 7, 2022

prgeor Nov 7, 2022

vdahiya12 Nov 8, 2022

lgtm-com bot commented Nov 7, 2022

lgtm-com bot commented Nov 8, 2022

lgtm-com bot commented Nov 8, 2022

lgtm-com bot commented Nov 8, 2022

lgtm-com bot commented Nov 9, 2022

lgtm-com bot commented Nov 9, 2022

prgeor Nov 9, 2022

vdahiya12 Nov 9, 2022

prgeor Nov 9, 2022

vdahiya12 Nov 9, 2022

lgtm-com bot commented Nov 10, 2022

		@@ -32,6 +33,20 @@

		os.environ["YCABLE_UNIT_TESTING"] = "1"

		exception_output_expected= """\

[ycabled] fix exception-handling logic for ycabled #306

[ycabled] fix exception-handling logic for ycabled #306

Conversation

vdahiya12 commented Oct 19, 2022 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Additional Information (Optional)

lgtm-com bot commented Oct 19, 2022

lgtm-com bot commented Oct 19, 2022

lgtm-com bot commented Oct 20, 2022

lgtm-com bot commented Oct 21, 2022

lgtm-com bot commented Oct 21, 2022

lgtm-com bot commented Oct 21, 2022

lgtm-com bot commented Oct 21, 2022

lgtm-com bot commented Oct 21, 2022

lgtm-com bot commented Nov 1, 2022

lgtm-com bot commented Nov 1, 2022

lgtm-com bot commented Nov 2, 2022

prgeor commented Nov 7, 2022

prgeor Nov 7, 2022

Choose a reason for hiding this comment

vdahiya12 Nov 8, 2022

Choose a reason for hiding this comment

lgtm-com bot commented Nov 7, 2022

lgtm-com bot commented Nov 8, 2022

lgtm-com bot commented Nov 8, 2022

lgtm-com bot commented Nov 8, 2022

lgtm-com bot commented Nov 9, 2022

lgtm-com bot commented Nov 9, 2022

prgeor Nov 9, 2022

Choose a reason for hiding this comment

vdahiya12 Nov 9, 2022

Choose a reason for hiding this comment

prgeor Nov 9, 2022

Choose a reason for hiding this comment

vdahiya12 Nov 9, 2022

Choose a reason for hiding this comment

lgtm-com bot commented Nov 10, 2022

vdahiya12 commented Oct 19, 2022 •

edited

Loading