[CMIS]Improved 400G link bring up sequence #254

prgeor · 2022-04-11T23:31:50Z

Description

Improved 400G link bring up sequence HLD

Ensure CMIS compliant transceiver Datapath are initialized and activated IFF the host Tx signal is good and admin_status is UP
get_all_namespace() API is not reentrant resulting in crash, so we cache the namespace at the xcvrd start
Datapath is deactivated if either 'admin_status' becomes down or 'host_tx_ready' signal becomes OFF.
An option "--skip_xcvrd_cmis_mgr" is added to skip launch of CMIS task manager for platforms that are in transition to using SFP-refactor code
Datapath is NOT impacted if Xcvrd crash/restarts

Motivation and Context

Deterministically improve the link bring up of 400G link that are using CMIS compliant transceivers.

How Has This Been Tested?

Tested on

Cisco 8800 chassis: Being validated by Cisco
Dell Z9332F

Additional Information (Optional)

sacnaik · 2022-04-12T04:58:07Z

sonic-xcvrd/xcvrd/xcvrd.py

+    keys = port_tbl.getKeys()
+
+    if lport in keys:
+       cmd = 'sonic-db-cli STATE_DB hget "PORT_TABLE|{}" host_tx_ready'.format(lport)


Pmon is global container, how this wok under multi NPU? I mean sonic-db-cli does not require to look namesapce db ?

Pmon is global container, how this wok under multi NPU? I mean sonic-db-cli does not require to look namesapce db ?

@prgeor updated that he would make this thing namespace aware i.e. xcvrd would subscribe/listen to all namespace's stateDB and then act accordingly. This piece yet to be added.
Prince may comment/update further

@sacnaik @shyam77git please check the updated code

sacnaik · 2022-04-12T05:00:59Z

sonic-xcvrd/xcvrd/xcvrd.py

                        # D.2.2 Software Deinitialization
                        api.set_datapath_deinit(host_lanes)
-                        api.set_lpmode(True)


You are changing generic code. Isn't this set_lpmode is required function?

Each Datapath is independent, this code will bring the whole module to low power 8 times if we have 8 data paths being initialized.

It sounds like this PR will perform CMIS application selection one by one for each datapath(i.e. logical port on the MAC) instead of a one-shot configuration for all the datapatches on a transceiver.

What's unclear to me is as follows:

Is there a switch ASIC available to support asynchronous mixed port mode in a port macro? (i.e. Is it possible to have both 1x100G-NZR + 2x100G-PAM4 on a single QSFPDD port, from MAC perspective?)

Is there a CMIS transceiver available to support asynchronous mixed port mode in a single module?

In the above scenario, each datapath will require a dedicated clock divisor while they all share the same reference clock source. As far as I know, this is NOT a supported port mode in Broadcom ASIC, but I'm not sure if this is also applicable to other vendors, could you please clarify if there is a such module available for testing? Thank you

Xcvr is lisenting to various port change events related to a ports speed, lane mapping etc and these events across various logical ports in the case of breakout port (say one physical port mapping to 4 logical ports) will arrive asynchronously, so we need to handle these events independently, even though it belong to same physical port.

sacnaik · 2022-04-12T05:03:31Z

sonic-xcvrd/xcvrd/xcvrd.py

                        self.port_dict[lport]['cmis_state'] = self.CMIS_STATE_READY
                        continue

                    # Skip if it's not a paged memory device
                    if api.is_flat_memory():
+                        self.log_notice("{}: skipping CMIS state machine for flat memory xcvr".format(lport))


You may be periodically logging this, then it is not right.

the state machine moves to CMIS_STATE_READY after this, and doesnot come again here unless the state is changed.

sonic-xcvrd/xcvrd/xcvrd.py

jaganbal-a · 2022-04-14T07:21:00Z

sonic-xcvrd/xcvrd/xcvrd.py

+                        # NOTE: Some CMIS compliant modules may have 'auto-squelch' feature where
+                        # the module won't take datapaths to Activated state if host tries to enable
+                        # the datapaths while there is no good Tx signal from the host-side.
+                        if not is_host_tx_ready(lport):


Can this be implemented as a event handler?

Subscribe to PORT_TABLE in state-db, then filter to host_tx_ready field.

Based on the host_tx_ready status, set the appropriate CMIS module state will be set here in the event handler.

Since the main CMIS task worker keep checking for CMIS state change for all ports, once the point (2) is set, the CMIS task worker will work on the appropriate state.

we can introduce a new state between CMIS_STATE_AP_CONF and CMIS_STATE_DP_INIT, say 'CMIS_STATE_WAIT_FOR_HOST' . so during module insertion the CMIS task worker will set to this state once it completes CMIS_STATE_AP_CONF and this state will be changed to CMIS_STATE_DP_INIT only in the host_tx_ready handler. this way it will be synchornized.

Keeping the host_tx_ready processing as event handler will have following benefit.

Keeping the host_tx_ready out of CMIS task worker to event handler will benefit other speed to follow the deterministic link bringup approach.

It will avoid continuous polling of host-tx-ready using Redis-DB query, which will save time when scaled to fully populated 48/72 ports.

The host_tx_ready event handler can invoke the xcvr_API_factory API say 'tx_disable' which will set appropriate CMIS state in case of CMIS module and it will disable/enable laser/Tx appropriately in case of non-cmis module, which can be leveraged eventually.

@jaganbal-a I think event handling may NOT work when Xcvrd restarts. Consider this:-

400G Port link is up.

Xcvrd crashes and in the meantime someone plugs in another 400G module

Xcvrd won't get another 'host_tx_ready' event because 'host_tx_ready' doesn't depend upon whether module is present or not. So, Xcvrd will keep waiting for 'host_tx_ready' or unless user does admin shut followed by admin up.

Another similar case as above

Xcvrd is running and got 'host_tx_ready' but module is absent

Xcvrd crashes and comes up

Module is plugged in but Xcvrd is not going to receive 'host_tx_ready' and link won't come up

@prince
It is captured as part of design that the host_tx_ready will be updated regardless of transceiver presence, and it is xcvrd job to check the host_tx_ready during module Init during insertion (a local host_tx_ready dictionary will help here but not polling), so the DP state will be moved to DP_INIT/DP_ACTIVATE state based on host_tx_ready. Above mentioned cases will be covered.

@jaganbal-a Xcvrd process don't have a persistent storage to capture the 'host_tx_ready' state, so it depends upon the Redis-DB i.e it needs to read the 'host_tx_ready' state from the DB. Let Xcvrd read the state when it requires rather than being notified. Because if we have to take the approach of notification, then we need to handle the above Xcvrd exit case separately by reading the DB which can be avoided if Xcvrd can read the DB when required.

@prgeor

Let Xcvrd read the state when it requires rather than being notified.

Can you clarify as to what is meant by 'when it requires'?
Does it mean continuous polling of host-tx-ready using Redis-DB query OR what?
When interface/port moves from config shut to no shut or vice-versa, won't it a notification of host_tx_ready from redisDB otherwise how would it xcvrd be notified of this? repeated/continuous polling indefinitely without any event happening in the system is waste of cpu cycles / system resources.
Also config interface shut/no shut are rare events along with optics removal/insertion
Also, per HLD we decided to go with notification based mechanism of host_tx_ready.
To handle xcvrd restart case, as Jagan mentioned above, better to check the host_tx_ready during module Init during insertion as source of truth.

@shyam77git @jaganbal-a please take a look again as i updated the code.

prgeor · 2022-04-19T23:24:32Z

@ds952811 can you review

stephenxs · 2022-04-20T15:30:49Z

sonic-xcvrd/xcvrd/xcvrd.py

@@ -870,9 +872,11 @@ def __init__(self, port_mapping):
        self.task_process = None
        self.port_dict = {}
        self.port_mapping = copy.deepcopy(port_mapping)
+        self.xcvr_table_helper = XcvrTableHelper()


If XcvrTableHelper is implemented as a class member, it should be initialized in task_worker instead of __init__.
This is because CmisManagerTask is a subprocess wrapper, which means a subprocess will be forked once task_run is called.
If it is initialized in __init__, the flow is

parent creates the XcvrTableHelper along with all the low level objects, like the sockets based on which the tables are accessed

and then the child is forked, all resources opened by the parent are duplicated

now there are two copies of all the resources but the parent's one will not really be used.

Of course, initializing it in __init__ and then duplicating it from parent to child should also work. But it looks like taking a detour to me.

@stephenxs what is the concern here? only the worker thread is going to use the socket, not the parent.

xcvr_table_helper is initialized in the parent and then duplicated in the client. but it will never be used by the parent.
so the resource allocated in the parent is wasted.

if it is initialized in __init__:

xcvr_table_helper initialized in parent. resource allocated for it but not used.

duplicated in the child during fork

if it is initialized in task_worker

xcvr_table_helper is initialized in the child before it starts to work.

no resource was allocated in the parent.

as I mentioned, it also works in the current way. so I'm ok if you do not decide to change it.

jaganbal-a · 2022-04-21T19:22:10Z

sonic-xcvrd/xcvrd/xcvrd.py

-            return
+        # 'index' can be -1 if STATE_DB|PORT_TABLE
+        # TODO:Make sure doesnot breakt non-CMIS
+        #ptype = _wrapper_get_sfp_type(pport)


I believe get_sfp_type() returns the sfp_type present in the port and not the port cage type.
So can you rename the variable and uncomment the code?

get_sfp_type() returns only the port/cage type, and it's a constant initialized at chassis.init()
e.g.
https://github.com/Azure/sonic-buildimage/blob/master/platform/broadcom/sonic-platform-modules-dell/z9332f/sonic_platform/chassis.py#L132
And a QSFP28 could be attached to the QSFPDD port on the box, and it's better to skip the CMIS init sequence for it, hence please uncomment the code.

@ds952811 form-factor of the transceiver is not the right check to determine whether the module follows CMIS or not. Even QSFP form-factor may follow CMIS sonic-net/sonic-platform-common#246 that follows CMIS

jaganbal-a · 2022-04-21T20:49:30Z

sonic-xcvrd/xcvrd/xcvrd.py

@@ -1150,6 +1182,14 @@ def task_worker(self):
                             self.CMIS_STATE_REMOVED]:
                    continue

+                # Handle the case when Xcvrd was NOT running when 'host_tx_ready' or 'admin_status'
+                # was updated or this is the first run so reconcile the above two attributes
+                if 'host_tx_ready' not in self.port_dict[lport]:


This code will not get hit if the port is CMIS_STATE_READY, so with config-interface shutdown the event will be returned in the above if condition and the modules will not be put in DP-DEACTIVATE.

If any runtime Tx fault handler being implemented and the host_tx_ready is set to false, then the event will be missed.

CMIS state reset happens if any port configuration is changed including admin or host_tx_ready status here :https://github.com/Azure/sonic-platform-daemons/blob/master/sonic-xcvrd/xcvrd/xcvrd.py#L919

ds952811 · 2022-04-22T00:35:16Z

sonic-xcvrd/xcvrd/xcvrd.py

    def task_worker(self):
-        self.log_notice("Starting...")
+        self.log_notice("Starting CmisTaskMgr with pid:{}".format(self.task_process.pid))


If you checked the syslog of pmon#xcvrd, the pid is available over there, on the other hand, we already have the 'CMIS' prefix in the CmisManagerTask.log_notice(), hence this is not necessary

ds952811 · 2022-04-22T00:38:28Z

sonic-xcvrd/xcvrd/xcvrd.py


        # APPL_DB for CONFIG updates, and STATE_DB for insertion/removal
-        sel, asic_context = port_mapping.subscribe_port_update_event(['APPL_DB', 'STATE_DB'])
+        sel, asic_context = port_mapping.subscribe_port_update_event()


While APPL_DB is for monitoring config updates arrived at swss#orchagent, the STATE_DB is for the transceiver insertion/removal detection, and it's not necessary to have both listeners registered all the time.
May I have a clarification on why we need this?

@ds952811 A new signal "host_tx_ready" is introduce for 400G QSFP-DD transceiver datapath activation as per this HLD where the OA will control this signal and Xcvrd will listen to any change of this signal in STATE_DB.

sonic-xcvrd/xcvrd/xcvrd_utilities/port_mapping.py

sonic-xcvrd/xcvrd/xcvrd.py

jaganbal-a · 2022-05-18T22:02:21Z

sonic-xcvrd/xcvrd/xcvrd.py

@@ -1284,6 +1327,13 @@ def task_worker(self):
                try:
                    # CMIS state transitions
                    if state == self.CMIS_STATE_INSERTED:
+                        if self.port_dict[lport]['host_tx_ready'] != 'True' or \


Please change it 'true' and 'false' for host_tx_ready handling through out the file as we discussed.
Also change it in test_xcvrd.py

sonic-xcvrd/xcvrd/xcvrd.py

jaganbal-a · 2022-06-04T15:52:03Z

sonic-xcvrd/xcvrd/xcvrd.py

+            if port_change_event.port_dict is not None and 'host_tx_ready' in port_change_event.port_dict:
+                self.port_dict[lport]['host_tx_ready'] = port_change_event.port_dict['host_tx_ready']
+            if port_change_event.port_dict is not None and 'admin_status' in port_change_event.port_dict:
+                self.port_dict[lport]['admin_status'] = port_change_event.port_dict['admin_status']
            self.port_dict[lport]['cmis_state'] = self.CMIS_STATE_INSERTED


Change the CMIS state to inserted and invoke reset_cmis() only when there is a change in value for the following fields speed / lanes / host_tx_ready or admin_status.
With current change, the cmis state is set to INSERTED even for oper_state state change in value in the DB for a Port.

the port change event doesn't have a mechanism to filter out the event based upon the corresponding PORT_TABLE. So, until we have this mechanism to filter out, we need this code here otherwise Xcvrd will miss optics jack out/in

jaganbal-a · 2022-06-09T02:12:05Z

sonic-xcvrd/xcvrd/xcvrd.py

+            if port_change_event.port_dict is not None and 'host_tx_ready' in port_change_event.port_dict:
+                self.port_dict[lport]['host_tx_ready'] = port_change_event.port_dict['host_tx_ready']
+            if port_change_event.port_dict is not None and 'admin_status' in port_change_event.port_dict:
+                self.port_dict[lport]['admin_status'] = port_change_event.port_dict['admin_status']
            self.port_dict[lport]['cmis_state'] = self.CMIS_STATE_INSERTED


this is redundant, reset_cmis() is doing this.

sonic-xcvrd/xcvrd/xcvrd.py

shyam77git · 2022-06-23T18:10:24Z

sonic-xcvrd/xcvrd/xcvrd.py

+            if 'admin_status' in port_change_event.port_dict and 'oper_status' in port_change_event.dict:
+                # At times 'admin_status' is NOT the same in the PORT_TABLE of APPL_DB and STATE_DB
+                # We dont have better way to check if 'admin_status' is from APPL_DB or STATE_DB so this
+                # check is put temporarily to list only to APPL_DB's admin_status and ignore that of STATE_DB


instead of 'list only' it should be 'listen only'

jaganbal-a · 2022-06-23T18:49:33Z

sonic-xcvrd/xcvrd/xcvrd.py

@@ -994,7 +994,10 @@ def on_port_update_event(self, port_change_event):
                self.port_dict[lport]['lanes'] = port_change_event.port_dict['lanes']
            if 'host_tx_ready' in port_change_event.port_dict:
                self.port_dict[lport]['host_tx_ready'] = port_change_event.port_dict['host_tx_ready']
-            if 'admin_status' in port_change_event.port_dict:
+            if 'admin_status' in port_change_event.port_dict and 'oper_status' in port_change_event.dict:


in port_change_event.dict: -> in port_change_event.port_dict:

shyam77git

LGTM

jaganbal-a · 2022-06-25T00:41:48Z

Link_bring_up_xcvrd_unit_test_log.txt

jaganbal-a · 2022-06-25T00:42:34Z

sonic-xcvrd/xcvrd/xcvrd.py

        if port_change_event.event_type == port_change_event.PORT_SET:
            if pport >= 0:
                self.port_dict[lport]['index'] = pport
-            if port_change_event.port_dict is not None and 'speed' in port_change_event.port_dict:
+            if 'speed' in port_change_event.port_dict and port_change_event.port_dict['speed'] != 'N/A':


Please add a trace for each event that is processed.

* Improved 400G link bring up sequence * Event based handling * Remove unused functions * Force DP to remain in DpInitialized State state on admin shutdown * Added test case * Skip CMIS task manager based upon flag * Addressed review comments * Fix xcvrd crash * Fix test failure * Listen only to 'APPL_DB's admin_status * Fix typo

sacnaik reviewed Apr 12, 2022

View reviewed changes

shyam77git reviewed Apr 14, 2022

View reviewed changes

sonic-xcvrd/xcvrd/xcvrd.py Show resolved Hide resolved

jaganbal-a reviewed Apr 14, 2022

View reviewed changes

prgeor marked this pull request as ready for review April 19, 2022 23:22

prgeor changed the title ~~Improved 400G link bring up sequence~~ [CMIS]Improved 400G link bring up sequence Apr 19, 2022

stephenxs reviewed Apr 20, 2022

View reviewed changes

jaganbal-a reviewed Apr 21, 2022

View reviewed changes

ds952811 reviewed Apr 22, 2022

View reviewed changes

sonic-xcvrd/xcvrd/xcvrd_utilities/port_mapping.py Show resolved Hide resolved

This was referenced May 2, 2022

On config interface shutdown, xcvrd to not reset CMIS state #258

Open

Consumer of same field subscribed to different DBs to have DBs/TABLE check prior to processing event #259

Open

keboliu reviewed May 7, 2022

View reviewed changes

sonic-xcvrd/xcvrd/xcvrd.py Show resolved Hide resolved

prgeor added 4 commits May 10, 2022 00:44

Improved 400G link bring up sequence

57cb73c

Event based handling

9b00b47

Remove unused functions

5c42c57

Force DP to remain in DpInitialized State state on admin shutdown

0204a07

prgeor force-pushed the 400G-link-UP branch from 70e754e to 0204a07 Compare May 10, 2022 00:45

prgeor added 2 commits May 10, 2022 19:18

Added test case

69d3a54

Skip CMIS task manager based upon flag

b9cb34d

jaganbal-a mentioned this pull request May 16, 2022

speed is non-existent in PORT_TABLE of State-DB, but it is being set to N/A sonic-net/sonic-swss#2276

Open

jaganbal-a reviewed May 18, 2022

View reviewed changes

sonic-xcvrd/xcvrd/xcvrd.py Show resolved Hide resolved

jaganbal-a reviewed May 18, 2022

View reviewed changes

jaganbal-a suggested changes May 20, 2022

View reviewed changes

sonic-xcvrd/xcvrd/xcvrd.py Show resolved Hide resolved

shyam77git mentioned this pull request Jun 1, 2022

HLD for 'Have a deterministic approach in SONiC for Interface Link bring-up sequence' sonic-net/SONiC#916

Merged

jaganbal-a reviewed Jun 4, 2022

View reviewed changes

jaganbal-a reviewed Jun 9, 2022

View reviewed changes

rlhui added the Chassis label Jun 10, 2022

prgeor added 2 commits June 20, 2022 22:59

Addressed review comments

be6d509

Fix xcvrd crash

647cff7

rlhui assigned prgeor Jun 22, 2022

prgeor added 2 commits June 22, 2022 18:02

Fix test failure

374972e

Listen only to 'APPL_DB's admin_status

b52a3f6

shyam77git reviewed Jun 23, 2022

View reviewed changes

jaganbal-a reviewed Jun 23, 2022

View reviewed changes

Fix typo

3d25e45

keboliu approved these changes Jun 24, 2022

View reviewed changes

jaganbal-a mentioned this pull request Jun 24, 2022

Orchagent changes for synchronizing npu/phy device Tx in the data path before enabling transceiver<CMIS compliant> Tx. sonic-net/sonic-swss#2277

Merged

shyam77git approved these changes Jun 24, 2022

View reviewed changes

jaganbal-a approved these changes Jun 25, 2022

View reviewed changes

sujinmkang approved these changes Jun 27, 2022

View reviewed changes

prgeor merged commit 7c0be24 into sonic-net:master Jun 28, 2022

This was referenced Jul 12, 2022

[Submodule] Update src/sonic-platform-daemons and src/sonic-swss to absorb link bring-up seq sonic-net/sonic-buildimage#11424

Open

Submodule update src/sonic-platform-daemons and src/sonic-swss sonic-net/sonic-buildimage#11456

Closed

prgeor added the Request for 202205 Branch label Jul 20, 2022

yxieca added the Included in 202205 Branch label Jul 28, 2022

[CMIS]Improved 400G link bring up sequence #254

[CMIS]Improved 400G link bring up sequence #254

Conversation

prgeor commented Apr 11, 2022 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Additional Information (Optional)

Choose a reason for hiding this comment

shyam77git Apr 13, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaganbal-a Apr 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaganbal-a Apr 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prgeor commented Apr 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ds952811 Apr 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaganbal-a May 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shyam77git left a comment

Choose a reason for hiding this comment

jaganbal-a commented Jun 25, 2022

Choose a reason for hiding this comment

prgeor commented Apr 11, 2022 •

edited

Loading

shyam77git Apr 13, 2022 •

edited

Loading

jaganbal-a Apr 14, 2022 •

edited

Loading

jaganbal-a Apr 15, 2022 •

edited

Loading

ds952811 Apr 22, 2022 •

edited

Loading

jaganbal-a May 18, 2022 •

edited

Loading