Consider supporting coherent QSFP-DD (OpenZR+) module control #80
Comments
Do you know how SONiC handles this issue? Who is taking care of the configuration order in SONiC?
How about rebooting QSFP-DD periodically until it gets the valid Tx signal from the ASIC instead of waiting for it? |
|
@ishidawataru
It seems that this issue is recognized by SONiC community, and configurations of
I see. I think it's technically possible and reasonable. @noguchiko @matsuo-tat |
|
It might be reasonable to introduce the rebooting logic in |
|
@KazuyaAnazawa @ishidawataru By the way, xcvrd from SONiC has already implemented a function to retry DataPath transition if it cannot transition up from |
Translation daemons also update the running datastore. In my understandings, the To discuss the architecture, I think that we need to define what (maybe new) attributes of |
|
Thank you so much for your comments.
Yes, I agree with this direction.
Definitely. Thank you for the comments. |
|
Tried to clarify what attributes of First, we need to extend
Then, I thought we may be able to make use of the following OpenConfig models for new model and
Given these configurations from north-layer, each component performs as follows:
@noguchiko Do you have any comments on the above from both implementation and architectural design point of view? |
|
@KazuyaAnazawa Thank you. I'll comment after confirming the details.. |
|
I think the structure is fine. However, it is better not to assume OpenConfig models in the architecture. It possibly decreases functional cohesion and make it difficult to support the feature in other standardized models. Also, current Therefore, I think it is better to focus on providing the feature with goldstone primitive models first. It allows users/controllers to use coherent QSFP-DDs in various modes on devices with Q2C/Q2A switch ASIC. After that, we can wrap the feature in standardized models (e.g. OpenConfig). It may improve usability of the feature. |
|
@noguchiko Thank you so much for your valuable comments. You are right. I will investigate CMIS/C-CMIS and xcvrd API in detail, and try to clarify features/attributes that should be provided by goldstone primitive models (i.e., The main focus of NEC side will be QSFP-DD module configuration by goldstone mgmt daemons, so could you please do this in NEC side in parallel? @ipi-claytonpascoal @iAgrawalGaurav |
|
@noguchiko @matsuo-tat The reason is as follows.
The sequence by picture is as follows.
For these reasons, I think we just need to add "network-interface-mode" and "host-interface-mode" at |
|
@KazuyaAnazawa Thank you for the update. It looks good to me. I have one question for clarification.
In order to identify the |
|
@noguchiko Thank you for checking.
Yes, your understanding is correct. The lane numbers for each module are provided in that platform specific file |
|
@KazuyaAnazawa
However, on future releases, OcNOS will no longer depend on interface port-breakout and ASIC configuration will be applied based only on AppSelCode. So, I would suggest south-ocnos also subscribe to goldstone-transponder and get host-interface-mode and network-interface-mode information. In this way, south-ocnos can handle the differences between OcNOS releases without any impact on the other components in the system. As per requirement 2, "Each Goldstone primitive model should be subscribed and handled by only one south daemon", I think that the intermediate-layer plugin may be necessary to subscribe to goldstone-transponder and replicate the data to both south-ocnos and south-xcvr. |
|
@ipi-claytonpascoal Thank you for your comments and useful information. Let me double check for clarification.
|
That is the plan for future OcNOS releases, a few adaptations may be required on OcNOS.
No, it does not. The current OcNOS release (6.4.0) still relies on the port breakout to configure the switch ASIC side. |
|
I think we need to support both current OcNOS release (port breakout / serdes rate configurations are supported) and its future releases (AppSelCode configuration is supported).
One of the solutions could be the introduction of intermediate-layer plugin so that above requirements are satisfied (Suggestion from @ipi-claytonpascoal). Do you have any comments or proposal for this? |
|
@KazuyaAnazawa @ipi-claytonpascoal
That is preferable, but the Goldstone management framework does not explicitly require that. There is an issue if multiple software components subscribe to the operational state request for the same model. In that case, the operational state values responded by each of the subscribers may conflict, and the datastore may not be able to resolve it. On the other hand, subscribing to the same configuration change by multiple components should not be an issue. Each component configures a different target, and if one fails, the failure can be propagated (by How about the following design?
|
|
@noguchiko Thank you for your comments.
I see, I overlooked it. Thank you for pointing it out. The design looks good to me, but let me look into it in detail. @ipi-claytonpascoal Do you have any comments or doubt? |
|
@KazuyaAnazawa The design proposed by @noguchiko looks fine. Just a suggestion, I think that it would be better if the changes in OcNOS behavior along the releases did not change how the north-bound configures the feature, in this case, the port-breakout configuration would be obsoleted in future OcNOS releases. Regarding the operational data, south-ocnos does not need to subscribe to the goldstone-transponder, subscription to goldstone-interfaces for the operational state of broken-out ports should be enough. |
|
@ipi-claytonpascoal Thank you so much for your suggestion. By the way, in that case, we may need to consider how to handle non-coherent QSFP-DDs because both coherent/non-coherent QSFP-DDs can be inserted on the same ports (also, @noguchiko @matsuo-tat What do you think about Clayton's proposal and handling non-coherent QSFP-DDs? |
|
@ipi-claytonpascoal @KazuyaAnazawa It looks good to me too. It simplifies the operation and will be valuable to users.
At least, it seems that it is not appropriate to manage non-coherent "transceivers" with a model called "transponder". There are multiple possible designs to solve this. For example, rename I don't have any solid ideas right now, but I'll try to consider about it. |
@KazuyaAnazawa @ipi-claytonpascoal Could you explain why this decision was made? The Goldstone primitive YANG models basically map to a specific hardware component on the device. (Hence You may argue we'll have redundant configurations in the primitive YANG models by following the above design. ( |
@noguchiko How about introducing I'm not sure if we need to keep using Do we really need to keep using |
|
@ishidawataru Thank you so much for your comments and advice. As Clayton told here, there is a difference between OcNOS (v6.4.0) and OcNOS (ver > 6.4.0) in controlling ASIC's SerDes. So, I tried to design south-ocnos so that both of OcNOS (ver >= 6.4.0) can be used on Goldstone. For OcNOS v.6.4.0 case, we can subscribe to That's why we made a decision of subscribing
|
|
@KazuyaAnazawa Do you plan to use OcNOS to control the pluggable transceivers as well? Or are you going to use
If OcNOS doesn't control the pluggable transceivers, I think we can use a random network-interface side configuration. |
|
@ishidawataru Thank you for your comments.
We would like to use
I think it is difficult because
Thank you for your suggestion. Yes, this can be one possible solution. |
In that case, how does OcNOS check if the given AppSelCode is valid when its peripheral control function is disabled? |
|
@ishidawataru However, if OcNOS internally checks if given |
@ishidawataru That's an important point. I understood the concept as follows, is this correct?
If so,
I think that's appropriate. We should introduce a new model |
|
@noguchiko Thank you for the summary. It matches the idea behind the architecture. (I think we should put the summary in README.md) |
|
@KazuyaAnazawa , your comment regarding the design decision between OcNOS releases is excellent. I do not have anything to add.
This is a work in progress but I will guarantee that this requirement will be met.
I agree with this approach and I do not think that OcNOS does this kind of validation. Even if it does validation could be bypassed in Goldstone since south-ocnos will guarantee that AppSelCode is valid. I will double-check that and confirm that OcNOS does not need to access SFF transceiver in any situation.
OpenConfig has the leaf /components/component/transceiver/config/module-functional-type that can be used to differentiate digital coherent optic and standard grey optic. If I am not mistaken grey optics are handled by the openconfig-platform-transceiver model and coherent modules are handled by openconfig-terminal-device. I hope that can help with |
|
@ipi-claytonpascoal
Also, @noguchiko @ishidawataru |
|
@KazuyaAnazawa I agree with the proposed design. Regarding the interfaces involved in the port breakout, the parent interface ( |
|
@ipi-claytonpascoal Thank you for your reply.
For better understanding, attach the figure describing how it works.
As described above, |
|
@KazuyaAnazawa I have one comment.
I think that south daemons should not return running-config (data in |
|
@noguchiko Thank you for pointing it out. So sorry, it was just my mistake. |
|
@KazuyaAnazawa Thank you very much for the detailed explanation. It is all clear now! |
|
@KazuyaAnazawa @ishidawataru I'm designing the The The model should support the following management features:
I think that we have the following two options for how to model it:
The following (collapsed) sections are examples for each option. I would appreciate any comments on which option or another we should take. Based on CMIS registersBased on CMIS registersTreemodule: goldstone-transceiver
+--rw modules
+--rw module* [index]
+--rw index -> ../config/index
+--rw config
| +--rw index
| :
+--ro state
| :
+--rw host-lanes
| +--rw host-lane* [index]
| +--rw index -> ../config/index
| +--rw config
| | +--rw index
| | +--rw app-sel-code
| | +--rw data-path-id # DataPathID [0-7] - typically first host lane number minus one
| | +--rw network-path-id # NPID [0-7] - typically first host lane number minus one
| | :
| +--ro state
| :
| +--ro data-path-state
| :
+--rw media-lanes
| +--rw media-lane* [index]
| +--rw index -> ../config/index
| +--rw config
| | +--rw index
| | +--rw target-output-power
| | +--rw frequency
| | :
| +--ro state
: :Configuration exampleConfiguration example for a mixed (heterogenous) multiplex application (described in "7.6.3 Network Path Applications") {
"modules": {
"module": [
{
"index": 0,
"config": {
"index": 0,
},
"host-lanes": {
"host-lane": [
{
"index": 1,
"app-sel-code": 1, # e.g. host rate 100G, 2 host lanes, media rate 400G
"data-path-id": 0,
"network-path-id": 0,
},
{
"index": 2,
"app-sel-code": 1, # e.g. host rate 100G, 2 host lanes, media rate 400G
"data-path-id": 0,
"network-path-id": 0,
},
{
"index": 3,
"app-sel-code": 1, # e.g. host rate 100G, 2 host lanes, media rate 400G
"data-path-id": 2,
"network-path-id": 0,
},
{
"index": 4,
"app-sel-code": 1, # e.g. host rate 100G, 2 host lanes, media rate 400G
"data-path-id": 2,
"network-path-id": 0,
},
{
"index": 5,
"app-sel-code": 2, # e.g. host rate 200G, 4 host lanes, media rate 400G
"data-path-id": 4,
"network-path-id": 0,
},
{
"index": 6,
"app-sel-code": 2, # e.g. host rate 200G, 4 host lanes, media rate 400G
"data-path-id": 4,
"network-path-id": 0,
},
{
"index": 7,
"app-sel-code": 2, # e.g. host rate 200G, 4 host lanes, media rate 400G
"data-path-id": 4,
"network-path-id": 0,
},
{
"index": 8,
"app-sel-code": 2, # e.g. host rate 200G, 4 host lanes, media rate 400G
"data-path-id": 4,
"network-path-id": 0,
},
]
},
"media-lanes": {
"media-lane": [
{
"index": 1,
"config": {
"target-output-power": 0.0,
"frequency": 193100000,
}
}
]
}
}
]
}
}Based on CMIS functional modelsBased on CMIS functional modelsTreemodule: goldstone-transceiver
+--rw modules
+--rw module* [index]
+--rw index -> ../config/index
+--rw config
| +--rw index
| :
+--ro state
| :
+--rw data-paths
| +--rw data-path* [id]
| +--rw id -> ../config/id
| +--rw config
| | +--rw id # DataPathID [0-7] - typically first host lane number minus one
| | :
| +--ro state
| | :
| | +--ro data-path-stete
| | :
| +--rw host-paths
| | +--rw host-path* [first-lane]
| | +--rw first-lane -> ../config/first-lane
| | +--rw config
| | | +--rw first-lane # The first host lane number of the Host Path
| | | +--rw app-sel-code # MediaInterfaceID of the application must be the same for all host paths
| | | :
| | +--ro state
| | :
| +--rw network-path
| | +--rw config
| | | +--rw media-lanes # Associated media lanes e.g. [1]
| | +--ro state
| | | :
| | +--rw media-lanes
| | +-- media-lane* [index]
| | +--rw index -> ../config/index
| | +--rw config
| | | +--rw index
| | | +--rw target-output-power
| | | +--rw frequency
| | | :
| | +--ro state
: : :Configuration exampleConfiguration example for a mixed (heterogenous) multiplex application (described in "7.6.3 Network Path Applications") {
"modules": {
"module": [
{
"index": 0,
"config": {
"index": 0,
},
"data-paths": {
"data-path": [
{
"id": 0,
"config": {
"id": 0,
},
"host-paths": {
"host-path": [
{
"first-lane": 1,
"config": {
"first-lane": 1,
"app-sel-code": 1, # e.g. host rate 100G, 2 host lanes, media rate 400G
}
},
{
"first-lane": 3,
"config": {
"first-lane": 3,
"app-sel-code": 1, # e.g. host rate 100G, 2 host lanes, media rate 400G
}
},
{
"first-lane": 5,
"config": {
"first-lane": 5,
"app-sel-code": 2, # e.g. host rate 200G, 4 host lanes, media rate 400G
}
}
]
},
"network-path": {
"config": {
"media-lanes": [1]
},
"media-lanes": {
"media-lane": [
{
"index": 1,
"config": {
"index": 1,
"target-output-power": 0.0,
"frequency": 193100000,
}
}
]
}
}
}
]
}
}
]
}
}Other featuresOther featuresmodule: goldstone-transceiver
+--rw modules
+--rw module* [index]
:
+--ro applications # Application Descriptors
| +--ro application* [code]
| +--ro code -> ../state/code
| +--ro state
| +--ro code
| +--ro host-interface-id
| +--ro media-interface-id
| +--ro host-lane-count
| +--ro media-lane-count
| +--ro host-lane-assignment-options
| +--ro media-lane-assignment-options
| +--ro np-application # ExtAppDescriptor
+--ro monitors # VDM items (TODO)
+--ro alarms # Alarms and warnings from Flags (TODO)
|
|
@noguchiko Can the second option represent the default empty configuration? (in the state container) |
Yes, it can. All The current configuration of the Data Path structure can be retrieved from the Upper memory page 11h "Lane and Data Path Status" (for |
|
@noguchiko Thank you for your proposal. I will take a look. Please give me some time. |
Does CMIS mandate that all host and network lanes be used for data paths? ( BTW I'm not saying that it's bad that we can't see the unused host/network lanes, just want to check the pros/cons of both options) |
No. Unused lanes are allowed. CMIS "6.2.3.2 Control Set Content" says: The special AppSel code value 0000b in the Data Path Configuration register of a host lane indicates that the lane (together with its associated resources) is unused and not part of a Data Path.
That's right.
I think that the second option is easier for software (controllers and the south daemon) to handle because it has less redundant information. |
How does the Also, could you show a configuration example of the second option where we have 2 data paths? Is it supported with the transceiver you're assuming? |
The north daemons can know the lanes capability from the application advertisement. It is the list of applications supported by the module. The north daemon can calculate the available lanes in the module by using module: goldstone-transceiver
+--rw modules
+--rw module* [index]
:
+--ro applications
| +--ro application* [code]
| +--ro code -> ../state/code
| +--ro state
| +--ro code
| +--ro host-interface-id
| +--ro media-interface-id
| +--ro host-lane-count # Number of host lanes
| +--ro media-lane-count # Number of media lanes
| +--ro host-lane-assignment-options # The lanes where the Application is supported on the module's host interface
| +--ro media-lane-assignment-options # The lanes where the Application is supported on the module's media interface
| +--ro np-application
:For example,
No, the transceiver we are assuming doesn't support multiple data paths. However, the CMIS specification (6.2.1 Functional Module Capabilities - Applications) allows multiple data paths in a module. Therefore, I think it would be good for that the model can represent multiple data paths. Configuration exampleCMIS (6.2.1.3 Multiple Application Instances and Multiple Applications) says: For example, a module may support one 400Gbps Application that is characterized by a 400GAUI-8 host interface and a 400GBASE-DR4 media interface combination, and a second 100Gbps Application that is characterized by a 100GAUI-2 host interface and a 100GBASE-DR media interface combination. The module may be programmable to work as one instance of the first Application or to work as up to four instances of the second Application. In this context, an application instance means a data path. The following is the configuration example ("up to four instances of the second Application"): {
"modules": {
"module": [
{
"index": 0,
"config": {
"index": 0,
},
"data-paths": {
"data-path": [
{
"id": 0,
"config": {
"id": 0,
},
"host-paths": {
"host-path": [
{
"first-lane": 1,
"config": {
"first-lane": 1,
"app-sel-code": 2, // the second application 100G to 100G
}
},
]
},
"network-path": {
"media-lanes": {
"media-lane": [
{
"index": 1,
"config": {
"index": 1,
"target-output-power": 0.0,
"frequency": 193100000,
}
}
]
}
}
}
{
"id": 2,
"config": {
"id": 2,
},
"host-paths": {
"host-path": [
{
"first-lane": 3,
"config": {
"first-lane": 3,
"app-sel-code": 2, // the second application 100G to 100G
}
},
]
},
"network-path": {
"media-lanes": {
"media-lane": [
{
"index": 2,
"config": {
"index": 2,
"target-output-power": 0.0,
"frequency": 193100000,
}
}
]
}
}
}
{
"id": 4,
"config": {
"id": 4,
},
"host-paths": {
"host-path": [
{
"first-lane": 5,
"config": {
"first-lane": 5,
"app-sel-code": 2, // the second application 100G to 100G
}
},
]
},
"network-path": {
"media-lanes": {
"media-lane": [
{
"index": 3,
"config": {
"index": 3,
"target-output-power": 0.0,
"frequency": 193100000,
}
}
]
}
}
}
{
"id": 6,
"config": {
"id": 6,
},
"host-paths": {
"host-path": [
{
"first-lane": 7,
"config": {
"first-lane": 7,
"app-sel-code": 2, // the second application 100G to 100G
}
},
]
},
"network-path": {
"media-lanes": {
"media-lane": [
{
"index": 4,
"config": {
"index": 4,
"target-output-power": 0.0,
"frequency": 193100000,
}
}
]
}
}
}
]
}
}
]
}
} |
|
@noguchiko Minor comment: If I understand correctly, the values in |
|
@noguchiko Then, my opinion is to take the second approach of goldstone-transceiver based on「6 Core Management Features」. It might be too trivial, but just one comment. |
|
@toru-mano Thank you for pointing it out, that is correct. I fixed it. |
|
@KazuyaAnazawa Thank you for your comments.
If we do so, users can use I think we should design the primitive model with the following point made by @ishidawataru:
Therefore, I think it is better to use CMIS-defined units for the @ishidawataru What do you think about this? In relation to this, I noticed one thing. The frequency configuration is a combination of the following CMIS attributes:
However, my first proposal (with a single config leaf node |
|
@noguchiko @KazuyaAnazawa I think we don't need to think much about the compatibility with |
|
I'm designing the details of The 2nd option (based on the CMIS functional model) does not provide this control granularity. It is possible to add per-lane attributes to the model in order to provide the control capability, but this would make it difficult to define and maintain a consistent model. So, in order to avoid introducing unnecessary restrictions and to simplify the model, now I think that the 1st option (per-lane, based on the CMIS registers) is appropriate. What do you think about this? |
|
@noguchiko Thank you for noticing. I agree to take the 1st option. |
|
@noguchiko Sorry for the delayed response. As you noticed, CMIS attributes should be administrated per lane rather than per data path (host path and network path) because all electrical lanes on a platform sometimes do not have same capability (e.g., it stems from different length of each lane). |
Summary of this issue.
This issue discusses how to support coherent QSFP-DD (OpenZR+) module control on Goldstone.
Specifically, the way to configure
network-interface-modeandhost-interface-modeof OpenZR+ module is discussed.NOTE: Here,
network-interface-moderefers to a pair of modulation-format/line-rate/FEC for line side whilehost-interface-moderefers to port breakout and SerDes configurations of switch ASIC side including modulation, late, lane count, speed, etc).The host-interface-mode is defined at SFF-8024.
The target platforms are
AGR400 (Qumran2C)/CSR440 (Qumran2A)from Edgecore, andS9510-28DC (Qumran2A)from Ufispace.As the first step, we would like to focus on
AGR400 (Q2C)platform at this point.Components for OpenZR+ module control
HALandsouth-layerwhich control OpenZR+ modules.xcvrdfrom SONiC is introduced forHALwhilesouth-xcvris introduced forsouth-layer(see figure).south-xcvrhandles OpenZR+ module controls, it subscribesgoldstone-transpondermodel.HALandsouth-layerfor OpenZR+ modules but also for switch ASIC. So we need to introduceHALandsouth-layerwhich control Q2C/Q2A switch ASIC.containerized OcNOSis introduced for HAL whilesouth-ocnosis introduced forsouth-layer(see figure).south-xcvrhandles OpenZR+ module controls, it subscribesgoldstone-interfacemodel.containerized OcNOSshould be purchased from IPI whilesouth-ocnoswill be enabled for these platforms.Issues
Main issue for this work
network-interface-modeandhost-interface-modeshould be configured at the same time.host-interface-modebyOcNOS. Then, OpenZR+ module should be configured according to the specifiedhost-interface-modeandnetwork-interface-modebyxcvrd.south-xcvrandsouth-ocnosshould cooperatively work for OpenZR+ module configuration.network-interface-modeandhost-interface-mode., and for getting supported mode list of the modules.south-xcvrandsouth-ocnoscan cooperatively work.Another issue
Requirements & Implementation design (suggestion).
First, several requirements for implementation are described.
Then, implementation design that satisfies the requirements is proposed.
Requirements
The running datastore should be updated by the north daemons only.
south-xcvrshould not updategoldstone-interfaceforsouth-ocnos.south-ocnosshould not updategoldstone-transponderforsouth-xcvr.Each Goldstone primitive model should be subscribed and handled by only one south daemon.
south-xcvrshould not subscribe bothgoldstone-transponderandgoldstone-interfaceorgoldstone-ocnos.south-ocnos.There may be several port breakout or SerDes configurations for different ASIC or gearbox. We should not depend on specification of a specific platform (
AGR400 (Q2C)in this case).State transition of QSFP-DD modules should be cared. According to CMIS/C-CMIS specification, the module must receive a valid input signal while performing initialization activities in the subsequent DPInit state. Therefore,
OcNOSandxcvrdshould be configured in an appropriate order.Implementation direction (suggestion)
I would like to propose an implementation design. The whole picture is shown below.
network-interface-modeandhost-interface-modefrom north layer, we need a new model for it (refer to the model asgoldstone-mode).intermediate-layeris introduced. This component performs the followings.AGR400/Q2C, i.e., supported port breakout / SerDes configuration variations, viaplatform.json. The supported mode list should not be hardcoded.goldstone-mode. According to the changes on the model, it configuressouth-xcvrandsouth-ocnosusing handlers implemented inside them. The configuration contents forsouth-xcvrandsouth-ocnoswill be generated by referringplatform.json..network-interface-modeandhost-interface-modeare specified. Validation includes the followings.network-interface-modeandhost-interface-modeare supported by the platform.network-interface-modeandhost-interface-modeare already configured from north-layer. If such configurations already exist, it should be notified to operators (goldstone users) and aborted.OcNOS, and the configures QSFP-DD module side viaxcvrd.Tasks
network-interface-modeandhost-interface-mode, and getting supported mode list.platform.json.Additional information
Supported mode list for Q2C. Details are also available here.
400GAUI-8 C2M
400GBASE-KR8
400GBASE-CR8
200GAUI-4 C2M
200GBASE-KR4
200GBASE-CR4
100GAUI-2 C2M
100GBASE-KR2
100GBASE-CR2
CAUI-2 C2M
CAUI-4 C2M
100GBASE-KR4
100GBASE-CR8
100CAUI-4 C2M
@ishidawataru @noguchiko @ipi-claytonpascoal @matsuo-tat @iAgrawalGaurav @santanukar2000
Could you please check the proposal and can we discuss this here?
(FYI @hitoshiirino-hcontt @konomimochizuki @s-homma @youhei-katayama)
The text was updated successfully, but these errors were encountered: