From 2e1f699a3ba1dbf88abeecb192f6bb5e390b8b8a Mon Sep 17 00:00:00 2001 From: Arista Jenkins Date: Thu, 27 Aug 2020 23:00:47 -0700 Subject: [PATCH 01/12] [chassis] Add HLD for fabric port handling in chassis --- doc/chassis/fabric.md | 154 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 doc/chassis/fabric.md diff --git a/doc/chassis/fabric.md b/doc/chassis/fabric.md new file mode 100644 index 0000000000..b51759b07a --- /dev/null +++ b/doc/chassis/fabric.md @@ -0,0 +1,154 @@ +# Fabric port support on Sonic + +# High Level Design Document +#### Rev 1 + +# Table of Contents +* [List of Tables](#list-of-tables) +* [List of Figures](#list-of-figures) +* [Revision](#revision) +* [About this Manual](#about-this-manual) +* [Scope](#scope) +* [Definitions/Abbreviations](#definitionsabbreviations) +* [1 Requirements](#1-requirements) +* [2 Design](#2-design) +* [3 Testing](#3-testing) +* [4 Future Work](#4-future-work) + +# List of Tables +* [Table 1: Abbreviations](#definitionsabbreviations) + +# List of Figures + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|--------------------| +| 1 | Aug-28 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Initial Version | + +# About this Manual + +This document provides an overview of the SONiC support for fabric ports that are present in a VOQ-based chassis. These fabric ports are used to interconnect the forwarding Network Processing Units within the VOQ chassis. + +# Scope + +This document covers: + +- Bring up of fabric ports in a VOQ chassis. +- Monitoring the fabric ports in forwarding and fabric chips. + +This document builds on top of the VOQ chassis architecture discussed [here](https://github.com/Azure/SONiC/blob/90c1289eaf89a70939e7c81e042e261947c6a850/doc/chassis/architecture.md) and the multi-ASIC architecture discussed [here](https://github.com/Azure/SONiC/blob/2f320430c8199132c686c06b5431ab93a86fb98f/doc/multi_asic/SONiC_multi_asic_hld.md). + +# Definitions/Abbreviations + +| | | | +|------|--------------------|--------------------------------| +| SSI | Supervisor SONiC Instance | SONiC OS instance on a central supervisor module that controls a cluster of forwarding instances and the interconnection fabric. | +| NPU | Network Processing Unit | Refers to the forwarding engine on a device that is responsible for packet forwarding. | +| ASIC | Application Specific Integrated Circuit | In addition to NPUs, also includes fabric chips that could forward packets or cells. | +| cell | Fabric Data Units | The data units that traverse a cell-based chassis fabric. | + +# 1 Requirements + +Fabric ports are used in systems in which there are multiple NPUs required to be connected. Traffic passes from one front panel port in a NPU over a fabric network to one or multiple front panel ports on one or other NPUs. The fabric network is formed using fabric ASICs. Fabric links on the fabric network connect fabric ports on NPUs to fabric ports on fabric ASICs. + +High level requirements: + +- SONiC needs to form a fabric network among NPUs, monitor and manage it. Monitoring could include link statistics, error monitoring and reporting, etc. +- SONiC should be able to initialize fabric asics and manage them similar to how NPUs are managed - using syncd and sairedis calls. + +# 2 Design + +## 2.1 Fabric ASICs + +Fabric asics are used to form a fabric network for connecting NPUs. For each fabric port on NPU, there is a fabric link in the fabric network connecting to a fabric port on a fabric asic. There are typically multiple fabric links between a pair of (NPU, fabric asic) to balance traffic. We use the same approach to initializing and managing fabric asics as we are doing today for NPUs. A typical chassis implementation will be to manage all the fabric ASICs in a chassis from the control card or the Supervior Sonic Instance (SSI). We will leverage the work done in the multi-ASIC HLD and instantiate groups of containers for the fabric ASICs. + +For each fabric ASIC, there will be: + +- Database container +- Swss container +- Syncd container + +Unlike forwarding ASICs, fabric ASICs do not have any front panel ports, but only fabric ports. So all the front panel port related containers like lldp can be disabled for fabric ASICs. + +## 2.2 Database Schemas + +``` +DEVICE_METADATA: { + "switch_type": “fabric” + "switch_id": {{switch_id}} +} +``` + +The switch_id needs to be assigned to be unique for each fabric ASIC. The SAI VOQ specification recommends that this number be assigned to be different than the switch_id assigned to the NPUs in the chassis. + +Fabric port statistics will be stored in table FABRIC_PORT_TABLE in STATE_DB. Typically, the statistics about a fabric port includes: + +- Status: Up or down. +- Remote peer: Peer name, fabric port. +- Link errors: CRC errors. +- Counters: RX cells, TX cells, FEC (corrected, uncorrected). + +Fabric port statistics will be updated periodically, say, for every 10s. + +``` +STATE_DB:FABRIC_PORT_TABLE:{{fabric_port_name}} + "lane": {{number}} + "status": “up”|”down” + "remote_switch_id": {{number}} + "remote_lane": {{number}} + “rx_cells”: {{number}} + “tx_cells”: {{number}} + "crc_errors": {{number}} + “fec_errors”: {{number}} + “fec_errors_corrected”: {{number}} +``` + +Note that the FABRIC_PORT_TABLE will also have entries in the Linecard Sonic instances because there are fabric ports in each NPU as well. + +## 2.3 System Initialization + +As part of multi-ASIC support, /etc/sonic/generated_services.conf contains the list of services which will be created for each asic when the system boots up. This is read by systemd-sonic-generator to generate the service files for each container that needs to run. + +Since the fabric ASIC doesn’t need lldp, bgpd and teamd containers to run, systemd-sonic-generator will be modified to not start these services for the fabric ASICs. + +## 2.4 Orchagent + +Orchagent creates the switch using the SAI API similar to creating the switch for an NPU, except that the switch type will be fabric. When the ASIC is initialized, all the fabric ports are initialized by default. The fabric ports are a subtype of SAI Port object and it can be obtained by getting all the fabric port objects from SAI. Since there are no front panel ports on a fabric ASIC, port_config.ini will be empty and portsyncd will not run. + +On fabric ASICs, OrchDaemon will only monitor and manage fabric ports. It will not maintain cpu port and front panel port related ochres, such as PortsOrch, IntfsOrch, NeighborOrch, VnetOrch, QosOrch, TunnelOrch, and etc. To simplify the change, we will just create FabricOrchDaemon inheriting OrchDaemon for fabric ASICs and this will only run FabricPortsOrch. + +## 2.5 Fabric Ports in Forwarding ASICs + +When a forwarding ASIC is initialized, the fabric ports are initialized by default by SAI. Orchagent will run FabricPortsOrch in addition to all the other orchs that needs to be run to manage the forwarding ASIC. Fabric port monitoring and handling is identical to what happens on a Fabric ASIC. + +## 2.6 Cli commands + +``` +> show fabric counters [asic_name] [port_id] + +asic2 fabric port counter (number of fabric ports: 192) + +PORT RxCells TxCells Crc Fec Corrected +------------------------------------------------------------------------- + 0 : 71660578 2 0 0 0 + 1 : 71659798 1 0 0 213 + 2 : 0 1 0 0 167 + 3 : 0 2 0 0 193 +``` + +# 3 Testing + +Fabric port testing will rely on sonic-mgmt tests that can run on chassis hardware. + +- Test fabric port mapping: To verify the fabric mapping, we can inspect the remote switch ID that are saved in the STATE_DB and match that with the known chassis architecture. + +- Test traffic and counters: Send traffic through the chassis and verify traffic going through fabric ports via counters. + +# 4 Future Work + +- In this proposal, all fabric ports on fabric asics or NPUs that join to form the fabric network will be enabled even when there are no peer ports available. We could provide a config model for the platforms to express the expected fabric connectivity and turn off unnecessary fabric ports. + +- Fabric ports that do not have a peer port will show up as a ‘down’ port. Fabric ports that do have a peer port could also go ‘down’ and there is no current way to differentiate this from a fabric port that does not have a peer port. This can be detected if the config model can express the expected fabric connectivity. + +- Monitor, detect and disable fabric ports that consistently show errors. + From fd06e67297a5e4646fad657711cb23e97e0461fd Mon Sep 17 00:00:00 2001 From: Arista Jenkins Date: Tue, 1 Sep 2020 17:17:34 -0700 Subject: [PATCH 02/12] [chassis] Update fabric HLD with hotswap handling and some future work --- doc/chassis/fabric.md | 35 ++++++++++++++++++++++------------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/doc/chassis/fabric.md b/doc/chassis/fabric.md index b51759b07a..78181e5a6b 100644 --- a/doc/chassis/fabric.md +++ b/doc/chassis/fabric.md @@ -23,7 +23,8 @@ # Revision | Rev | Date | Author | Change Description | |:---:|:-----------:|:------------------:|--------------------| -| 1 | Aug-28 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Initial Version | +| 1 | Aug-28 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Initial Version | +| 1.1 | Sep-1 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Add hotswap handling | # About this Manual @@ -49,18 +50,18 @@ This document builds on top of the VOQ chassis architecture discussed [here](htt # 1 Requirements -Fabric ports are used in systems in which there are multiple NPUs required to be connected. Traffic passes from one front panel port in a NPU over a fabric network to one or multiple front panel ports on one or other NPUs. The fabric network is formed using fabric ASICs. Fabric links on the fabric network connect fabric ports on NPUs to fabric ports on fabric ASICs. +Fabric ports are used in systems in which there are multiple forwarding ASICs are required to be connected. Traffic passes from one front panel port in a forwarding ASIC over a fabric network to one or multiple front panel ports on one or other ASICs. The fabric network is formed using fabric ASICs. Fabric links on the fabric network connect fabric ports on forwarding ASICs to fabric ports on fabric ASICs. High level requirements: -- SONiC needs to form a fabric network among NPUs, monitor and manage it. Monitoring could include link statistics, error monitoring and reporting, etc. -- SONiC should be able to initialize fabric asics and manage them similar to how NPUs are managed - using syncd and sairedis calls. +- SONiC needs to form a fabric network among forwarding ASICs, monitor and manage it. Monitoring could include link statistics, error monitoring and reporting, etc. +- SONiC should be able to initialize fabric asics and manage them similar to how forwarding ASICs are managed - using syncd and sairedis calls. # 2 Design ## 2.1 Fabric ASICs -Fabric asics are used to form a fabric network for connecting NPUs. For each fabric port on NPU, there is a fabric link in the fabric network connecting to a fabric port on a fabric asic. There are typically multiple fabric links between a pair of (NPU, fabric asic) to balance traffic. We use the same approach to initializing and managing fabric asics as we are doing today for NPUs. A typical chassis implementation will be to manage all the fabric ASICs in a chassis from the control card or the Supervior Sonic Instance (SSI). We will leverage the work done in the multi-ASIC HLD and instantiate groups of containers for the fabric ASICs. +Fabric asics are used to form a fabric network for connecting forwarding ASICs. For each fabric port on a forwarding ASIC, there is a fabric link in the fabric network connecting to a fabric port on a fabric asic. There are typically multiple fabric links between a pair of (NPU, fabric asic) to balance traffic. We use the same approach to initializing and managing fabric asics as we are doing today for forwarding ASICs. A typical chassis implementation will be to manage all the fabric ASICs in a chassis from the control card or the Supervior Sonic Instance (SSI). We will leverage the work done in the multi-ASIC HLD and instantiate groups of containers for the fabric ASICs. For each fabric ASIC, there will be: @@ -79,7 +80,7 @@ DEVICE_METADATA: { } ``` -The switch_id needs to be assigned to be unique for each fabric ASIC. The SAI VOQ specification recommends that this number be assigned to be different than the switch_id assigned to the NPUs in the chassis. +The switch_id needs to be assigned to be unique for each fabric ASIC. The SAI VOQ specification recommends that this number be assigned to be different than the switch_id assigned to the forwarding ASICs in the chassis. Fabric port statistics will be stored in table FABRIC_PORT_TABLE in STATE_DB. Typically, the statistics about a fabric port includes: @@ -103,7 +104,7 @@ STATE_DB:FABRIC_PORT_TABLE:{{fabric_port_name}} “fec_errors_corrected”: {{number}} ``` -Note that the FABRIC_PORT_TABLE will also have entries in the Linecard Sonic instances because there are fabric ports in each NPU as well. +Note that the FABRIC_PORT_TABLE will also have entries in the Linecard Sonic instances because there are fabric ports in each forwarding ASIC as well. ## 2.3 System Initialization @@ -111,17 +112,21 @@ As part of multi-ASIC support, /etc/sonic/generated_services.conf contains the l Since the fabric ASIC doesn’t need lldp, bgpd and teamd containers to run, systemd-sonic-generator will be modified to not start these services for the fabric ASICs. -## 2.4 Orchagent +## 2.4 Fabric Card Hotswap -Orchagent creates the switch using the SAI API similar to creating the switch for an NPU, except that the switch type will be fabric. When the ASIC is initialized, all the fabric ports are initialized by default. The fabric ports are a subtype of SAI Port object and it can be obtained by getting all the fabric port objects from SAI. Since there are no front panel ports on a fabric ASIC, port_config.ini will be empty and portsyncd will not run. +PMON will be responsible for detecting card presence and hotswap events using the get_change_event API. A new systemd service will be responsbile for turning on/off the service files for the syncd, database and swss containers that manage each fabric ASIC. When the fabric card is removed, the containers that manage the fabric ASICs that are part of that fabric card will be stopped. These will be re-started when the fabric card is inserted later. -On fabric ASICs, OrchDaemon will only monitor and manage fabric ports. It will not maintain cpu port and front panel port related ochres, such as PortsOrch, IntfsOrch, NeighborOrch, VnetOrch, QosOrch, TunnelOrch, and etc. To simplify the change, we will just create FabricOrchDaemon inheriting OrchDaemon for fabric ASICs and this will only run FabricPortsOrch. +## 2.5 Orchagent -## 2.5 Fabric Ports in Forwarding ASICs +Orchagent creates the switch using the SAI API similar to creating the switch for a forwarding ASIC, except that the switch type will be fabric. When the ASIC is initialized, all the fabric ports are initialized by default. The fabric ports are a subtype of SAI Port object and it can be obtained by getting all the fabric port objects from SAI. Since there are no front panel ports on a fabric ASIC, port_config.ini will be empty and portsyncd will not run. + +On fabric ASICs, OrchDaemon will only monitor and manage fabric ports. It will not maintain cpu port and front panel port related ochres, such as PortsOrch, IntfsOrch, NeighborOrch, VnetOrch, QosOrch, TunnelOrch, and etc. To simplify the change, we will just create FabricOrchDaemon inheriting OrchDaemon for fabric ASICs and this will only run FabricPortsOrch, the module responsible for managing fabric ports. + +## 2.6 Fabric Ports in Forwarding ASICs When a forwarding ASIC is initialized, the fabric ports are initialized by default by SAI. Orchagent will run FabricPortsOrch in addition to all the other orchs that needs to be run to manage the forwarding ASIC. Fabric port monitoring and handling is identical to what happens on a Fabric ASIC. -## 2.6 Cli commands +## 2.7 Cli commands ``` > show fabric counters [asic_name] [port_id] @@ -136,6 +141,10 @@ PORT RxCells TxCells Crc Fec Corrected 3 : 0 2 0 0 193 ``` +### 2.8 Fabric Status + +In a later phase, a `show fabric status` command will be added to show the remote switch ID and link ID for each fabric link of an ASIC. This will be obtained from the SAI_PORT_ATTR_FABRIC_REACHABILITY port attribute of the fabric port. Note that for fabric links that do not have a link partner because of the configuration of the chassis, this will show the status as `down`. The status will also be `down` for fabric links that are down due to some other physical error. To identify links that are down due to error vs links that are not expected to be up because of the chassis connectivity, we need to build up a list of expected fabric connectivity for each ASIC. This can be computed ahead of time based on the vendor configuration and populated in the minigraph. This will be implemented in a later phase. + # 3 Testing Fabric port testing will rely on sonic-mgmt tests that can run on chassis hardware. @@ -146,7 +155,7 @@ Fabric port testing will rely on sonic-mgmt tests that can run on chassis hardwa # 4 Future Work -- In this proposal, all fabric ports on fabric asics or NPUs that join to form the fabric network will be enabled even when there are no peer ports available. We could provide a config model for the platforms to express the expected fabric connectivity and turn off unnecessary fabric ports. +- In this proposal, all fabric ports on fabric ASICs or forwarding ASICs that join to form the fabric network will be enabled even when there are no peer ports available. We could provide a config model for the platforms to express the expected fabric connectivity and turn off unnecessary fabric ports. - Fabric ports that do not have a peer port will show up as a ‘down’ port. Fabric ports that do have a peer port could also go ‘down’ and there is no current way to differentiate this from a fabric port that does not have a peer port. This can be detected if the config model can express the expected fabric connectivity. From 72f68a972d3e6c17ee04db20731eb77aae8f7ca9 Mon Sep 17 00:00:00 2001 From: Arista Jenkins Date: Tue, 1 Sep 2020 17:19:59 -0700 Subject: [PATCH 03/12] minor typo --- doc/chassis/fabric.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/chassis/fabric.md b/doc/chassis/fabric.md index 78181e5a6b..fee4042e29 100644 --- a/doc/chassis/fabric.md +++ b/doc/chassis/fabric.md @@ -141,7 +141,7 @@ PORT RxCells TxCells Crc Fec Corrected 3 : 0 2 0 0 193 ``` -### 2.8 Fabric Status +### 2.7.1 Fabric Status In a later phase, a `show fabric status` command will be added to show the remote switch ID and link ID for each fabric link of an ASIC. This will be obtained from the SAI_PORT_ATTR_FABRIC_REACHABILITY port attribute of the fabric port. Note that for fabric links that do not have a link partner because of the configuration of the chassis, this will show the status as `down`. The status will also be `down` for fabric links that are down due to some other physical error. To identify links that are down due to error vs links that are not expected to be up because of the chassis connectivity, we need to build up a list of expected fabric connectivity for each ASIC. This can be computed ahead of time based on the vendor configuration and populated in the minigraph. This will be implemented in a later phase. From ac9c04152c9bf576c99e22bfb04f2e385f14a6b5 Mon Sep 17 00:00:00 2001 From: Arista Jenkins Date: Tue, 20 Oct 2020 22:16:16 -0700 Subject: [PATCH 04/12] Move fabric.md to voq directory and update the contents to include more description of counters --- doc/{chassis => voq}/fabric.md | 69 ++++++++++++++++++++++------------ 1 file changed, 46 insertions(+), 23 deletions(-) rename doc/{chassis => voq}/fabric.md (73%) diff --git a/doc/chassis/fabric.md b/doc/voq/fabric.md similarity index 73% rename from doc/chassis/fabric.md rename to doc/voq/fabric.md index fee4042e29..f2b3372001 100644 --- a/doc/chassis/fabric.md +++ b/doc/voq/fabric.md @@ -25,6 +25,7 @@ |:---:|:-----------:|:------------------:|--------------------| | 1 | Aug-28 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Initial Version | | 1.1 | Sep-1 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Add hotswap handling | +| 2 | Oct-20 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Update counter information | # About this Manual @@ -37,7 +38,7 @@ This document covers: - Bring up of fabric ports in a VOQ chassis. - Monitoring the fabric ports in forwarding and fabric chips. -This document builds on top of the VOQ chassis architecture discussed [here](https://github.com/Azure/SONiC/blob/90c1289eaf89a70939e7c81e042e261947c6a850/doc/chassis/architecture.md) and the multi-ASIC architecture discussed [here](https://github.com/Azure/SONiC/blob/2f320430c8199132c686c06b5431ab93a86fb98f/doc/multi_asic/SONiC_multi_asic_hld.md). +This document builds on top of the VOQ chassis architecture discussed [here](https://github.com/Azure/SONiC/blob/master/doc/voq/architecture.md) and the multi-ASIC architecture discussed [here](https://github.com/Azure/SONiC/blob/2f320430c8199132c686c06b5431ab93a86fb98f/doc/multi_asic/SONiC_multi_asic_hld.md). # Definitions/Abbreviations @@ -61,7 +62,7 @@ High level requirements: ## 2.1 Fabric ASICs -Fabric asics are used to form a fabric network for connecting forwarding ASICs. For each fabric port on a forwarding ASIC, there is a fabric link in the fabric network connecting to a fabric port on a fabric asic. There are typically multiple fabric links between a pair of (NPU, fabric asic) to balance traffic. We use the same approach to initializing and managing fabric asics as we are doing today for forwarding ASICs. A typical chassis implementation will be to manage all the fabric ASICs in a chassis from the control card or the Supervior Sonic Instance (SSI). We will leverage the work done in the multi-ASIC HLD and instantiate groups of containers for the fabric ASICs. +Fabric asics are used to form a fabric network for connecting forwarding ASICs. For each fabric port on a forwarding ASIC, there is a fabric link in the fabric network connecting to a fabric port on a fabric ASIC. There are typically multiple fabric links between a pair of (NPU, fabric ASIC) to balance traffic. We use the same approach to initializing and managing fabric ASICs as we are doing today for forwarding ASICs. A typical chassis implementation will be to manage all the fabric ASICs in a chassis from the control card or the Supervior Sonic Instance (SSI). We will leverage the work done in the multi-ASIC HLD and instantiate groups of containers for the fabric ASICs. For each fabric ASIC, there will be: @@ -69,48 +70,70 @@ For each fabric ASIC, there will be: - Swss container - Syncd container -Unlike forwarding ASICs, fabric ASICs do not have any front panel ports, but only fabric ports. So all the front panel port related containers like lldp can be disabled for fabric ASICs. +Unlike forwarding ASICs, fabric ASICs do not have any front panel ports, but only fabric ports. So all the front panel port related containers like lldp, teamd and bgpd can be disabled for fabric ASICs. ## 2.2 Database Schemas ``` -DEVICE_METADATA: { +DEVICE_METADATA|localhost: { "switch_type": “fabric” "switch_id": {{switch_id}} } ``` -The switch_id needs to be assigned to be unique for each fabric ASIC. The SAI VOQ specification recommends that this number be assigned to be different than the switch_id assigned to the forwarding ASICs in the chassis. +Each fabric ASIC must be assigned a unique switch_id. The SAI VOQ specification recommends that this number be assigned to be different than the switch_id assigned to the forwarding ASICs in the chassis. -Fabric port statistics will be stored in table FABRIC_PORT_TABLE in STATE_DB. Typically, the statistics about a fabric port includes: +Fabric port status will be polled periodically and stored in table STATE_DB|FABRIC_PORT_TABLE. Typically, fabric port status about a fabric port includes: -- Status: Up or down. -- Remote peer: Peer name, fabric port. -- Link errors: CRC errors. -- Counters: RX cells, TX cells, FEC (corrected, uncorrected). - -Fabric port statistics will be updated periodically, say, for every 10s. +- Status: Up or down +- If port is down, we may have some more information indicating reason e.g. CRC or misaligned +- If port is up, we should know remote peer information including peer switch_id and peer fabric port. ``` STATE_DB:FABRIC_PORT_TABLE:{{fabric_port_name}} "lane": {{number}} - "status": “up”|”down” - "remote_switch_id": {{number}} - "remote_lane": {{number}} - “rx_cells”: {{number}} - “tx_cells”: {{number}} - "crc_errors": {{number}} - “fec_errors”: {{number}} - “fec_errors_corrected”: {{number}} + "status": “up|down” + "crc": “yes” # if status: down + "misaligned": “yes” # if status: down + "remote_switch_id": {{number}} # if status: up + "remote_lane": {{number}} # if status: up +``` + +Fabric port statistics include the following port counters: + +``` + SAI_PORT_STAT_IF_IN_OCTETS, + SAI_PORT_STAT_IF_IN_ERRORS, + SAI_PORT_STAT_IF_IN_FABRIC_DATA_UNITS, + SAI_PORT_STAT_IF_IN_FEC_CORRECTABLE_FRAMES, + SAI_PORT_STAT_IF_IN_FEC_NOT_CORRECTABLE_FRAMES, + SAI_PORT_STAT_IF_IN_FEC_SYMBOL_ERRORS, + SAI_PORT_STAT_IF_OUT_OCTETS, + SAI_PORT_STAT_IF_OUT_FABRIC_DATA_UNITS ``` -Note that the FABRIC_PORT_TABLE will also have entries in the Linecard Sonic instances because there are fabric ports in each forwarding ASIC as well. +FabricPortsOrch defines the port counters in FLEX_COUNTER_DB and syncd's existing FlexCounters thread periodically collects and saves these counters in COUNTER_DB. “show” cli commands read COUNTER_DB and display statistics information. + +Fabric port also has a couple of queue counters. Similar to the port counters, the queue counters are also polled with FLEX_COUNTER_DB. +``` + SAI_QUEUE_STAT_WATERMARK_LEVEL, + SAI_QUEUE_STAT_CURR_OCCUPANCY_BYTES, + SAI_QUEUE_STAT_CURR_OCCUPANCY_LEVEL +``` + +Note that Linecard Sonic instances will also have STATE_DB|FABRIC_PORT_TABLE as well as port/queue counters because there are fabric ports in forwarding ASICs as well. ## 2.3 System Initialization As part of multi-ASIC support, /etc/sonic/generated_services.conf contains the list of services which will be created for each asic when the system boots up. This is read by systemd-sonic-generator to generate the service files for each container that needs to run. -Since the fabric ASIC doesn’t need lldp, bgpd and teamd containers to run, systemd-sonic-generator will be modified to not start these services for the fabric ASICs. +Since the fabric ASIC doesn’t need lldp, bgpd and teamd containers to run, systemd-sonic-generator will be modified to not start these services for the fabric ASICs. A per-platform file called `asic_disabled_services` can list the services that are not needed for a given ASIC and systemd-sonic-generator will not generate the service files for these containers. For example, +``` +0,lldp,teamd,bgp +1,lldp,teamd,bgp +2,lldp,teamd,bgp +``` +will not start lldp, teamd and bgp containers for ASICs 0, 1 and 2. ## 2.4 Fabric Card Hotswap @@ -129,7 +152,7 @@ When a forwarding ASIC is initialized, the fabric ports are initialized by defau ## 2.7 Cli commands ``` -> show fabric counters [asic_name] [port_id] +> show fabric counters -n [port_id] asic2 fabric port counter (number of fabric ports: 192) From 53fb2fffd739fb6d96e780a6d13e903175e4fdb8 Mon Sep 17 00:00:00 2001 From: Arista Jenkins Date: Tue, 17 Nov 2020 21:14:11 -0800 Subject: [PATCH 05/12] Minor edit to add a note about container starts for fabric ASICs --- doc/voq/fabric.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/voq/fabric.md b/doc/voq/fabric.md index f2b3372001..2dc52fb286 100644 --- a/doc/voq/fabric.md +++ b/doc/voq/fabric.md @@ -26,6 +26,7 @@ | 1 | Aug-28 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Initial Version | | 1.1 | Sep-1 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Add hotswap handling | | 2 | Oct-20 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Update counter information | +| 2.1 | Nov-17 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Minor update on container starts | # About this Manual @@ -135,6 +136,8 @@ Since the fabric ASIC doesn’t need lldp, bgpd and teamd containers to run, sys ``` will not start lldp, teamd and bgp containers for ASICs 0, 1 and 2. +NOTE: Longer term, we would like to use the FEATURE table to control which containers need to be started for fabric chips. However, that requires multi-ASIC support for the FEATURE table. This will be pursued as a separate project. + ## 2.4 Fabric Card Hotswap PMON will be responsible for detecting card presence and hotswap events using the get_change_event API. A new systemd service will be responsbile for turning on/off the service files for the syncd, database and swss containers that manage each fabric ASIC. When the fabric card is removed, the containers that manage the fabric ASICs that are part of that fabric card will be stopped. These will be re-started when the fabric card is inserted later. From dc021dc4cb7f9159109181e4b15179146c9fa1d0 Mon Sep 17 00:00:00 2001 From: Arista Jenkins Date: Tue, 1 Dec 2020 17:16:35 -0800 Subject: [PATCH 06/12] Add Everflow for VOQ chassis HLD v1 --- doc/voq/everflow.md | 86 ++++++++++++++++++++++++++++++++++++ images/voq/Everflow_voq.png | Bin 0 -> 27807 bytes 2 files changed, 86 insertions(+) create mode 100644 doc/voq/everflow.md create mode 100644 images/voq/Everflow_voq.png diff --git a/doc/voq/everflow.md b/doc/voq/everflow.md new file mode 100644 index 0000000000..fbf2060b58 --- /dev/null +++ b/doc/voq/everflow.md @@ -0,0 +1,86 @@ +# Everflow support on VOQ Chassis + +# High Level Design Document +#### Rev 1 + +# Table of Contents +* [List of Tables](#list-of-tables) +* [List of Figures](#list-of-figures) +* [Revision](#revision) +* [About this Manual](#about-this-manual) +* [Scope](#scope) +* [Definitions/Abbreviations](#definitionsabbreviations) +* [1 Requirements](#1-requirements) +* [2 Design](#2-design) +* [3 Testing](#3-testing) +* [4 Future Work](#4-future-work) + +# List of Tables +* [Table 1: Abbreviations](#definitionsabbreviations) + +# List of Figures + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|--------------------| +| 1 | Dec-1 2020 | Song Yuan, Eswaran Baskaran (Arista Networks) | Initial Version | + +# About this Manual + +This document provides an overview of the SONiC support for everflow configuration in a VOQ-based chassis. In a VOQ-based chassis, the everflow configuration is applied on a linecard and the mirror destination could be in a different linecard. We propose a solution where the packet is rewritten with the GRE header in the ingress linecard before it’s sent over to the mirror destination. + +# Scope + +This document builds on top of the VOQ chassis architecture discussed [here](https://github.com/Azure/SONiC/blob/master/doc/voq/architecture.md) and the Everflow design discussed [here](https://github.com/Azure/SONiC/wiki/Everflow-High-Level-Design#3138-mirror-api). The goal of this document is to describe the configuration and design for everflow mirroring sessions for a VOQ-based chassis. + +# Definitions/Abbreviations + +| | | | +|------|--------------------|--------------------------------| +| FSI | Forwarding SONiC Instance | SONiC OS instance on a linecard module that controls one or more forwarding ASICs. | +| ASIC | Application Specific Integrated Circuit | Refers to the forwarding engine on a device that is responsible for packet forwarding. | +| ERSPAN | Encapsulated Remote Switched Port ANalyzer | Another name for Everflow mirroring sessions | + +# 1 Requirements + +In a VOQ based chassis, the mirror source and destination ports could be on different linecards. This configuration must be accepted and the packet that goes out of the destination port must have the correct encapsulation header based on the configuration. + +# 2 Design + +## 2.1 Problem + +In a regular non-VOQ switch, given the destination IP of the mirror session, mirrororch is responsible for determining the outgoing interface and the destination MAC address by querying routeorch and neighorch. Once these details are determined, mirrororch makes the SAI create_mirror_session API call to create the mirror session. For example, the destination port is set in `SAI_MIRROR_SESSION_ATTR_MONITOR_PORT`, the destination mac for the packet is set in `SAI_MIRROR_SESSION_ATTR_DST_MAC_ADDRESS` and so on. + +In addition, mirrororch is also responsible for reacting to changes in the route or nexthop and update the session by making appropriate SAI update calls. + +In a VOQ chassis, the monitor port could be on another linecard and packet rewrite is done in the egress pipeline of that linecard. As the ports of other linecards are only represented as SYSTEM_PORTs in each FSI, we need either changes in SAI or changes in SONiC to support everflow. + +### 2.1.1 Proposal + +The idea is to fully rewrite the packet in the ingress chip and send it out of a recycle port in the ingress ASIC. After the recycle, the packet goes to the egress ASIC where the packet gets switched out of the mirror destination port. + +![](../../images/voq/Everflow_voq.png) + +The mechanism to enable the recycle port is outside the scope of this document. + +## 2.2 Option 1 - Implicit Recycle + +This option can be implemented with minimal changes to SONiC because the bulk of the handling is done inside SAI. The SAI implementation can be enhanced to accept SYSTEM_PORTs as arguments for the `SAI_MIRROR_SESSION_ATTR_MONITOR_PORT` attribute. When a mirror session is created with a destination IP, mirrororch resolves the IP using routeorch and neighorch as before. The following changes are proposed for the VOQ chassis. +Upon receiving mirrororch’s request for a remote neighbor’s information, neighorch must be able to return neighbor information like interface alias. +If the neighbor information returned from neighorch turns out to be a remote neighbor, mirrororch needs to use the SYSTEM_PORT of the remote interface as the mirror destination port. + +SAI implementation will program the hardware to send the mirrored packets in this session out of the recycle port. When the packet re-enters the pipeline, the destination MAC of this packet will be the neighbor’s destination MAC address and it will be bridged in the ingress ASIC to the correct egress destination port. SAI will also be responsible for programming the FDB entry in the ingress ASIC appropriately. + +## 2.3 Option 2 - Explicit Recycle + +This option can be implemented with minimal changes to SAI as long as a recycle port can be created. The recycle port needs to be enabled as an L3 port for this mechanism to work because it requires the packet to be routed after recycle. The following changes are proposed. +Upon receiving mirrororch’s request for a remote neighbor’s information, neighorch must be able to return neighbor information like interface alias. +If the neighbor information returned from neighorch turns out to be a remote neighbor, mirrororch needs to use the recycle port as the mirror destination port and the router mac as the destination MAC address. + +SAI implementation is the same as a single-chip system. The mirrored packets are rewritten in the ingress ASIC and sent out of the recycle port. When the packet re-enters the pipeline, it gets routed because the destination MAC is the router MAC. The destination IP of this packet is the destination IP configured on the mirror session, so it will be routed by the ingress ASIC. If the destination IP is reachable via multiple nexthops, the ingress ASIC will loadbalance the traffic as needed. + +# 3 Testing +TBD + +# 4 Future Work +TBD diff --git a/images/voq/Everflow_voq.png b/images/voq/Everflow_voq.png new file mode 100644 index 0000000000000000000000000000000000000000..6965437c0383b8d8f4a768e0043c3e6b6d1b2a54 GIT binary patch literal 27807 zcmeGERX`nE6E%tgfdGL30fGc~_uwuG?zV7ucXx;2?(XjH?(Xic!SyV*WPkrT=RV(u zd!b?V>N#un^sX8;s)M8?1V4ZJ_6Y<87lfD8x-SO5qJs0h?Y;1!GDo3Fqxyhi-| zQo{WFxKh>@`bMUDARwfXI+~h@LX@NfT3VW#1EUmVpR65Z0t0`^XnJ&YbaxDNbZYi! z#w4h!ZlED-e30t`$(HV{H{q$}u~b6UjRR$mAHUOM2&0qO_|#7CG4)yOhSZaSsFnp? z=nZr$XU97W@#Duna9owfmK=~CbI__E!Jc7YRR*35hX}Xd5ZIx+xIh$;5O-j8H9)H% zq8JcmJx@q@>Ld`O9N7>tjA@`O!O;?R559dvup~dsAgs*-7hvBg8p(^!AqCjgGd!h>rHE{`U5EcMJSS4eX<~ z-VVzR3=rd=>Ty$O_cOpg)z$+D>x+wnkOR-5K)^r|K|TP_K!Lv?0Y)I;f1QJX5CcC! zKt4qIgFpg55rKbF8DRgWf(2xJ`1c%C+=k7O6aTLqY{2vP%QX17 ze`T>X;lu}sOX2ccSnJ_3Qh%qW#pn8ji;K%)t*g%_Bk=Pdao{&jd_!AXOEwxB2L}gg z2L@^jYXcfOR#sLTT6!9KdMaQJDjO$rTP;T_a~p!c3;DMk0X-WXYa>frBMWof_j0wg zE$nPL@$uhp^ylaAdg?hE{dXsGn}3=GY#_~h3Jo1KEzSR%n5~ii{|B-6l)uIPYS-Vl z<9IKOP2NV&n%~0AOwZhw>%Y$B`0GypN8*3={991U$WhM}AYcTPv;nq>i|IS_KT`kq zl>b*rrT-)u7-;`j@_$nPNAkT1Y?9VSz(i{(fsQw za5%X>anSsk8eE_39X3!wKzKle1$gBgK~GYlQ$lBP`aJND7aa6Q0R3YCdi-Wm--5s_ z(wtYJte6NsX#qW|V7@s%-FUu%w)5qM`hl_MaU0t*hqD!S{YIO8>ha6QwL8vprS|<) zm-IV_I^D=2Xe{UmkpB)+F!)HY|Gol~(7=!b{K3NsdBFZ0HJDfExS)uCjs{83xb7!P z$DjXYahLSmnmES+1^bf;*Xs$K6z;rw1O30^?>Rw(`P)6fp*+Eb{yol+GFV#1y$$}{ zjd0@*DERXF1sCN1xaHOb7ff2@pSpXl0|l?21U*6i6eYC51seSS3l$TxdYfS$@1jtQ zizbnL)ZYbAAX@Teh$Ye{GPvYLH(wKj_Jflv6cHLoq}#hjw-Ih+gAc`1&@kwW-Ayj_ zmMe47z~RB>Q>`L%v;F!3rY@4mQB2A#rUC8J-q5Eb)7bB%Dr9*oL3ppjEj zay(*nC~b|866e3cY={Ku&BV}Rp!jc;E^L95=l`GW|N9+6#?k7zJERyRn=jehoZw)- z9!D%M+Q;5jOn?GQMa|zYX=7I<7XHs7bdBOk&Mz9w{k@M=YVL5p7TLo4pqloY9ZNAR zIi1(Uq2@q%aJhAH(#Sw>va8@wVJV88Mku!`qovRH&jtXE3!=s1GFu0p6Y*U-y<7#) zh~A#J>_ovHRDhWGzfGcz54cGvDo5S?w@HAaB3PQ#UsdY#>uz@WRavZay^yKYm0X{# zswnDg^#o1TnJpX252Y}weCKxl-;Ik94zykgPO(&($nJP9Ry>(WSV$<)_H0FIsm4gB z`0E$}0YSDK7}(Te*L~OEDSN%{_7eBb!R80+?MFt90>tA#+uH^|_=-NO>tSxZ?nDOf zREbixKqq9So~r+^L?WHMkv;k`!N59?a@9D>Me82cGavkw7I%Jsx9hMpC%e}!Cuk}K zzCH!;W7%9@#Lr=&WPkIPT@SiHRjX)hW2hBm$2gZjaFSW$LQIITnT z@J>N6k$D@^Xl2X&u73E_F>YqY1{l+&L95%-92KjpW;hSWw|e8ja(bg{%f{TZ&L%*G zkQJFsOAIoNthz|d&D!DbJa>+$zaI=ds!%*=%61(oWJr1^7hYGh@HTez_f(u;tmH0K zFrrmxO^U?sjRPOuTG3@`)7$pz=mIhfYthQoniaabx^kqmf70vp!8W?TIA?ir+0gS< z1ny11I|~gvZ&n?|qthR(Qqx}^XV!5f&y(l(gMHK;$??N8mn)KyWh!}DMIkPf&}^hc zQ^p_q@Hyn5rPgI9Ki6ZpSZzJFBjWk$Zb|-1D_laR*_2D^uKy~BoccG>-}AzYki>F- z%uEISB6;C)P*4x>rheu6V#!)#5nfZEBCUz%%Wt82{89eqVYXONx984=(2eJ#+iCb) zV$on^E1D&W(c`5mR+}qdTMv(n zI>fNItoRw{tjfBfPUrho#dP2-(1?(HY$<%Xm;m2{nM4wp%4xJn9gZO?KAPh+5}ihp zq^+ZGamIR~+wFE7lq(uuY{)N;k5qN?$2~2!}!PLo{acW7g=g;_$X4V|?Q@iD>*9&hSb)vG7aZ_(6qa zz45+_W%a9p7_fxt;Za)(L_mX&S$8=sD7;4 zTnRqnIsxI4P}D=PDM56ShnE)ls~H9w!(r`^eu_bpapoBhgcPdNzMonOf2$@zBDfBk z5j>E#ZWb9Ecde%;7K{E$YOniIDE_28t3O9;xD0dLQ5Bz=B;$>GyEJwN^HqqAA6m#R z3BLv?NC7JxMXlB}aNhn#&F*j(y`a0baKpzWAmiHCS!?rz2_JxcPzn`az|BR1gpz znqDbDUc9N;&3#7oIjKHMI{j9*HK^83KHd5D2l}-;^XxH8mtxUbl(S&zeI%84{H4b&^`dP7`CjSj>#<1g2y+(0;@bi?%y(1RNF$%QOZ@OQ8mYT2hs|Eih-zhTxba948~#L&v9RFU zH34WiNd$9_C&y&Xj~MH=g-h{9xBW0oc2_B9cuYKQH}0SAQtjt<-YT}v(%T&!TUdUX@wcoMd(>pi5g8iKT9i=whMH;7`Z!HD%|0$QUBfhK z(ml=NsKuSI=B!TF-hiwH%NySlfXyZT_3-Y1f$oqoY!~?K3|`v4e|wCbeRrZ*YbtKW zsP$FrG*wA)VQn3!j8lIkvso)$Vau}fq;P7#JyGmzY<>Ew8|P)gv z^Njrq0&M>%Jr|5%fGYptITmyI_OAh9Ruib;v?!OqE%Hsg61ZfE3uUUYJ00=7$mNUr zoTCPV0--h?8E8y(QBfb}PF<>5)lw*<2|y7q=z)r)`kdw&IW;&w3jFEQUNODd;KxpY z7~46%<^6%Q1pQ?hvFO#*6$IX3vL^&CXgKahy$&3wcUu4|ShP%qCzD@h05fuk;6K_2 zHQ~l(Tz3FrVod=S$~k#@yCv^bUhVROfVi39-5}PD@T-)ya3p>!_+i*_-;B>U4b)FIrcE%|z`wwZZ{oC4gr=K_GF58! z;{Lr(%Lp!hr?acYC-}r@da_w=c>1uv9`DYt4joYUise>LqYf4r!4Qfs*;;F^zXVK} z)^4|l_rE0N3!~x^d~9=A_xaC=;u7xIrV2RA3hQM!MV&gExe|5)oX!Hy3k`DIqYWTb z#vh7JrYn=zLeQV&luf*uX1zGmC zKBNve{Dk5S-r^0?s|7py-Fj-g%J57~iNnF)^+8yJ3`br5RCtB2XvT46)A)#vaQCJj z_gA{vGB2^~OQLHL)AxKEZcwQ1#p1d~Zk@e@8*oIMmbm$DEeN0im-Xf1*nHRU`w0hb zV8z>31vG=+kcCOEt)FG${j8MHy&)YsYeBf0;d;C2YgciVk{N4YDdgcF%g;3$SS8Bk zuhsUIG~3M8aYf@%q%iyzixofTUMe}_$eb)_e@&LC*L#g5(RaE~bVf!0Ye)77d2Z4E zNx+az} zB6t7QBT0Z3a;m3r0kiMcu*G%Eq^w6RbfMZZv*@&O?rtdRw_!;nLD8PdZs<|}OW}Vl zZJ~}_pqN@P!(LBt94SGyf_daeNp70t`SQ5<W_(jj{+(o zB)oOo0}k(cvii2fL4}r%b~}*mJZUx(cAJBx{0W5$@5$aA3Bdn9BN+g+Dg)|v#qX>HdQtRA;Drhu2`#}Z^Qb5^V8Kx9&6a;oKa zY;mcktf58g-MHTd63zHzd4CM14PM}IaTd@a${?S-sMbp}yzsIe_^;KRNe<>xy7qlO zy{l7|!mlzcq}5X^!Ot|Nhi)q0-nQO{j{1!{)=xsd{ae`pOw~< z4jiAmdaV)@Pc@YbIye(D^Ti4Ud(%Z+69wwIt%;SVg8gF|+?s?Ksje?5KLv*mzMD_> zf$vBDnM{Jf$@D11-D`TC3hUn_FM>Ah3R#iLpT5JF2eP7!l47e4P!q^VV@DWTU}tYtKpxY#zyVOM8G7ir(G1%(lKBz+;S~6Y@v{Djmy&=p};*bh7$Lse*s(e0V}&NnW8_R6m;B zx0{g3q?KvrRx6x}NqhUw$QEXaSUm8O-x@YQ=q>foMKO~Cs`iHnoMz}8G`R+m&`O^9 zavT$_;rf}D5I3Oz#aDQ;fbf=>ZBJWrr=N`hM0FNcqjx3|I8O2LYbS~Kc&3}n&i$kh zVkYG;!I?8^kCR2RdHJ_ggTN%S_KdyG7s0_>p6WPmkhtou>_6EorV-kAk*)G+* z`<1cNT$sJG1KZP^`&YuL%#C- z@63QG(LPEgLcJlmC^8d!;aK`y-H{aesT^^VLk9hrYjk?yr5eyEGEs$>D4$v$2Hg=c zlg3gAM<7D5K?TmCj;T=gT#&-{^(~c!TD^r-yeQ7r$p!fzEW)X2Eedx|8g?K8PrbJ` zo$-`3Qh&U?#A8D;m7}ZiG{uK{(EPWQnBAwMnM8Pzq0=maR-G%oB#6Ws?u%I;Z zK2oB5%%)J!nO=8@5bz;t>+Yr5p&&kJ^^s^xw}+0^Z7V^!`1h|ASsLQS>irnX zMH`_#%0)u{J<2HgTARqN+!~&4m>*ZjWY@#s#%Vu-SPf|CEAec8Qayo-vd%%bI`0On z0=4}|)tKTp^~jL@@j~k#18Z0+yM3d{)|V^jK)^N3U7=k4Me%H>xarZ8=XuI9K%-kUaRnbe3P@likJhLE^Mnc_O8cN05&tuaRpt z?zPWPdm`4FjPD)3K@uK*(-$y_8TM(Np9f_++J59iK zrG$6~HCW_IF%gFLY0D*^J2G~*BL2LhMjB8GAk8w#bTB1eD-HHrwvAO=Hg&;^@OGo^q^O$mzm1qQ&nE) z=778V-2l+WoLQL0C6g(sQRgeXjTXNKC}>pPcaa|$CiHp;U4w!luI?a&=i}ka*R#xF zzZ^bQ9zAeS!x#V9r~An)1>J{GJ0chHZCE8ks*?0LPI-Ep^?f-L`T~^`zJmuVRRA~8 zPTiFUA887!B>S$fmz=m*AM3~p1SbT2NZYb0gPmd-hu2P}SA*xv)F;>UM=*R1hvV6n zr<~72>#7XbOo^~<3UHod*i1bUBaEhUCim6WF-VDq)?Ah`8BTN>onD%#l7V{L(8%Dn z+5F8nZWE-`1A9SKXOUXiNGoB^wb7Hi&>wRS#fhg+_Ga;(*XS`26*Ws;l_hdz4-LqQ zDy-j%Q^@MMvGH(6gF(WM!p+TWD{_?o zF|fYm;6OVcKgG6IUszEx61OLu3E0#SdzBeZ zW0~DJ*m7z1fTPp$fvXiBJo@|09%Ma62KeX=u#{jx(`kZltqur1xQw3f)y)*l4-mt( zKh~71HrtDrsWurldnkt5vF8%&?kH?8iPaCseKyIFjK8g;C7a0!9x(>?2IvFK`|Lu+lEVx)d*DdS+BzOAbFX!ug0aHLm83N4n8u&#r0cn zh9`=e(J`V-HanabU3$5I&y9+EycE`HKC8$SpIyTmYa>u5R6ym4$4e;5lpj&ca8BQ~ zGg8l&DigPlH8Beq+rp*TUQ09sO6IYRn^ z%_;N8y0(6FmxO7mbEWwC&8`RP(mpDRlUwgu7tA7#%w_7!M7aSt?rPkZWjF0yp>w2= z4*SA!Xf)hzVq?hVMVjo6i9<1%?VA}Rw`kj_1{?`D{J<~x4cyVhFHg%ux!o^D!nvB} z>Va^JGW}i(ichnH>10-xZKKFk&8{d@aYhiv^cSNfTb)khuruEn72|W)8gIups|J3vT~?!O+9%w)<6wD{ zrJl^2-ENc)>nnk2l)84W!{|QfE_P>O-WYiQu({)gt>W3f7SKp2k_DC7 zVJWjw;ZI}4%y|m@;@+Q-<0J*&GNQd$-0tl5pB7b0UmYLNm^M;L7n6e=F9gcr(;!${ zq(&0ydDGb+%OAHxO*z52#WAUX&&s%F2XppMS=G@?Lzy#g$=J1P$9Nj50_=rcKzpg~VASE3o(#?CJzzBgk3UDL+NG6-)A>5$ za5|rY;!XU9)1_R|=ya)eD(eT#U?M}lm~?fT$xMM(Xj;MkAbzaq3?^cNV_QT7U{s-w zOLdNN^g|%?44Cl)I>6+jjzwDvXR*~RS~;3BxyE_-b1PN~udEF%u*P&oFAlMIJE5`_45 zxab7|zs*}YJUP$*2%|5o1L{KgRj$Agnvv_f*m=Hsi>=f0m9)F>*sgqUNSs}&!jsn5 zTl8<8=%}K_N}wHFR5oU{HQ$OUogvOhCk^I~zH61nDFFIYp^bd|5iR97ho$V@BH2pe9~OHU-wslGHhOspCq1f0K|>ka0&foR1p08Nv_`PS@T>9sw# zbErh1>H-86cHPRYEOs>**B8cM_m@h)pJX0H-R8(mh8S;tVgs!~nS3pVLuS*K(~y;7 zioq^H0uU2=O51zayPN&1(UM#?*jw54CihUytl8&8?xM@$FUF!Ht}-$&1*jBi5-RDm zd6CrS;3ZuMbfsM+c`*|bw>lE3e*NdFrrmra=)C)soZD|1N_WJVVSPs?H#YX>Gy^Gq zKJA?40a2XDvX;x7fEg`LH)$p2(yq;T9$wpPFfx2EZMi%H;Y?^3 z8FfZf&4@TtuJAa}7M~mk(_V~gc;0+M$1X3TZhKZJ&UJiMZ4;W1SLF!}=}x}({IY(u zSgkIPjxv`EeqtcGy{i@COxJVrTYea2sHdEJT5YC+TKgj#WymfCci)eFwWUvDBeCEW zaRmgSASOHdwYQ|0efRjVlr1kVwoy%zO!DIoM2aW`eKq*__$%f_wsN@Xhyl$G0vePU zp}$T`XgAmd&p5n0)8u&loYZu`zFie1sq*}Vr z?ApBDU>gIJABs6x@!wrVFeQ>{m+I_OygaSE?)mt)=iQFbmp^zup|Q_b%bFC<-=ixF z!&X3I(y9MWSZeJucJ|;wUX#!Ua~yk9y-hB;KSfn)*N;_CuL?%=#(~2b$2lW3$FLa3 zUXB0DxR*seNqxOrTGu=7v#UEZENKUjmt@7rcvqVC3bG1xc5F?Sf={q}t5ubZm4tZ` zrEg47_zSb4F|g_{@x4sxd_Jey=OThzf7_u5Al4BK4Wg;2S2x3pjn%7sf=p-m#K#Q@ z?*4G-(%VpH<^&1Jd$0=%uK6Xv&T@C_!5P%p_GMP?7{6_+5ce;P!sa8|MzR`RB`UOGdFNm3t@j%?8$t423%mxyUWr%Q_dSA~8u6&? zcCsO(xD+(Y=gNg`H*A%~@{ z9r&A@I<|LDU5I;P8Kv)ULcY@-M1{(#?~ad7=+`$nq$oBP#i+x_tKgby?;x645Rz-S zrv7(-<`M;lO*-hkxzc2r27oXp1=~qvkxbDFfwca$Gc#!1S~65dey%P6Vm0=;{<*AO zv{GOOXV@c~w*)=niUKvpv^;;;Q|lp9S+1Uy=<^`@sNrM0y8IrV!#7M z{n8;rKKC-B)!JDEbr9UQHGi--97m9^Zh)kNLPxld+d`KVrlT`>GDu?F-}DTzj89WMAQFO!asZ)An+0jImXa3oKtO=ep*aMLOrafJxx%Xq!3dOGgHq8+m7N>V{aUruszZ#P!l`mFCais%Jgpj}C+*BN82m^>2 zIrCExX`%0nB^Hpd{7SAQg%;i9R#Jh;bc?53i+C}rOL?nt-kSdo^i*3PY;;2EBPY<@ zdueeHdA^uqq16Ybx-pfjP{Hg!=Q!9xQ-$?54j<4E;NBq9lZWdJ)Qz( z63KQu8ezHzdulT_;g+nYQXks>JNk>spAeXmSYv_If!5*6s|;>etj4PMc6HSnA>+COZs926Z}Yx?}{kz?!1Iz`=9#C7Mi~ zy0_&5eE}Wjcu8(YQThU_ncm@;=qh7{@AE3x&4RvP!@NBEL%N$C&P0Gvuyz68Nd4JL z8x=ff496bNWkWKZXq;ouRTwFZzId#b@o@%xKxno2HYFMSWSAB!Cv7BlK`j zth1H%p1Q~%I0)uN9hs<$Nc$s)Df5?T-tUQ9spLX|Z$Ztb%UidMyOP z=~vF>uXHlLntd=S)q28V*j%{0oe-FLpvM$jH;;~vHBUmQuB}>T95G*uK{2^)dCXT@ z@b-a*-VJLpaU_typ_Z=0w@`|4H#VNam8)N}UN`%Y7i@Fb1tI-gETw6s(VD$=f1G*5 zIfP2%tb`6QucG?hAx_4x-%)CMtmvr(F};+s~s9VO=iA1L$`pxS3R%rqTUZ* zcjC9+gO)={x0|v!FifO#O3~z$wN=#lc>JWcL{@35-rGzRE>~DKg>`(ZRE&V0Kf6CO zxNUtlBg|^I)llcJ{blVPDFA|t;Nydh)docQh~`GqdGh_LN&I!Di{E`*jje^QY(XDe zjQyrWo{6Gk;PWFkHlF4Jk%yMPprIL7uyOkP`m(h4W86V+=ykid{wS$!(A(CiD>I}d z`C2qL>sJBmlJm*n$9jK10}ehAA>|K6UoYxtV)W>PzYChK zrZovR1yqY%Q4}ksk|YbgZEn9PAY`dH?PPFCSD=e-bvQH^?4Qbe^?p2Ywl^i3CJC($ z>UMLnFBCubh|Myl#3Me|{Gxjj1?q6L18sDF0yA}-tp$WZv8mw1i8rIqJ^IV@|foNF>^i~u>iCe(kdF-|z{NN=2({4kRCEo8T><;2c1PerTgEK zSMx=?N04~w*d+7WZ&LQ0g~W~Sr%82jtj1WBp}wV*biXz)URSLJ?V{J@SBLxLihO6}k z0hls;HyPvvJ!jCht#f30U&La`W5LOW)xDk}h&)~q3YTgkef?1;v%(eFLe~x67(@>e zg5({cz3i;|<33kuj>$b7O}8C6wPoK0!%-wcCvCR>%c28YYf_7y-sy7qAY0!s-^qr<)e zXSefunVB^o6^-xZnBS+baeP05g(|at>|VxeO!V{leG=yBXgyn7$Qh+=6Yq$$`!XzR zw%(u)(^t427R7sRKg<8|lRt{$>f~;$CS`Fyoeb^cZT048El^e2n+q{m)cS&+6=wyg zwhjW)EZ9{P)j6VnhZq8J=}eTl)I10+&nk6Jg?-_Nt!7q4nSr<46){5}{=m-|6DnL7 z-10>;CHc`9OXp^+=vE_f*PA|1ru!_c&i5N13*9LYNS*|xFA zU&=}`%P(YkB#rx2M8zf8bTsh$(RegDP8M+n zP44m}P&TWx^A#Fa|GF-E)R6tr8Iv!0Vu>V4Uw)9Egp;@rDgMy-?ZgENW}f5Bh(_nc0}>j$3+o|P9Yk?N6%A}fJ_F2J4y3??CWZfeEFOq z`fdC%R}_`dVa-)0Lr9tokdqKBD;G>Ld0_gpeHdm4T$l&=guSlr*X*^h$u$~|UkE+P zSuOQO-~}p-JPqlUNB}xV1G&}HF-+n_j$-0$Tk~0=DM;Oj6d-MB5$3cON-*Ubok673 zS#mIdlMPz`Zf_t}@qEgsOxM-Am|DMhI0oqIWR1@?Zp@Go5i0|5Na{+~58USSZqMSr}tqJvaNJ1!uDR+w9PM;gbHe1bdDg-{K>VTo~2VVLB{i*pM zco61{E^7Kqe+T-h8(s2an&COJqMOKyUFBkh zGBSlz1VF+M`YP242|!gr{G^n_rWYkXx8GM)uwx{m{qE$vky-Y2!=1G}&g>KH5Xr&k zuBEI87=Mw#Dz-hv)!DGVy{0mOwrIq8)?aIJHJCpQwq&^$lue&eee`B(AY8j^YD(;6 zfPlAxj9OfpIbMtNjfPX3dxP$1nymFMbks@X?*by(D-j7jrQzJBOxnG%U$xb8v&Ry2 z<`)%+u4oR9a&BS8m;MWMyAf4sD{XcvpO7gFdNo3av}Cf}Ox(=pK3+EsRdo;|h=L`w zcuN9u$)&YBrW_5|%~=b$Dp;tK*!jbXm2SR#=_e8#Q!NT%6S^vdR>yNT)e&awtE5$w z4djbJ4TExQ*{yKx72E-?g|H!PiuPn?^8)jn0G+Q))Lyj5rU&;%RzX{H6@V=L4fZ!# zpWknhTRBCc=|GfNaD(lqac=m(Xc|sn~I7nGDar?KB{Hw z+ZkxqV*42uhrJr?;uux4uR$-`vV@YLv!wQybZey`o^jZZxdyq!Q*OC;uW3V)K8$L~R*3#$mP#DUBADWT)ez_G% zXx@W(6(9FxOdzL-utAmKkiyMma$qY1<4gFB(ewBrZ;#Fnk&ez!@oru`bL(l*`)s9k zfIO<@8>acaLSyU0=LhhwK(I&U=!s6=d&&~|Ve-N zr&=^y`kV8v2OHzYBn)SiOCWuyl}Xe8`cR&2aVE)g=%^4SXe^mITpfp&g$014Q@tI) zD4EtA26R~;1Dvfz9KS4=`x|KDkI>bc5fEFXvbf86cwhMyqy^$a%V<|dAcj1^1dxA0 zKdOvI<-Dm!JADH>rm7@AdtCnPH`lW;WhBw-Q_=Qd4CDs-LFteBAGjzyQ{*&UbZ&8o zS1C1E6g@7ryuso`FXR0RCreYxq?~VHb7J<>n(JLf#ab-k!4T7zjlUQN3N>I1ZNW(& z4gdiqA(!sp_$HY}L*9yG!JmwU;A50fqpLhwT3L|Lgy^8tKh!EkLP_;JtvV^6-FGhB zU$kGQ?{s!+mR=_lYv~$zzO*y)2zc%5pdtrw&1AF}F|K}NGTBvRh`iLRyDaCv_KmXh zD4S6$2`^Y>OsY4hnLXkf1YN9jY%A^hyjuUx5PTH|=@zR<0^zzHTU%-X%GBmw)-17O z>R^0`?WMD5dqzZ);Zeb)>x&#dUy!{{GYRHDH=+n2>g_7hpOmY%DDC#-+JQdaz!60` z_|Z~tu1W$}%iPLl%yCKYocw6=Bvq^Q_1#r2L=Q%%ghA(Z;<==Mb&%;+WzetCd8UGf z>L%LSqC-}+V(gloD2$a;mjX+Me1sIeA5}uP5TH}`IeXRC;x2pc?;N|iIKt}$tU z&GAxDroB^)JOdCr9-DH91K3xxzP7iz%((9*!1@T&JqeyMd*wjKx7=4NS6uk)QCg!>H?irRJL^=J-69~!&FCYW==L$Nzh0+W;Qohr`Z zbahl2>NH~^z@_J8tBYKfr#IG)bJGZ z>)e#{-CmaDSBci{sw>?YFZ&F7@V-5~NVZv%C%9{T*GMy4{?ZAqt$R}8b@jXEoTzk3 zsS{(R>@+#oV*XYpM;E>P{AVA|YhCE^tU>A$zH#fBwkKK-hW;^}S}xzNG1sd2+2?VL z;lFst!aq8cm>J!h;fs1N!i&}jp&Cn*=L$<1%KMuVVrTUsKl~K4M8oEk?-R#n=BuFE z&(HRlPzQ)wEqbTtnrG`12ae;oiD0ssP?_q7QTeaAWS7FCGyDiE#6p!w501co@gIoy z;kHB>&3@h&PcNZNFBjcLG2ZL(hlY2h1OJhWbzhXbD_G-gzL{YH}w7Rf?fFog&YWw<$>N(CW^z>Uhls1qEHg5v)!xoL#4M z4uPY;4f5fbXKFCZr}o31S#CUhIdW(E*@3R5o}o#Y_8#M5X**%N8o5=icsoh8?8Qqu?qMzfi3XhDZ8~pW~U2)S(<=r3An5%dI8bK0GO> z5AU&+Mc@MlOop2dDwXog8pYWBLL5xiB-DV%+lyj|Xtf(^*l;|}{NY-Y6UEr!lPoyk zS|Xh-v)GmQG~P)b`i9-%`~X5|6W!{%q7AYVGSlt$TA0!E&VQ}o`4^&lLSMKr>!Qu> zoI)GWa3cMZAgK8=oJb-CWcK97ArJJIX^Ka_{ms0)XY=A(T@ux_1mz_BY}abJS)w}@ z#W*(mDtM~J?R2F}2B^s3VedrPqZ-THrLBa4579DJ13JhSKE|H^gh%?tUek0G&OF8N zOD12KAUohX{#I?R_LNlnGwS^BWe4+NpyT*wP@Fh)A<%0ico{O>n>d}nQzkOM*4YF( zX|S?O@iy@FW};IIi;Q8LP4pI%?JZz-ZAhs~KRO`B4uy}Vk&W*pS1u$>e>hEcst6A> zv3fURWe6+~LhN4_V8=Z4$wHO1q*(17gU()wGg)#>5vomNQY2YW=WafhnO;$68qBkVKO=J`SYT9J;Lmsmp{DE&AxJ)(28DC|*u3cwQZ@o%BF^ zfQFJax0XeMm{$fG@5(+)R5&b~=P}8_Q)^_%z68ek{TROfNL244)#D#VaD!KeT5XvH-<@Kc z`%97TKt$(m?r1Ve`S7%@qBk}T96SzJ3@!@Z!|&mm++@U*5CIab2D1+|js`Ln7;z#;9wChKLQkN~$XxcqpEOTS z@)?JUnhLv|&~6wtCfvD=y53^7%J0Vt@CmLVIj&G?h!~Tfuqr9^>%g`S9acn8EOM zS0fnzb75Bv<<;OHzpouMtCzm!Y**jx4qiw-I1720AS^%RyEVN>l9nIH z$+gmsJ}`L1RDGowF2QLJRd{q?WfuzV#~2#IBH0OFHP9WY(a<|R7u{)=PX7S|6CS7& zkZV)yA4Nh6h%;v;2N-Jt`Q^LUK{`{#vL;xN_?N&Ch+RL$<^c;2oEbq?mEP>`o~%CGy|zgD}6UkU=M73oX=PEg+XVpCKIkg`ooCP~Kh#ej!!Q@y`%&uDs+Zmscp&S4cCQ}^_=XDLYz_J`7c%vtw?tSd(^>~)cjD;{ zDMpcPpF4Q%0NM3jy&^-97WNUWvcyYG(rHH1ylkzz4JJCjn`UKRE)SHdShH2(A+$XB8VGR~Im=GQ*xUC(F`0lAP& z+-VU!+V6fE$}07!v(V%Dox5C^oK1$iCYN;Ard-MgyReS+hVg#MX)s{eAI_0g02)r< zCG8w7F=bePICk7MdRACrce#s<@r)VnbUTI#h03j}(@qF9!|qt>a#QjBvkpXT%yDc^OlE9}58d@L?U&WkT`8P4BJeFtRwey=Kx72x#+m>2 ze66gXF3BaKz*^2O+ZXxM>?*Zv8&+=rj^@;-zP*#Rwjb)!J{5l{HyzVZlutKnx{Y`` z`3uDRnZFQxqx-@l=3Q>1fni0nt0Rm@++X$7=7FIA$F=HMvxT=vp}Q`zKY>ijW#S7h z{i}t|53_eWsiaP`@jIpH@~J*Wn~Xm6R$7-9EaMwBRxU z^#T#d!6zCM$2{9{>I@9j)G!bSGNik0*Wi$HN&X_W;(4WeJgSpZh-SQe1VPK^)W6r{ zM9!CgV!n1RAD=HK+yzqFlxBCD%`oN%H$Ob>{I+nn20YHMofYfjSwtF`#K(WT`6fIh zS2!c1CAu5@?d*52gAHb1CJ8hllNazs6-4&L!inxeug~;mhj}@a-u-)_)Z4JeTY=eL zV-?tG!7OD4yP05sjBgr%1Lv{gxZJ|RHT%k(P%8$K@c4dXZalPf!xTmdFZ|va(^eGk z#qqw))BF7M{)1igU}PGi@s^7sGcrgd|Ni?ghU(S7zOi_ax<{f9TR;ONVB-=g>h8^6 zbc$Hi^M(r-2lP!d8BeAlY{0h*{;&3~`YY=0TN8sc3aE+z}n3g;8v4ndQ&iSsulah>Wi}n6){AzW*Xt)mBIJqV}Vei+_*?Gz0W3yRrXZ`e*zN{k4T{{!}8rkgd5ql#BuY zggmhEYu;2=>a?Z1=Kd`~KhuV2cJc#CtojpQnvyp@eq`UQMPB3nSvNjF7YjH7M23p} zwelqz_L!zscKum-{w26FUE#)m$6$a!r2>2dE~nB(*#GRN0AQ$$+y&ylE>;Fy?8W42 zvA~~2k!+ysNI~rQ?<7ooz$Ln9&?0IT^B4NM28we9)*ius#WxP@;^0iizdQ~uS->v*?FVT8MT+QPP=sj>?_K$`*AqC{ z#igC=MDm|)q{Fh2ZvP3{?;!nu4B7D%?b=|L=^hZmQB~rtq{ocj-voRIjU))@cm$<# z$xtnRO-79#0;w%5r0r8xap%rm(KI#IgjBp$8HU;u#rpT{B#Qiv2A+`~lp5BSwp-_^ z#7!KBk-|%NSsFb}=#`!FIC{)iD0C-^^h%yrS`BtlbgUdFa_hskgohK41_KUhj~5)~ znt29Bl+JG^TxVl{R4imA3+~n=sOrM=JIQz;Rk!9SgaOR;;Fn63dJ?aW3$TF6oo{x2 zqg#+jD`JZm>*maYQGWc9CGKM&HZH5~YYo!qXt-zD?s!oW1&@KS!3n5^TepBT-0QvD z#DPo77!7+k`dV|O%4QT&JD$LG&+W#oL||y4mLvZ(cUj10g!P8cNtxykKqqzE+S~I& z)ZP;EtNes7LrLUeM3t7kN7?hd^`wD%1zHRd(b3U&y(Zt8PHyu*_y`TNB*MoZ-XLO> zL{ajYa97(+#wPA?y=!EiuCXW2R{|h+-Vc1Rc7PcIxh#eZ>u7;CmA5lq3&GXvpSiD} zESw!pJLauGq9Bm_va$i&vtZMQai~a&<4gKI7JAUsK!}YAk;%7Sb4kgoY>0--%$n~8M0B$N9!`f>2*(tM?Ga#Y4l=oXt#jr#Rb-U%8q-^C@;+96 z3;(QYnGktr@#ZO4ZzM?=3;o%g*H($DrDwE8nyCHqK!(KOd9&~7=KU`fmMlp;hOgVJ zt%v1Ls~zV|x*7P*0%G(06y48a86=cv8{F0B()i8g^I_D#f?b%3n+%)0-193el$fOt z^d_o50&kWti(G_O4}fatp82ljqTxuT9!h!Wb+Qn~;=KJaMUaJYq+2Co`6D@p*5{zX z3<)LiLCNk+0v+gO!h=#B@`cO3M*Pa-Q?8*pt@YlGxVTlJ7z*7p&X1R4>nQ zWnR(@mPN!_ZM8imtT8*GYm<5jw-Irim4sOa3DNh>VN7)|hp~)^tvJ1GQNuPT8%V+C zuxntaU(9~Ax-0bBs;6TCOhL9DLXfMsd=-X0Ndtjg75Qr+PEeX~CSem--*Cm^Eq-wn z+U_Dn$ZCq)d41xxOqU=yI~cA`P@wGQakp5z-uRyFcMF15b!5sa?st>umzf3rSRZar zyxJA;0qd3D$L0F9%*vl>73HlFkPgHgUdd;M~Yr<~^b-@gsz1ZPK`z0j5( zUMe3*zroUSew5bQw)&`%v4@B&4(9U<+wN{}B9Hs8^;FPE|I7`@HW%l|rZ@aA8cXNM zwI$7a;T-EzN*OX|dkTiN?|$VZl1iV*hUv!ulN@(&2+@z1NU%-^?$GB_ksXTEk4VuF7(*duQ?hV(-86^@jZyg=P=sA$BKZ z&>NO8L9W18Sx=X`G0qJ{5j>GPc1i%^kss8_I~J&79{X{ROmDRv6~wmgZ|! ztOy0!b9}|N=h|`mz^Yfme6v!j39TmS~g>&=tlVj(GWVOh8@l zkp;MuBwj@s@|P0a&*i5iqkNH^p0{tBlan)?i|S4@H=^khG%$8@Dnplz4N>!1EGn_o zC5c)77xj&MZGWhpbe@-VfC54=XR{&ZD@wtaL5f zcRGlpOT~bJy}4E1&Paos%oJbhnoe#AerbtYXB1VfaUA`P+js6&QgG>M#Q7>3n{|qA zKO|gV&Uf1w5!ZC3qm*8bXxuNa9ZOxCv9%HLV~;tZ_XnRJ^sPq z%0{UL&j8L2LaGbiz6vo;36tl=3dngCkXz$V(86VwfU7D53mQVLB5K(~5eQY~#x)NU z2-OP{uq>nO>`D!Uo1D!rbd%~`LlL3RXM19SO0^Q3C3;VbN6F*nZ;}AqFlc(TcCuev ziYo>(HIH6=mRj$1WheN6xa(wM@=8bem8hi2^i-#Wn+(R4-z0ILtQQ=%H4Y^rmhP zYuvy6bnVMjP2IbOhCF>uO|MmJKJl0quGq5Zx1|T1@cgKjU2U23v8($0fKdb@p=Px% z^kxEuYtR5h2PfXXO;lq|{K4pt$N|t*O|vN9uU`OBOTY8nR_i3Kc+^pvtT> zKTd4cLm}$Hm<&7ahbXTcZ-V$4VJ#oh@`k? z(M!+Vg%=@gGqS#SGOS<}F1h8wzvcTv7GOkode;~MtM%t!o4jlLvSduhIxwXw>kX=H z*~0-=nNBR|A9V?WKVHUp7N%cS*Tu`Q+WlEt`k*7OG_^I?>N*uDAgo2eExp6wPw*;C z{eCJ%6@o+MV{0IGb)76KoG=_ighnpu^2f&LNP%`1r~GIp_G0S@B#P3*oQUq>cu-az z+-W&j_?C24%sK9giWl_Yn(phWm|#Vzl7UR@5NL-`$E!-8l6~h`;l1eAmEI)nkyq*k zR-b7D1J^L_?`e2VNN!Iv+MsrqXx#(LC^6+Cy6^No4{NA4?hLbxh-aDI#Jw;BxCz!vyfdi{qKi{fn{40h zEF7cz>3ns8*au-s6!y_AHbs>eO@`;5KurK2^5^Cx8jDu#^nj?tf-$)7l@a_}%o6Be zY8Wl=i@!U@aYe;cjhv6bcl=bk!TPtZg=hw#hQE248h8pKbnvqY>1U=WLL(wmBG8mq zRv45TlojMjVS9{N$(_;oI0KZ5N13T*`K5?}73-mD`cahZz2$acu%J&GAl{C(e0IMD z>wl=WGjprM3Ua~1><`P{vj;H0d8WDeFFVtY2%uS{I|=ar#i>OBr=D{#o%q+wBmlCg z2pnIu>Zt#s+5h*VKquiaBKk`JXx0SOzIHmP`oB$qfEp2q1ad+&Iq#Lfq=`9pK=-G& zd#C<&LW!462SP^|J{9&~3V{3Wn;9uHAEn1v;d|L7NkL&>D z02lAv*4g^!NAuI*mA>zN^(FqqD4aZy!kM5S{VYGV`Jo&IlC2D8B?5dRA~%o0QOUtf zDXv<_xkRISmrsXVlPjQ{3xmNHZ93ghnfJS6)j$s@D+1BF;Om9<)&s@8A(!==sTxrb zpO>;1XFofK*UGyYzo*h3uMaC2)I56A`~G1+tub7i=j`p>q6d5PA8Hl@zem#up{q9N zNOiEv_L^Iafa(CSTukIKWEZg=+l!!0;4##@1=n^5knMOZx7-N5TUiiiQreWeKm4%P zuV?}fM3s9G%XA^D*l$zS>XDQ@*8ym%N$>3(p9zcnt0azFupHx?8%Cju9BeP`Dk}54=g~fq8CGlrLX#eu&Ijp@!i&6cDs@AIpPkMz(Gd@xwsZ6YO`?NDTKc1>Q{Oz z;Qx%6PIPPy!1|G6DsB`sLK2{#2fcZ=<8PlSgc3zGKHfH99eBn_4W?lv1p&2xhJA%- z$XtiFLn)I}0q_fEKGWvncL{jo&Q6NWH?5T;f(c(T>~D>|$&(3>`yui8TVeZROYN)| zmmt9K&www&g!wJX4ml$glH*$s$H<+B{gAZBv);QKzgVQk@Q)J^H#4g!ISO9gvyvKS@Sd}$3@GGR$j7Zx$AA2U& zf!C36c)tSl27&m`3*=FBvF^~lpTn#6D5Ws z-nx1xa!6@adie*mAc*1&exS9arXt&TdjjLjh}MaFG_OP*pY*8q`9GX!5~%k=t;+ z8p&ze+(@avV=??%lTIg5EbNP%>Q!zOe~(nSI7^umcF@o!;xW5MG6HG<^~sQULOFhy zo7xVq9M2gyOdyc2>Bnlb#4*nqNBEa`nKthx)i;xBL9<9*%0GRVv)PkIANL)h7)cco#B8BVfvoJ= z4R=;R4)zMQ2Q3tb9mLu&LQX<%Gd)}`Qw__ISB_v>+zY0faQ*I&WrA>R)aIR}qRgSd zA7%N3kfls1Z$btQ z30IUP-Arv6C2e^@7kv*fe06`q&mH0f^(<@n&~_!P3ILJP!Hn}Tp8}1P-1qimS8Oy= zg+J-*4*CFk9R+s7d2+W#yRXYQe^GaG|!P$Jy z-l6T_cXwO_$X)#9_(Hz0J!~v6r!*E+3LwSp%7>aC$d=CL;V%>>1-fQZEKT@H$c(#Q ztJ63OyoTwu_aty_DHXT3wYg0mfQkDI0#0sq>>T~Kcdi*{0@STc$48@EUwmwK<7^=e zuBhqq>}$QN`mI~>_S|{ZT+_6;Yh}EtXJ9*)KY{XgD2PPqsc60)K9qanS#Hr;6@G`@ z&Fxrx(L#f%gNKr_;CEhqLarK0!@D!^@+Zij9#ohb$pL1b@i^pLd)sp@rg62ETJr(7 zg4Tt|uW4+TzhY>3Bp^M6$Fq#H57Bu@&+U&Qhngc0Op?P=gj>wT_9!9f&lG$?_F2$2 z=Ke%%u{A|-J{-B;75bFKMt^B~F02~rnQKQ)IS5WvYrBHVciPcpcPk6sJwj#=4oyh1{jEeEO8ac+sh@%eU;qjk6LBi)nUofs5Zb#ThguP~)PAZUO<)AA;yJ z^Sq-zwI0C&%_}>R<9y3zO^+q8IP?X=81`)acT)kQd_ZM_R6Y}m2&9EUR*WsyM3U>q z4OI|aTsu2PZXb|j4*IYv@1Qe$yexoE>`+7=01qd)n4>r(<|o;&tcGG@dCg8894--^ z1|UOwNC2qg=qQ(%^{0GbqzYCs#(Sam2Z;pKqG87o<%6GpVT`8$JpAWr|5rc)_+Uu4 z_4prKliHyq44Y+MXiR>*)|ZmE>o`@VW-3ASOcGvsf>(X>Xd4U!eACaE1$u!9 zIsJc^d;x&-@**tvVU6#AAV@$HK9^(DVdA&xT0R3Cu=V2qb6V31dL@;`- zo#nuL=nPiRoeE@iG zKu_7o@pG9RC`sh^2Gp@j?)tW5GSSt{09;VO z!L&3ZQ$~IMeV-DYYEgcKh?&yIeF#r)(14JY`}jY0~|9(lDgV`HvT% zA5qOzI~}P5@Asc9HGZLA`S8mQOO}Y;#QL;{1EcScVa|5ZO0SFk`By|ByJC3pgPvL8Tr4w*& z*J+bQv@u&jjAk1>?c4ZnwqEiE1qbJSRrwN)3`ro-yIE8Oe1@wWT(*46MB1PiApMVE zUDZ=XU2%pdYKCk0Ae4ti1#T2wEjRlRRMXH&5q69wX00=j&n4d+7W?8Li*obeo^GHc z5Bdi1XhYiQOdGT_RO!+8)&2%Mnk7;I=jn>3BtKT*s~7rGOP<>k4&?{^iemP zM^)L;e#uJk=MIztiu|2vOX*XN5TV~#>jVo7^!Vmn)yK`^fRd3dJm$(_DJ<(6FEutC zdL`ky+Q&KbHny#Mf|ODm^%L!HA5{SWzLe;YIQi6Lp;(Tq*YAYae2bxZx6Xad4d=dm z%qMs*5}Bcd4*3c|5v2aQ&t- zjfC5J_WRgjUm|}zRg|cJnsr~YfX?%P_)s&&q<^l@gB+cjY)&U9!Q#*5SrB$Zv3ShDd;$6XQ!a=ah}MkWga7lybpn=dcAEQ__rCOYj3gF6wnlr5K@ zq8J}I>$2#VSRN-Qe45^_32pRDANY1Zz4c@>b+o@y%;|`3;M&IMM%Bfquam{f4z<&k zm0!4BZBKoq-@Th@xw_&Z$ve5_PCU}$?%^(D!XX>bDglv}Zh7~l@!2r~d}oT^XVQ;v zef*-$75<$%MgLVg`B2vj6f^Qg}pC|PcZ`h+4%==rUxKTsn^zg#-u*%IU&zJZ}_lE#f>zwJk?vCD|EvND k3UH$S&!YeT!n=oaCoa@Dbc2Nj{m-e?l(ZB}<;{Zr3w<8mzW@LL literal 0 HcmV?d00001 From 30b27dc6b3287d477830a406754834a7e3b48620 Mon Sep 17 00:00:00 2001 From: Arista Jenkins Date: Tue, 1 Dec 2020 17:22:25 -0800 Subject: [PATCH 07/12] Remove incorrect merge --- doc/voq/everflow.md | 86 ------------------------------------ images/voq/Everflow_voq.png | Bin 27807 -> 0 bytes 2 files changed, 86 deletions(-) delete mode 100644 doc/voq/everflow.md delete mode 100644 images/voq/Everflow_voq.png diff --git a/doc/voq/everflow.md b/doc/voq/everflow.md deleted file mode 100644 index fbf2060b58..0000000000 --- a/doc/voq/everflow.md +++ /dev/null @@ -1,86 +0,0 @@ -# Everflow support on VOQ Chassis - -# High Level Design Document -#### Rev 1 - -# Table of Contents -* [List of Tables](#list-of-tables) -* [List of Figures](#list-of-figures) -* [Revision](#revision) -* [About this Manual](#about-this-manual) -* [Scope](#scope) -* [Definitions/Abbreviations](#definitionsabbreviations) -* [1 Requirements](#1-requirements) -* [2 Design](#2-design) -* [3 Testing](#3-testing) -* [4 Future Work](#4-future-work) - -# List of Tables -* [Table 1: Abbreviations](#definitionsabbreviations) - -# List of Figures - -# Revision -| Rev | Date | Author | Change Description | -|:---:|:-----------:|:------------------:|--------------------| -| 1 | Dec-1 2020 | Song Yuan, Eswaran Baskaran (Arista Networks) | Initial Version | - -# About this Manual - -This document provides an overview of the SONiC support for everflow configuration in a VOQ-based chassis. In a VOQ-based chassis, the everflow configuration is applied on a linecard and the mirror destination could be in a different linecard. We propose a solution where the packet is rewritten with the GRE header in the ingress linecard before it’s sent over to the mirror destination. - -# Scope - -This document builds on top of the VOQ chassis architecture discussed [here](https://github.com/Azure/SONiC/blob/master/doc/voq/architecture.md) and the Everflow design discussed [here](https://github.com/Azure/SONiC/wiki/Everflow-High-Level-Design#3138-mirror-api). The goal of this document is to describe the configuration and design for everflow mirroring sessions for a VOQ-based chassis. - -# Definitions/Abbreviations - -| | | | -|------|--------------------|--------------------------------| -| FSI | Forwarding SONiC Instance | SONiC OS instance on a linecard module that controls one or more forwarding ASICs. | -| ASIC | Application Specific Integrated Circuit | Refers to the forwarding engine on a device that is responsible for packet forwarding. | -| ERSPAN | Encapsulated Remote Switched Port ANalyzer | Another name for Everflow mirroring sessions | - -# 1 Requirements - -In a VOQ based chassis, the mirror source and destination ports could be on different linecards. This configuration must be accepted and the packet that goes out of the destination port must have the correct encapsulation header based on the configuration. - -# 2 Design - -## 2.1 Problem - -In a regular non-VOQ switch, given the destination IP of the mirror session, mirrororch is responsible for determining the outgoing interface and the destination MAC address by querying routeorch and neighorch. Once these details are determined, mirrororch makes the SAI create_mirror_session API call to create the mirror session. For example, the destination port is set in `SAI_MIRROR_SESSION_ATTR_MONITOR_PORT`, the destination mac for the packet is set in `SAI_MIRROR_SESSION_ATTR_DST_MAC_ADDRESS` and so on. - -In addition, mirrororch is also responsible for reacting to changes in the route or nexthop and update the session by making appropriate SAI update calls. - -In a VOQ chassis, the monitor port could be on another linecard and packet rewrite is done in the egress pipeline of that linecard. As the ports of other linecards are only represented as SYSTEM_PORTs in each FSI, we need either changes in SAI or changes in SONiC to support everflow. - -### 2.1.1 Proposal - -The idea is to fully rewrite the packet in the ingress chip and send it out of a recycle port in the ingress ASIC. After the recycle, the packet goes to the egress ASIC where the packet gets switched out of the mirror destination port. - -![](../../images/voq/Everflow_voq.png) - -The mechanism to enable the recycle port is outside the scope of this document. - -## 2.2 Option 1 - Implicit Recycle - -This option can be implemented with minimal changes to SONiC because the bulk of the handling is done inside SAI. The SAI implementation can be enhanced to accept SYSTEM_PORTs as arguments for the `SAI_MIRROR_SESSION_ATTR_MONITOR_PORT` attribute. When a mirror session is created with a destination IP, mirrororch resolves the IP using routeorch and neighorch as before. The following changes are proposed for the VOQ chassis. -Upon receiving mirrororch’s request for a remote neighbor’s information, neighorch must be able to return neighbor information like interface alias. -If the neighbor information returned from neighorch turns out to be a remote neighbor, mirrororch needs to use the SYSTEM_PORT of the remote interface as the mirror destination port. - -SAI implementation will program the hardware to send the mirrored packets in this session out of the recycle port. When the packet re-enters the pipeline, the destination MAC of this packet will be the neighbor’s destination MAC address and it will be bridged in the ingress ASIC to the correct egress destination port. SAI will also be responsible for programming the FDB entry in the ingress ASIC appropriately. - -## 2.3 Option 2 - Explicit Recycle - -This option can be implemented with minimal changes to SAI as long as a recycle port can be created. The recycle port needs to be enabled as an L3 port for this mechanism to work because it requires the packet to be routed after recycle. The following changes are proposed. -Upon receiving mirrororch’s request for a remote neighbor’s information, neighorch must be able to return neighbor information like interface alias. -If the neighbor information returned from neighorch turns out to be a remote neighbor, mirrororch needs to use the recycle port as the mirror destination port and the router mac as the destination MAC address. - -SAI implementation is the same as a single-chip system. The mirrored packets are rewritten in the ingress ASIC and sent out of the recycle port. When the packet re-enters the pipeline, it gets routed because the destination MAC is the router MAC. The destination IP of this packet is the destination IP configured on the mirror session, so it will be routed by the ingress ASIC. If the destination IP is reachable via multiple nexthops, the ingress ASIC will loadbalance the traffic as needed. - -# 3 Testing -TBD - -# 4 Future Work -TBD diff --git a/images/voq/Everflow_voq.png b/images/voq/Everflow_voq.png deleted file mode 100644 index 6965437c0383b8d8f4a768e0043c3e6b6d1b2a54..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 27807 zcmeGERX`nE6E%tgfdGL30fGc~_uwuG?zV7ucXx;2?(XjH?(Xic!SyV*WPkrT=RV(u zd!b?V>N#un^sX8;s)M8?1V4ZJ_6Y<87lfD8x-SO5qJs0h?Y;1!GDo3Fqxyhi-| zQo{WFxKh>@`bMUDARwfXI+~h@LX@NfT3VW#1EUmVpR65Z0t0`^XnJ&YbaxDNbZYi! z#w4h!ZlED-e30t`$(HV{H{q$}u~b6UjRR$mAHUOM2&0qO_|#7CG4)yOhSZaSsFnp? z=nZr$XU97W@#Duna9owfmK=~CbI__E!Jc7YRR*35hX}Xd5ZIx+xIh$;5O-j8H9)H% zq8JcmJx@q@>Ld`O9N7>tjA@`O!O;?R559dvup~dsAgs*-7hvBg8p(^!AqCjgGd!h>rHE{`U5EcMJSS4eX<~ z-VVzR3=rd=>Ty$O_cOpg)z$+D>x+wnkOR-5K)^r|K|TP_K!Lv?0Y)I;f1QJX5CcC! zKt4qIgFpg55rKbF8DRgWf(2xJ`1c%C+=k7O6aTLqY{2vP%QX17 ze`T>X;lu}sOX2ccSnJ_3Qh%qW#pn8ji;K%)t*g%_Bk=Pdao{&jd_!AXOEwxB2L}gg z2L@^jYXcfOR#sLTT6!9KdMaQJDjO$rTP;T_a~p!c3;DMk0X-WXYa>frBMWof_j0wg zE$nPL@$uhp^ylaAdg?hE{dXsGn}3=GY#_~h3Jo1KEzSR%n5~ii{|B-6l)uIPYS-Vl z<9IKOP2NV&n%~0AOwZhw>%Y$B`0GypN8*3={991U$WhM}AYcTPv;nq>i|IS_KT`kq zl>b*rrT-)u7-;`j@_$nPNAkT1Y?9VSz(i{(fsQw za5%X>anSsk8eE_39X3!wKzKle1$gBgK~GYlQ$lBP`aJND7aa6Q0R3YCdi-Wm--5s_ z(wtYJte6NsX#qW|V7@s%-FUu%w)5qM`hl_MaU0t*hqD!S{YIO8>ha6QwL8vprS|<) zm-IV_I^D=2Xe{UmkpB)+F!)HY|Gol~(7=!b{K3NsdBFZ0HJDfExS)uCjs{83xb7!P z$DjXYahLSmnmES+1^bf;*Xs$K6z;rw1O30^?>Rw(`P)6fp*+Eb{yol+GFV#1y$$}{ zjd0@*DERXF1sCN1xaHOb7ff2@pSpXl0|l?21U*6i6eYC51seSS3l$TxdYfS$@1jtQ zizbnL)ZYbAAX@Teh$Ye{GPvYLH(wKj_Jflv6cHLoq}#hjw-Ih+gAc`1&@kwW-Ayj_ zmMe47z~RB>Q>`L%v;F!3rY@4mQB2A#rUC8J-q5Eb)7bB%Dr9*oL3ppjEj zay(*nC~b|866e3cY={Ku&BV}Rp!jc;E^L95=l`GW|N9+6#?k7zJERyRn=jehoZw)- z9!D%M+Q;5jOn?GQMa|zYX=7I<7XHs7bdBOk&Mz9w{k@M=YVL5p7TLo4pqloY9ZNAR zIi1(Uq2@q%aJhAH(#Sw>va8@wVJV88Mku!`qovRH&jtXE3!=s1GFu0p6Y*U-y<7#) zh~A#J>_ovHRDhWGzfGcz54cGvDo5S?w@HAaB3PQ#UsdY#>uz@WRavZay^yKYm0X{# zswnDg^#o1TnJpX252Y}weCKxl-;Ik94zykgPO(&($nJP9Ry>(WSV$<)_H0FIsm4gB z`0E$}0YSDK7}(Te*L~OEDSN%{_7eBb!R80+?MFt90>tA#+uH^|_=-NO>tSxZ?nDOf zREbixKqq9So~r+^L?WHMkv;k`!N59?a@9D>Me82cGavkw7I%Jsx9hMpC%e}!Cuk}K zzCH!;W7%9@#Lr=&WPkIPT@SiHRjX)hW2hBm$2gZjaFSW$LQIITnT z@J>N6k$D@^Xl2X&u73E_F>YqY1{l+&L95%-92KjpW;hSWw|e8ja(bg{%f{TZ&L%*G zkQJFsOAIoNthz|d&D!DbJa>+$zaI=ds!%*=%61(oWJr1^7hYGh@HTez_f(u;tmH0K zFrrmxO^U?sjRPOuTG3@`)7$pz=mIhfYthQoniaabx^kqmf70vp!8W?TIA?ir+0gS< z1ny11I|~gvZ&n?|qthR(Qqx}^XV!5f&y(l(gMHK;$??N8mn)KyWh!}DMIkPf&}^hc zQ^p_q@Hyn5rPgI9Ki6ZpSZzJFBjWk$Zb|-1D_laR*_2D^uKy~BoccG>-}AzYki>F- z%uEISB6;C)P*4x>rheu6V#!)#5nfZEBCUz%%Wt82{89eqVYXONx984=(2eJ#+iCb) zV$on^E1D&W(c`5mR+}qdTMv(n zI>fNItoRw{tjfBfPUrho#dP2-(1?(HY$<%Xm;m2{nM4wp%4xJn9gZO?KAPh+5}ihp zq^+ZGamIR~+wFE7lq(uuY{)N;k5qN?$2~2!}!PLo{acW7g=g;_$X4V|?Q@iD>*9&hSb)vG7aZ_(6qa zz45+_W%a9p7_fxt;Za)(L_mX&S$8=sD7;4 zTnRqnIsxI4P}D=PDM56ShnE)ls~H9w!(r`^eu_bpapoBhgcPdNzMonOf2$@zBDfBk z5j>E#ZWb9Ecde%;7K{E$YOniIDE_28t3O9;xD0dLQ5Bz=B;$>GyEJwN^HqqAA6m#R z3BLv?NC7JxMXlB}aNhn#&F*j(y`a0baKpzWAmiHCS!?rz2_JxcPzn`az|BR1gpz znqDbDUc9N;&3#7oIjKHMI{j9*HK^83KHd5D2l}-;^XxH8mtxUbl(S&zeI%84{H4b&^`dP7`CjSj>#<1g2y+(0;@bi?%y(1RNF$%QOZ@OQ8mYT2hs|Eih-zhTxba948~#L&v9RFU zH34WiNd$9_C&y&Xj~MH=g-h{9xBW0oc2_B9cuYKQH}0SAQtjt<-YT}v(%T&!TUdUX@wcoMd(>pi5g8iKT9i=whMH;7`Z!HD%|0$QUBfhK z(ml=NsKuSI=B!TF-hiwH%NySlfXyZT_3-Y1f$oqoY!~?K3|`v4e|wCbeRrZ*YbtKW zsP$FrG*wA)VQn3!j8lIkvso)$Vau}fq;P7#JyGmzY<>Ew8|P)gv z^Njrq0&M>%Jr|5%fGYptITmyI_OAh9Ruib;v?!OqE%Hsg61ZfE3uUUYJ00=7$mNUr zoTCPV0--h?8E8y(QBfb}PF<>5)lw*<2|y7q=z)r)`kdw&IW;&w3jFEQUNODd;KxpY z7~46%<^6%Q1pQ?hvFO#*6$IX3vL^&CXgKahy$&3wcUu4|ShP%qCzD@h05fuk;6K_2 zHQ~l(Tz3FrVod=S$~k#@yCv^bUhVROfVi39-5}PD@T-)ya3p>!_+i*_-;B>U4b)FIrcE%|z`wwZZ{oC4gr=K_GF58! z;{Lr(%Lp!hr?acYC-}r@da_w=c>1uv9`DYt4joYUise>LqYf4r!4Qfs*;;F^zXVK} z)^4|l_rE0N3!~x^d~9=A_xaC=;u7xIrV2RA3hQM!MV&gExe|5)oX!Hy3k`DIqYWTb z#vh7JrYn=zLeQV&luf*uX1zGmC zKBNve{Dk5S-r^0?s|7py-Fj-g%J57~iNnF)^+8yJ3`br5RCtB2XvT46)A)#vaQCJj z_gA{vGB2^~OQLHL)AxKEZcwQ1#p1d~Zk@e@8*oIMmbm$DEeN0im-Xf1*nHRU`w0hb zV8z>31vG=+kcCOEt)FG${j8MHy&)YsYeBf0;d;C2YgciVk{N4YDdgcF%g;3$SS8Bk zuhsUIG~3M8aYf@%q%iyzixofTUMe}_$eb)_e@&LC*L#g5(RaE~bVf!0Ye)77d2Z4E zNx+az} zB6t7QBT0Z3a;m3r0kiMcu*G%Eq^w6RbfMZZv*@&O?rtdRw_!;nLD8PdZs<|}OW}Vl zZJ~}_pqN@P!(LBt94SGyf_daeNp70t`SQ5<W_(jj{+(o zB)oOo0}k(cvii2fL4}r%b~}*mJZUx(cAJBx{0W5$@5$aA3Bdn9BN+g+Dg)|v#qX>HdQtRA;Drhu2`#}Z^Qb5^V8Kx9&6a;oKa zY;mcktf58g-MHTd63zHzd4CM14PM}IaTd@a${?S-sMbp}yzsIe_^;KRNe<>xy7qlO zy{l7|!mlzcq}5X^!Ot|Nhi)q0-nQO{j{1!{)=xsd{ae`pOw~< z4jiAmdaV)@Pc@YbIye(D^Ti4Ud(%Z+69wwIt%;SVg8gF|+?s?Ksje?5KLv*mzMD_> zf$vBDnM{Jf$@D11-D`TC3hUn_FM>Ah3R#iLpT5JF2eP7!l47e4P!q^VV@DWTU}tYtKpxY#zyVOM8G7ir(G1%(lKBz+;S~6Y@v{Djmy&=p};*bh7$Lse*s(e0V}&NnW8_R6m;B zx0{g3q?KvrRx6x}NqhUw$QEXaSUm8O-x@YQ=q>foMKO~Cs`iHnoMz}8G`R+m&`O^9 zavT$_;rf}D5I3Oz#aDQ;fbf=>ZBJWrr=N`hM0FNcqjx3|I8O2LYbS~Kc&3}n&i$kh zVkYG;!I?8^kCR2RdHJ_ggTN%S_KdyG7s0_>p6WPmkhtou>_6EorV-kAk*)G+* z`<1cNT$sJG1KZP^`&YuL%#C- z@63QG(LPEgLcJlmC^8d!;aK`y-H{aesT^^VLk9hrYjk?yr5eyEGEs$>D4$v$2Hg=c zlg3gAM<7D5K?TmCj;T=gT#&-{^(~c!TD^r-yeQ7r$p!fzEW)X2Eedx|8g?K8PrbJ` zo$-`3Qh&U?#A8D;m7}ZiG{uK{(EPWQnBAwMnM8Pzq0=maR-G%oB#6Ws?u%I;Z zK2oB5%%)J!nO=8@5bz;t>+Yr5p&&kJ^^s^xw}+0^Z7V^!`1h|ASsLQS>irnX zMH`_#%0)u{J<2HgTARqN+!~&4m>*ZjWY@#s#%Vu-SPf|CEAec8Qayo-vd%%bI`0On z0=4}|)tKTp^~jL@@j~k#18Z0+yM3d{)|V^jK)^N3U7=k4Me%H>xarZ8=XuI9K%-kUaRnbe3P@likJhLE^Mnc_O8cN05&tuaRpt z?zPWPdm`4FjPD)3K@uK*(-$y_8TM(Np9f_++J59iK zrG$6~HCW_IF%gFLY0D*^J2G~*BL2LhMjB8GAk8w#bTB1eD-HHrwvAO=Hg&;^@OGo^q^O$mzm1qQ&nE) z=778V-2l+WoLQL0C6g(sQRgeXjTXNKC}>pPcaa|$CiHp;U4w!luI?a&=i}ka*R#xF zzZ^bQ9zAeS!x#V9r~An)1>J{GJ0chHZCE8ks*?0LPI-Ep^?f-L`T~^`zJmuVRRA~8 zPTiFUA887!B>S$fmz=m*AM3~p1SbT2NZYb0gPmd-hu2P}SA*xv)F;>UM=*R1hvV6n zr<~72>#7XbOo^~<3UHod*i1bUBaEhUCim6WF-VDq)?Ah`8BTN>onD%#l7V{L(8%Dn z+5F8nZWE-`1A9SKXOUXiNGoB^wb7Hi&>wRS#fhg+_Ga;(*XS`26*Ws;l_hdz4-LqQ zDy-j%Q^@MMvGH(6gF(WM!p+TWD{_?o zF|fYm;6OVcKgG6IUszEx61OLu3E0#SdzBeZ zW0~DJ*m7z1fTPp$fvXiBJo@|09%Ma62KeX=u#{jx(`kZltqur1xQw3f)y)*l4-mt( zKh~71HrtDrsWurldnkt5vF8%&?kH?8iPaCseKyIFjK8g;C7a0!9x(>?2IvFK`|Lu+lEVx)d*DdS+BzOAbFX!ug0aHLm83N4n8u&#r0cn zh9`=e(J`V-HanabU3$5I&y9+EycE`HKC8$SpIyTmYa>u5R6ym4$4e;5lpj&ca8BQ~ zGg8l&DigPlH8Beq+rp*TUQ09sO6IYRn^ z%_;N8y0(6FmxO7mbEWwC&8`RP(mpDRlUwgu7tA7#%w_7!M7aSt?rPkZWjF0yp>w2= z4*SA!Xf)hzVq?hVMVjo6i9<1%?VA}Rw`kj_1{?`D{J<~x4cyVhFHg%ux!o^D!nvB} z>Va^JGW}i(ichnH>10-xZKKFk&8{d@aYhiv^cSNfTb)khuruEn72|W)8gIups|J3vT~?!O+9%w)<6wD{ zrJl^2-ENc)>nnk2l)84W!{|QfE_P>O-WYiQu({)gt>W3f7SKp2k_DC7 zVJWjw;ZI}4%y|m@;@+Q-<0J*&GNQd$-0tl5pB7b0UmYLNm^M;L7n6e=F9gcr(;!${ zq(&0ydDGb+%OAHxO*z52#WAUX&&s%F2XppMS=G@?Lzy#g$=J1P$9Nj50_=rcKzpg~VASE3o(#?CJzzBgk3UDL+NG6-)A>5$ za5|rY;!XU9)1_R|=ya)eD(eT#U?M}lm~?fT$xMM(Xj;MkAbzaq3?^cNV_QT7U{s-w zOLdNN^g|%?44Cl)I>6+jjzwDvXR*~RS~;3BxyE_-b1PN~udEF%u*P&oFAlMIJE5`_45 zxab7|zs*}YJUP$*2%|5o1L{KgRj$Agnvv_f*m=Hsi>=f0m9)F>*sgqUNSs}&!jsn5 zTl8<8=%}K_N}wHFR5oU{HQ$OUogvOhCk^I~zH61nDFFIYp^bd|5iR97ho$V@BH2pe9~OHU-wslGHhOspCq1f0K|>ka0&foR1p08Nv_`PS@T>9sw# zbErh1>H-86cHPRYEOs>**B8cM_m@h)pJX0H-R8(mh8S;tVgs!~nS3pVLuS*K(~y;7 zioq^H0uU2=O51zayPN&1(UM#?*jw54CihUytl8&8?xM@$FUF!Ht}-$&1*jBi5-RDm zd6CrS;3ZuMbfsM+c`*|bw>lE3e*NdFrrmra=)C)soZD|1N_WJVVSPs?H#YX>Gy^Gq zKJA?40a2XDvX;x7fEg`LH)$p2(yq;T9$wpPFfx2EZMi%H;Y?^3 z8FfZf&4@TtuJAa}7M~mk(_V~gc;0+M$1X3TZhKZJ&UJiMZ4;W1SLF!}=}x}({IY(u zSgkIPjxv`EeqtcGy{i@COxJVrTYea2sHdEJT5YC+TKgj#WymfCci)eFwWUvDBeCEW zaRmgSASOHdwYQ|0efRjVlr1kVwoy%zO!DIoM2aW`eKq*__$%f_wsN@Xhyl$G0vePU zp}$T`XgAmd&p5n0)8u&loYZu`zFie1sq*}Vr z?ApBDU>gIJABs6x@!wrVFeQ>{m+I_OygaSE?)mt)=iQFbmp^zup|Q_b%bFC<-=ixF z!&X3I(y9MWSZeJucJ|;wUX#!Ua~yk9y-hB;KSfn)*N;_CuL?%=#(~2b$2lW3$FLa3 zUXB0DxR*seNqxOrTGu=7v#UEZENKUjmt@7rcvqVC3bG1xc5F?Sf={q}t5ubZm4tZ` zrEg47_zSb4F|g_{@x4sxd_Jey=OThzf7_u5Al4BK4Wg;2S2x3pjn%7sf=p-m#K#Q@ z?*4G-(%VpH<^&1Jd$0=%uK6Xv&T@C_!5P%p_GMP?7{6_+5ce;P!sa8|MzR`RB`UOGdFNm3t@j%?8$t423%mxyUWr%Q_dSA~8u6&? zcCsO(xD+(Y=gNg`H*A%~@{ z9r&A@I<|LDU5I;P8Kv)ULcY@-M1{(#?~ad7=+`$nq$oBP#i+x_tKgby?;x645Rz-S zrv7(-<`M;lO*-hkxzc2r27oXp1=~qvkxbDFfwca$Gc#!1S~65dey%P6Vm0=;{<*AO zv{GOOXV@c~w*)=niUKvpv^;;;Q|lp9S+1Uy=<^`@sNrM0y8IrV!#7M z{n8;rKKC-B)!JDEbr9UQHGi--97m9^Zh)kNLPxld+d`KVrlT`>GDu?F-}DTzj89WMAQFO!asZ)An+0jImXa3oKtO=ep*aMLOrafJxx%Xq!3dOGgHq8+m7N>V{aUruszZ#P!l`mFCais%Jgpj}C+*BN82m^>2 zIrCExX`%0nB^Hpd{7SAQg%;i9R#Jh;bc?53i+C}rOL?nt-kSdo^i*3PY;;2EBPY<@ zdueeHdA^uqq16Ybx-pfjP{Hg!=Q!9xQ-$?54j<4E;NBq9lZWdJ)Qz( z63KQu8ezHzdulT_;g+nYQXks>JNk>spAeXmSYv_If!5*6s|;>etj4PMc6HSnA>+COZs926Z}Yx?}{kz?!1Iz`=9#C7Mi~ zy0_&5eE}Wjcu8(YQThU_ncm@;=qh7{@AE3x&4RvP!@NBEL%N$C&P0Gvuyz68Nd4JL z8x=ff496bNWkWKZXq;ouRTwFZzId#b@o@%xKxno2HYFMSWSAB!Cv7BlK`j zth1H%p1Q~%I0)uN9hs<$Nc$s)Df5?T-tUQ9spLX|Z$Ztb%UidMyOP z=~vF>uXHlLntd=S)q28V*j%{0oe-FLpvM$jH;;~vHBUmQuB}>T95G*uK{2^)dCXT@ z@b-a*-VJLpaU_typ_Z=0w@`|4H#VNam8)N}UN`%Y7i@Fb1tI-gETw6s(VD$=f1G*5 zIfP2%tb`6QucG?hAx_4x-%)CMtmvr(F};+s~s9VO=iA1L$`pxS3R%rqTUZ* zcjC9+gO)={x0|v!FifO#O3~z$wN=#lc>JWcL{@35-rGzRE>~DKg>`(ZRE&V0Kf6CO zxNUtlBg|^I)llcJ{blVPDFA|t;Nydh)docQh~`GqdGh_LN&I!Di{E`*jje^QY(XDe zjQyrWo{6Gk;PWFkHlF4Jk%yMPprIL7uyOkP`m(h4W86V+=ykid{wS$!(A(CiD>I}d z`C2qL>sJBmlJm*n$9jK10}ehAA>|K6UoYxtV)W>PzYChK zrZovR1yqY%Q4}ksk|YbgZEn9PAY`dH?PPFCSD=e-bvQH^?4Qbe^?p2Ywl^i3CJC($ z>UMLnFBCubh|Myl#3Me|{Gxjj1?q6L18sDF0yA}-tp$WZv8mw1i8rIqJ^IV@|foNF>^i~u>iCe(kdF-|z{NN=2({4kRCEo8T><;2c1PerTgEK zSMx=?N04~w*d+7WZ&LQ0g~W~Sr%82jtj1WBp}wV*biXz)URSLJ?V{J@SBLxLihO6}k z0hls;HyPvvJ!jCht#f30U&La`W5LOW)xDk}h&)~q3YTgkef?1;v%(eFLe~x67(@>e zg5({cz3i;|<33kuj>$b7O}8C6wPoK0!%-wcCvCR>%c28YYf_7y-sy7qAY0!s-^qr<)e zXSefunVB^o6^-xZnBS+baeP05g(|at>|VxeO!V{leG=yBXgyn7$Qh+=6Yq$$`!XzR zw%(u)(^t427R7sRKg<8|lRt{$>f~;$CS`Fyoeb^cZT048El^e2n+q{m)cS&+6=wyg zwhjW)EZ9{P)j6VnhZq8J=}eTl)I10+&nk6Jg?-_Nt!7q4nSr<46){5}{=m-|6DnL7 z-10>;CHc`9OXp^+=vE_f*PA|1ru!_c&i5N13*9LYNS*|xFA zU&=}`%P(YkB#rx2M8zf8bTsh$(RegDP8M+n zP44m}P&TWx^A#Fa|GF-E)R6tr8Iv!0Vu>V4Uw)9Egp;@rDgMy-?ZgENW}f5Bh(_nc0}>j$3+o|P9Yk?N6%A}fJ_F2J4y3??CWZfeEFOq z`fdC%R}_`dVa-)0Lr9tokdqKBD;G>Ld0_gpeHdm4T$l&=guSlr*X*^h$u$~|UkE+P zSuOQO-~}p-JPqlUNB}xV1G&}HF-+n_j$-0$Tk~0=DM;Oj6d-MB5$3cON-*Ubok673 zS#mIdlMPz`Zf_t}@qEgsOxM-Am|DMhI0oqIWR1@?Zp@Go5i0|5Na{+~58USSZqMSr}tqJvaNJ1!uDR+w9PM;gbHe1bdDg-{K>VTo~2VVLB{i*pM zco61{E^7Kqe+T-h8(s2an&COJqMOKyUFBkh zGBSlz1VF+M`YP242|!gr{G^n_rWYkXx8GM)uwx{m{qE$vky-Y2!=1G}&g>KH5Xr&k zuBEI87=Mw#Dz-hv)!DGVy{0mOwrIq8)?aIJHJCpQwq&^$lue&eee`B(AY8j^YD(;6 zfPlAxj9OfpIbMtNjfPX3dxP$1nymFMbks@X?*by(D-j7jrQzJBOxnG%U$xb8v&Ry2 z<`)%+u4oR9a&BS8m;MWMyAf4sD{XcvpO7gFdNo3av}Cf}Ox(=pK3+EsRdo;|h=L`w zcuN9u$)&YBrW_5|%~=b$Dp;tK*!jbXm2SR#=_e8#Q!NT%6S^vdR>yNT)e&awtE5$w z4djbJ4TExQ*{yKx72E-?g|H!PiuPn?^8)jn0G+Q))Lyj5rU&;%RzX{H6@V=L4fZ!# zpWknhTRBCc=|GfNaD(lqac=m(Xc|sn~I7nGDar?KB{Hw z+ZkxqV*42uhrJr?;uux4uR$-`vV@YLv!wQybZey`o^jZZxdyq!Q*OC;uW3V)K8$L~R*3#$mP#DUBADWT)ez_G% zXx@W(6(9FxOdzL-utAmKkiyMma$qY1<4gFB(ewBrZ;#Fnk&ez!@oru`bL(l*`)s9k zfIO<@8>acaLSyU0=LhhwK(I&U=!s6=d&&~|Ve-N zr&=^y`kV8v2OHzYBn)SiOCWuyl}Xe8`cR&2aVE)g=%^4SXe^mITpfp&g$014Q@tI) zD4EtA26R~;1Dvfz9KS4=`x|KDkI>bc5fEFXvbf86cwhMyqy^$a%V<|dAcj1^1dxA0 zKdOvI<-Dm!JADH>rm7@AdtCnPH`lW;WhBw-Q_=Qd4CDs-LFteBAGjzyQ{*&UbZ&8o zS1C1E6g@7ryuso`FXR0RCreYxq?~VHb7J<>n(JLf#ab-k!4T7zjlUQN3N>I1ZNW(& z4gdiqA(!sp_$HY}L*9yG!JmwU;A50fqpLhwT3L|Lgy^8tKh!EkLP_;JtvV^6-FGhB zU$kGQ?{s!+mR=_lYv~$zzO*y)2zc%5pdtrw&1AF}F|K}NGTBvRh`iLRyDaCv_KmXh zD4S6$2`^Y>OsY4hnLXkf1YN9jY%A^hyjuUx5PTH|=@zR<0^zzHTU%-X%GBmw)-17O z>R^0`?WMD5dqzZ);Zeb)>x&#dUy!{{GYRHDH=+n2>g_7hpOmY%DDC#-+JQdaz!60` z_|Z~tu1W$}%iPLl%yCKYocw6=Bvq^Q_1#r2L=Q%%ghA(Z;<==Mb&%;+WzetCd8UGf z>L%LSqC-}+V(gloD2$a;mjX+Me1sIeA5}uP5TH}`IeXRC;x2pc?;N|iIKt}$tU z&GAxDroB^)JOdCr9-DH91K3xxzP7iz%((9*!1@T&JqeyMd*wjKx7=4NS6uk)QCg!>H?irRJL^=J-69~!&FCYW==L$Nzh0+W;Qohr`Z zbahl2>NH~^z@_J8tBYKfr#IG)bJGZ z>)e#{-CmaDSBci{sw>?YFZ&F7@V-5~NVZv%C%9{T*GMy4{?ZAqt$R}8b@jXEoTzk3 zsS{(R>@+#oV*XYpM;E>P{AVA|YhCE^tU>A$zH#fBwkKK-hW;^}S}xzNG1sd2+2?VL z;lFst!aq8cm>J!h;fs1N!i&}jp&Cn*=L$<1%KMuVVrTUsKl~K4M8oEk?-R#n=BuFE z&(HRlPzQ)wEqbTtnrG`12ae;oiD0ssP?_q7QTeaAWS7FCGyDiE#6p!w501co@gIoy z;kHB>&3@h&PcNZNFBjcLG2ZL(hlY2h1OJhWbzhXbD_G-gzL{YH}w7Rf?fFog&YWw<$>N(CW^z>Uhls1qEHg5v)!xoL#4M z4uPY;4f5fbXKFCZr}o31S#CUhIdW(E*@3R5o}o#Y_8#M5X**%N8o5=icsoh8?8Qqu?qMzfi3XhDZ8~pW~U2)S(<=r3An5%dI8bK0GO> z5AU&+Mc@MlOop2dDwXog8pYWBLL5xiB-DV%+lyj|Xtf(^*l;|}{NY-Y6UEr!lPoyk zS|Xh-v)GmQG~P)b`i9-%`~X5|6W!{%q7AYVGSlt$TA0!E&VQ}o`4^&lLSMKr>!Qu> zoI)GWa3cMZAgK8=oJb-CWcK97ArJJIX^Ka_{ms0)XY=A(T@ux_1mz_BY}abJS)w}@ z#W*(mDtM~J?R2F}2B^s3VedrPqZ-THrLBa4579DJ13JhSKE|H^gh%?tUek0G&OF8N zOD12KAUohX{#I?R_LNlnGwS^BWe4+NpyT*wP@Fh)A<%0ico{O>n>d}nQzkOM*4YF( zX|S?O@iy@FW};IIi;Q8LP4pI%?JZz-ZAhs~KRO`B4uy}Vk&W*pS1u$>e>hEcst6A> zv3fURWe6+~LhN4_V8=Z4$wHO1q*(17gU()wGg)#>5vomNQY2YW=WafhnO;$68qBkVKO=J`SYT9J;Lmsmp{DE&AxJ)(28DC|*u3cwQZ@o%BF^ zfQFJax0XeMm{$fG@5(+)R5&b~=P}8_Q)^_%z68ek{TROfNL244)#D#VaD!KeT5XvH-<@Kc z`%97TKt$(m?r1Ve`S7%@qBk}T96SzJ3@!@Z!|&mm++@U*5CIab2D1+|js`Ln7;z#;9wChKLQkN~$XxcqpEOTS z@)?JUnhLv|&~6wtCfvD=y53^7%J0Vt@CmLVIj&G?h!~Tfuqr9^>%g`S9acn8EOM zS0fnzb75Bv<<;OHzpouMtCzm!Y**jx4qiw-I1720AS^%RyEVN>l9nIH z$+gmsJ}`L1RDGowF2QLJRd{q?WfuzV#~2#IBH0OFHP9WY(a<|R7u{)=PX7S|6CS7& zkZV)yA4Nh6h%;v;2N-Jt`Q^LUK{`{#vL;xN_?N&Ch+RL$<^c;2oEbq?mEP>`o~%CGy|zgD}6UkU=M73oX=PEg+XVpCKIkg`ooCP~Kh#ej!!Q@y`%&uDs+Zmscp&S4cCQ}^_=XDLYz_J`7c%vtw?tSd(^>~)cjD;{ zDMpcPpF4Q%0NM3jy&^-97WNUWvcyYG(rHH1ylkzz4JJCjn`UKRE)SHdShH2(A+$XB8VGR~Im=GQ*xUC(F`0lAP& z+-VU!+V6fE$}07!v(V%Dox5C^oK1$iCYN;Ard-MgyReS+hVg#MX)s{eAI_0g02)r< zCG8w7F=bePICk7MdRACrce#s<@r)VnbUTI#h03j}(@qF9!|qt>a#QjBvkpXT%yDc^OlE9}58d@L?U&WkT`8P4BJeFtRwey=Kx72x#+m>2 ze66gXF3BaKz*^2O+ZXxM>?*Zv8&+=rj^@;-zP*#Rwjb)!J{5l{HyzVZlutKnx{Y`` z`3uDRnZFQxqx-@l=3Q>1fni0nt0Rm@++X$7=7FIA$F=HMvxT=vp}Q`zKY>ijW#S7h z{i}t|53_eWsiaP`@jIpH@~J*Wn~Xm6R$7-9EaMwBRxU z^#T#d!6zCM$2{9{>I@9j)G!bSGNik0*Wi$HN&X_W;(4WeJgSpZh-SQe1VPK^)W6r{ zM9!CgV!n1RAD=HK+yzqFlxBCD%`oN%H$Ob>{I+nn20YHMofYfjSwtF`#K(WT`6fIh zS2!c1CAu5@?d*52gAHb1CJ8hllNazs6-4&L!inxeug~;mhj}@a-u-)_)Z4JeTY=eL zV-?tG!7OD4yP05sjBgr%1Lv{gxZJ|RHT%k(P%8$K@c4dXZalPf!xTmdFZ|va(^eGk z#qqw))BF7M{)1igU}PGi@s^7sGcrgd|Ni?ghU(S7zOi_ax<{f9TR;ONVB-=g>h8^6 zbc$Hi^M(r-2lP!d8BeAlY{0h*{;&3~`YY=0TN8sc3aE+z}n3g;8v4ndQ&iSsulah>Wi}n6){AzW*Xt)mBIJqV}Vei+_*?Gz0W3yRrXZ`e*zN{k4T{{!}8rkgd5ql#BuY zggmhEYu;2=>a?Z1=Kd`~KhuV2cJc#CtojpQnvyp@eq`UQMPB3nSvNjF7YjH7M23p} zwelqz_L!zscKum-{w26FUE#)m$6$a!r2>2dE~nB(*#GRN0AQ$$+y&ylE>;Fy?8W42 zvA~~2k!+ysNI~rQ?<7ooz$Ln9&?0IT^B4NM28we9)*ius#WxP@;^0iizdQ~uS->v*?FVT8MT+QPP=sj>?_K$`*AqC{ z#igC=MDm|)q{Fh2ZvP3{?;!nu4B7D%?b=|L=^hZmQB~rtq{ocj-voRIjU))@cm$<# z$xtnRO-79#0;w%5r0r8xap%rm(KI#IgjBp$8HU;u#rpT{B#Qiv2A+`~lp5BSwp-_^ z#7!KBk-|%NSsFb}=#`!FIC{)iD0C-^^h%yrS`BtlbgUdFa_hskgohK41_KUhj~5)~ znt29Bl+JG^TxVl{R4imA3+~n=sOrM=JIQz;Rk!9SgaOR;;Fn63dJ?aW3$TF6oo{x2 zqg#+jD`JZm>*maYQGWc9CGKM&HZH5~YYo!qXt-zD?s!oW1&@KS!3n5^TepBT-0QvD z#DPo77!7+k`dV|O%4QT&JD$LG&+W#oL||y4mLvZ(cUj10g!P8cNtxykKqqzE+S~I& z)ZP;EtNes7LrLUeM3t7kN7?hd^`wD%1zHRd(b3U&y(Zt8PHyu*_y`TNB*MoZ-XLO> zL{ajYa97(+#wPA?y=!EiuCXW2R{|h+-Vc1Rc7PcIxh#eZ>u7;CmA5lq3&GXvpSiD} zESw!pJLauGq9Bm_va$i&vtZMQai~a&<4gKI7JAUsK!}YAk;%7Sb4kgoY>0--%$n~8M0B$N9!`f>2*(tM?Ga#Y4l=oXt#jr#Rb-U%8q-^C@;+96 z3;(QYnGktr@#ZO4ZzM?=3;o%g*H($DrDwE8nyCHqK!(KOd9&~7=KU`fmMlp;hOgVJ zt%v1Ls~zV|x*7P*0%G(06y48a86=cv8{F0B()i8g^I_D#f?b%3n+%)0-193el$fOt z^d_o50&kWti(G_O4}fatp82ljqTxuT9!h!Wb+Qn~;=KJaMUaJYq+2Co`6D@p*5{zX z3<)LiLCNk+0v+gO!h=#B@`cO3M*Pa-Q?8*pt@YlGxVTlJ7z*7p&X1R4>nQ zWnR(@mPN!_ZM8imtT8*GYm<5jw-Irim4sOa3DNh>VN7)|hp~)^tvJ1GQNuPT8%V+C zuxntaU(9~Ax-0bBs;6TCOhL9DLXfMsd=-X0Ndtjg75Qr+PEeX~CSem--*Cm^Eq-wn z+U_Dn$ZCq)d41xxOqU=yI~cA`P@wGQakp5z-uRyFcMF15b!5sa?st>umzf3rSRZar zyxJA;0qd3D$L0F9%*vl>73HlFkPgHgUdd;M~Yr<~^b-@gsz1ZPK`z0j5( zUMe3*zroUSew5bQw)&`%v4@B&4(9U<+wN{}B9Hs8^;FPE|I7`@HW%l|rZ@aA8cXNM zwI$7a;T-EzN*OX|dkTiN?|$VZl1iV*hUv!ulN@(&2+@z1NU%-^?$GB_ksXTEk4VuF7(*duQ?hV(-86^@jZyg=P=sA$BKZ z&>NO8L9W18Sx=X`G0qJ{5j>GPc1i%^kss8_I~J&79{X{ROmDRv6~wmgZ|! ztOy0!b9}|N=h|`mz^Yfme6v!j39TmS~g>&=tlVj(GWVOh8@l zkp;MuBwj@s@|P0a&*i5iqkNH^p0{tBlan)?i|S4@H=^khG%$8@Dnplz4N>!1EGn_o zC5c)77xj&MZGWhpbe@-VfC54=XR{&ZD@wtaL5f zcRGlpOT~bJy}4E1&Paos%oJbhnoe#AerbtYXB1VfaUA`P+js6&QgG>M#Q7>3n{|qA zKO|gV&Uf1w5!ZC3qm*8bXxuNa9ZOxCv9%HLV~;tZ_XnRJ^sPq z%0{UL&j8L2LaGbiz6vo;36tl=3dngCkXz$V(86VwfU7D53mQVLB5K(~5eQY~#x)NU z2-OP{uq>nO>`D!Uo1D!rbd%~`LlL3RXM19SO0^Q3C3;VbN6F*nZ;}AqFlc(TcCuev ziYo>(HIH6=mRj$1WheN6xa(wM@=8bem8hi2^i-#Wn+(R4-z0ILtQQ=%H4Y^rmhP zYuvy6bnVMjP2IbOhCF>uO|MmJKJl0quGq5Zx1|T1@cgKjU2U23v8($0fKdb@p=Px% z^kxEuYtR5h2PfXXO;lq|{K4pt$N|t*O|vN9uU`OBOTY8nR_i3Kc+^pvtT> zKTd4cLm}$Hm<&7ahbXTcZ-V$4VJ#oh@`k? z(M!+Vg%=@gGqS#SGOS<}F1h8wzvcTv7GOkode;~MtM%t!o4jlLvSduhIxwXw>kX=H z*~0-=nNBR|A9V?WKVHUp7N%cS*Tu`Q+WlEt`k*7OG_^I?>N*uDAgo2eExp6wPw*;C z{eCJ%6@o+MV{0IGb)76KoG=_ighnpu^2f&LNP%`1r~GIp_G0S@B#P3*oQUq>cu-az z+-W&j_?C24%sK9giWl_Yn(phWm|#Vzl7UR@5NL-`$E!-8l6~h`;l1eAmEI)nkyq*k zR-b7D1J^L_?`e2VNN!Iv+MsrqXx#(LC^6+Cy6^No4{NA4?hLbxh-aDI#Jw;BxCz!vyfdi{qKi{fn{40h zEF7cz>3ns8*au-s6!y_AHbs>eO@`;5KurK2^5^Cx8jDu#^nj?tf-$)7l@a_}%o6Be zY8Wl=i@!U@aYe;cjhv6bcl=bk!TPtZg=hw#hQE248h8pKbnvqY>1U=WLL(wmBG8mq zRv45TlojMjVS9{N$(_;oI0KZ5N13T*`K5?}73-mD`cahZz2$acu%J&GAl{C(e0IMD z>wl=WGjprM3Ua~1><`P{vj;H0d8WDeFFVtY2%uS{I|=ar#i>OBr=D{#o%q+wBmlCg z2pnIu>Zt#s+5h*VKquiaBKk`JXx0SOzIHmP`oB$qfEp2q1ad+&Iq#Lfq=`9pK=-G& zd#C<&LW!462SP^|J{9&~3V{3Wn;9uHAEn1v;d|L7NkL&>D z02lAv*4g^!NAuI*mA>zN^(FqqD4aZy!kM5S{VYGV`Jo&IlC2D8B?5dRA~%o0QOUtf zDXv<_xkRISmrsXVlPjQ{3xmNHZ93ghnfJS6)j$s@D+1BF;Om9<)&s@8A(!==sTxrb zpO>;1XFofK*UGyYzo*h3uMaC2)I56A`~G1+tub7i=j`p>q6d5PA8Hl@zem#up{q9N zNOiEv_L^Iafa(CSTukIKWEZg=+l!!0;4##@1=n^5knMOZx7-N5TUiiiQreWeKm4%P zuV?}fM3s9G%XA^D*l$zS>XDQ@*8ym%N$>3(p9zcnt0azFupHx?8%Cju9BeP`Dk}54=g~fq8CGlrLX#eu&Ijp@!i&6cDs@AIpPkMz(Gd@xwsZ6YO`?NDTKc1>Q{Oz z;Qx%6PIPPy!1|G6DsB`sLK2{#2fcZ=<8PlSgc3zGKHfH99eBn_4W?lv1p&2xhJA%- z$XtiFLn)I}0q_fEKGWvncL{jo&Q6NWH?5T;f(c(T>~D>|$&(3>`yui8TVeZROYN)| zmmt9K&www&g!wJX4ml$glH*$s$H<+B{gAZBv);QKzgVQk@Q)J^H#4g!ISO9gvyvKS@Sd}$3@GGR$j7Zx$AA2U& zf!C36c)tSl27&m`3*=FBvF^~lpTn#6D5Ws z-nx1xa!6@adie*mAc*1&exS9arXt&TdjjLjh}MaFG_OP*pY*8q`9GX!5~%k=t;+ z8p&ze+(@avV=??%lTIg5EbNP%>Q!zOe~(nSI7^umcF@o!;xW5MG6HG<^~sQULOFhy zo7xVq9M2gyOdyc2>Bnlb#4*nqNBEa`nKthx)i;xBL9<9*%0GRVv)PkIANL)h7)cco#B8BVfvoJ= z4R=;R4)zMQ2Q3tb9mLu&LQX<%Gd)}`Qw__ISB_v>+zY0faQ*I&WrA>R)aIR}qRgSd zA7%N3kfls1Z$btQ z30IUP-Arv6C2e^@7kv*fe06`q&mH0f^(<@n&~_!P3ILJP!Hn}Tp8}1P-1qimS8Oy= zg+J-*4*CFk9R+s7d2+W#yRXYQe^GaG|!P$Jy z-l6T_cXwO_$X)#9_(Hz0J!~v6r!*E+3LwSp%7>aC$d=CL;V%>>1-fQZEKT@H$c(#Q ztJ63OyoTwu_aty_DHXT3wYg0mfQkDI0#0sq>>T~Kcdi*{0@STc$48@EUwmwK<7^=e zuBhqq>}$QN`mI~>_S|{ZT+_6;Yh}EtXJ9*)KY{XgD2PPqsc60)K9qanS#Hr;6@G`@ z&Fxrx(L#f%gNKr_;CEhqLarK0!@D!^@+Zij9#ohb$pL1b@i^pLd)sp@rg62ETJr(7 zg4Tt|uW4+TzhY>3Bp^M6$Fq#H57Bu@&+U&Qhngc0Op?P=gj>wT_9!9f&lG$?_F2$2 z=Ke%%u{A|-J{-B;75bFKMt^B~F02~rnQKQ)IS5WvYrBHVciPcpcPk6sJwj#=4oyh1{jEeEO8ac+sh@%eU;qjk6LBi)nUofs5Zb#ThguP~)PAZUO<)AA;yJ z^Sq-zwI0C&%_}>R<9y3zO^+q8IP?X=81`)acT)kQd_ZM_R6Y}m2&9EUR*WsyM3U>q z4OI|aTsu2PZXb|j4*IYv@1Qe$yexoE>`+7=01qd)n4>r(<|o;&tcGG@dCg8894--^ z1|UOwNC2qg=qQ(%^{0GbqzYCs#(Sam2Z;pKqG87o<%6GpVT`8$JpAWr|5rc)_+Uu4 z_4prKliHyq44Y+MXiR>*)|ZmE>o`@VW-3ASOcGvsf>(X>Xd4U!eACaE1$u!9 zIsJc^d;x&-@**tvVU6#AAV@$HK9^(DVdA&xT0R3Cu=V2qb6V31dL@;`- zo#nuL=nPiRoeE@iG zKu_7o@pG9RC`sh^2Gp@j?)tW5GSSt{09;VO z!L&3ZQ$~IMeV-DYYEgcKh?&yIeF#r)(14JY`}jY0~|9(lDgV`HvT% zA5qOzI~}P5@Asc9HGZLA`S8mQOO}Y;#QL;{1EcScVa|5ZO0SFk`By|ByJC3pgPvL8Tr4w*& z*J+bQv@u&jjAk1>?c4ZnwqEiE1qbJSRrwN)3`ro-yIE8Oe1@wWT(*46MB1PiApMVE zUDZ=XU2%pdYKCk0Ae4ti1#T2wEjRlRRMXH&5q69wX00=j&n4d+7W?8Li*obeo^GHc z5Bdi1XhYiQOdGT_RO!+8)&2%Mnk7;I=jn>3BtKT*s~7rGOP<>k4&?{^iemP zM^)L;e#uJk=MIztiu|2vOX*XN5TV~#>jVo7^!Vmn)yK`^fRd3dJm$(_DJ<(6FEutC zdL`ky+Q&KbHny#Mf|ODm^%L!HA5{SWzLe;YIQi6Lp;(Tq*YAYae2bxZx6Xad4d=dm z%qMs*5}Bcd4*3c|5v2aQ&t- zjfC5J_WRgjUm|}zRg|cJnsr~YfX?%P_)s&&q<^l@gB+cjY)&U9!Q#*5SrB$Zv3ShDd;$6XQ!a=ah}MkWga7lybpn=dcAEQ__rCOYj3gF6wnlr5K@ zq8J}I>$2#VSRN-Qe45^_32pRDANY1Zz4c@>b+o@y%;|`3;M&IMM%Bfquam{f4z<&k zm0!4BZBKoq-@Th@xw_&Z$ve5_PCU}$?%^(D!XX>bDglv}Zh7~l@!2r~d}oT^XVQ;v zef*-$75<$%MgLVg`B2vj6f^Qg}pC|PcZ`h+4%==rUxKTsn^zg#-u*%IU&zJZ}_lE#f>zwJk?vCD|EvND k3UH$S&!YeT!n=oaCoa@Dbc2Nj{m-e?l(ZB}<;{Zr3w<8mzW@LL From 9b9616997166a9510424274e2793db8a3bf02d38 Mon Sep 17 00:00:00 2001 From: Arista Jenkins Date: Tue, 1 Dec 2020 17:28:06 -0800 Subject: [PATCH 08/12] Remove fabric HLD from everflow branch --- doc/voq/fabric.md | 189 ---------------------------------------------- 1 file changed, 189 deletions(-) delete mode 100644 doc/voq/fabric.md diff --git a/doc/voq/fabric.md b/doc/voq/fabric.md deleted file mode 100644 index 2dc52fb286..0000000000 --- a/doc/voq/fabric.md +++ /dev/null @@ -1,189 +0,0 @@ -# Fabric port support on Sonic - -# High Level Design Document -#### Rev 1 - -# Table of Contents -* [List of Tables](#list-of-tables) -* [List of Figures](#list-of-figures) -* [Revision](#revision) -* [About this Manual](#about-this-manual) -* [Scope](#scope) -* [Definitions/Abbreviations](#definitionsabbreviations) -* [1 Requirements](#1-requirements) -* [2 Design](#2-design) -* [3 Testing](#3-testing) -* [4 Future Work](#4-future-work) - -# List of Tables -* [Table 1: Abbreviations](#definitionsabbreviations) - -# List of Figures - -# Revision -| Rev | Date | Author | Change Description | -|:---:|:-----------:|:------------------:|--------------------| -| 1 | Aug-28 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Initial Version | -| 1.1 | Sep-1 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Add hotswap handling | -| 2 | Oct-20 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Update counter information | -| 2.1 | Nov-17 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Minor update on container starts | - -# About this Manual - -This document provides an overview of the SONiC support for fabric ports that are present in a VOQ-based chassis. These fabric ports are used to interconnect the forwarding Network Processing Units within the VOQ chassis. - -# Scope - -This document covers: - -- Bring up of fabric ports in a VOQ chassis. -- Monitoring the fabric ports in forwarding and fabric chips. - -This document builds on top of the VOQ chassis architecture discussed [here](https://github.com/Azure/SONiC/blob/master/doc/voq/architecture.md) and the multi-ASIC architecture discussed [here](https://github.com/Azure/SONiC/blob/2f320430c8199132c686c06b5431ab93a86fb98f/doc/multi_asic/SONiC_multi_asic_hld.md). - -# Definitions/Abbreviations - -| | | | -|------|--------------------|--------------------------------| -| SSI | Supervisor SONiC Instance | SONiC OS instance on a central supervisor module that controls a cluster of forwarding instances and the interconnection fabric. | -| NPU | Network Processing Unit | Refers to the forwarding engine on a device that is responsible for packet forwarding. | -| ASIC | Application Specific Integrated Circuit | In addition to NPUs, also includes fabric chips that could forward packets or cells. | -| cell | Fabric Data Units | The data units that traverse a cell-based chassis fabric. | - -# 1 Requirements - -Fabric ports are used in systems in which there are multiple forwarding ASICs are required to be connected. Traffic passes from one front panel port in a forwarding ASIC over a fabric network to one or multiple front panel ports on one or other ASICs. The fabric network is formed using fabric ASICs. Fabric links on the fabric network connect fabric ports on forwarding ASICs to fabric ports on fabric ASICs. - -High level requirements: - -- SONiC needs to form a fabric network among forwarding ASICs, monitor and manage it. Monitoring could include link statistics, error monitoring and reporting, etc. -- SONiC should be able to initialize fabric asics and manage them similar to how forwarding ASICs are managed - using syncd and sairedis calls. - -# 2 Design - -## 2.1 Fabric ASICs - -Fabric asics are used to form a fabric network for connecting forwarding ASICs. For each fabric port on a forwarding ASIC, there is a fabric link in the fabric network connecting to a fabric port on a fabric ASIC. There are typically multiple fabric links between a pair of (NPU, fabric ASIC) to balance traffic. We use the same approach to initializing and managing fabric ASICs as we are doing today for forwarding ASICs. A typical chassis implementation will be to manage all the fabric ASICs in a chassis from the control card or the Supervior Sonic Instance (SSI). We will leverage the work done in the multi-ASIC HLD and instantiate groups of containers for the fabric ASICs. - -For each fabric ASIC, there will be: - -- Database container -- Swss container -- Syncd container - -Unlike forwarding ASICs, fabric ASICs do not have any front panel ports, but only fabric ports. So all the front panel port related containers like lldp, teamd and bgpd can be disabled for fabric ASICs. - -## 2.2 Database Schemas - -``` -DEVICE_METADATA|localhost: { - "switch_type": “fabric” - "switch_id": {{switch_id}} -} -``` - -Each fabric ASIC must be assigned a unique switch_id. The SAI VOQ specification recommends that this number be assigned to be different than the switch_id assigned to the forwarding ASICs in the chassis. - -Fabric port status will be polled periodically and stored in table STATE_DB|FABRIC_PORT_TABLE. Typically, fabric port status about a fabric port includes: - -- Status: Up or down -- If port is down, we may have some more information indicating reason e.g. CRC or misaligned -- If port is up, we should know remote peer information including peer switch_id and peer fabric port. - -``` -STATE_DB:FABRIC_PORT_TABLE:{{fabric_port_name}} - "lane": {{number}} - "status": “up|down” - "crc": “yes” # if status: down - "misaligned": “yes” # if status: down - "remote_switch_id": {{number}} # if status: up - "remote_lane": {{number}} # if status: up -``` - -Fabric port statistics include the following port counters: - -``` - SAI_PORT_STAT_IF_IN_OCTETS, - SAI_PORT_STAT_IF_IN_ERRORS, - SAI_PORT_STAT_IF_IN_FABRIC_DATA_UNITS, - SAI_PORT_STAT_IF_IN_FEC_CORRECTABLE_FRAMES, - SAI_PORT_STAT_IF_IN_FEC_NOT_CORRECTABLE_FRAMES, - SAI_PORT_STAT_IF_IN_FEC_SYMBOL_ERRORS, - SAI_PORT_STAT_IF_OUT_OCTETS, - SAI_PORT_STAT_IF_OUT_FABRIC_DATA_UNITS -``` - -FabricPortsOrch defines the port counters in FLEX_COUNTER_DB and syncd's existing FlexCounters thread periodically collects and saves these counters in COUNTER_DB. “show” cli commands read COUNTER_DB and display statistics information. - -Fabric port also has a couple of queue counters. Similar to the port counters, the queue counters are also polled with FLEX_COUNTER_DB. -``` - SAI_QUEUE_STAT_WATERMARK_LEVEL, - SAI_QUEUE_STAT_CURR_OCCUPANCY_BYTES, - SAI_QUEUE_STAT_CURR_OCCUPANCY_LEVEL -``` - -Note that Linecard Sonic instances will also have STATE_DB|FABRIC_PORT_TABLE as well as port/queue counters because there are fabric ports in forwarding ASICs as well. - -## 2.3 System Initialization - -As part of multi-ASIC support, /etc/sonic/generated_services.conf contains the list of services which will be created for each asic when the system boots up. This is read by systemd-sonic-generator to generate the service files for each container that needs to run. - -Since the fabric ASIC doesn’t need lldp, bgpd and teamd containers to run, systemd-sonic-generator will be modified to not start these services for the fabric ASICs. A per-platform file called `asic_disabled_services` can list the services that are not needed for a given ASIC and systemd-sonic-generator will not generate the service files for these containers. For example, -``` -0,lldp,teamd,bgp -1,lldp,teamd,bgp -2,lldp,teamd,bgp -``` -will not start lldp, teamd and bgp containers for ASICs 0, 1 and 2. - -NOTE: Longer term, we would like to use the FEATURE table to control which containers need to be started for fabric chips. However, that requires multi-ASIC support for the FEATURE table. This will be pursued as a separate project. - -## 2.4 Fabric Card Hotswap - -PMON will be responsible for detecting card presence and hotswap events using the get_change_event API. A new systemd service will be responsbile for turning on/off the service files for the syncd, database and swss containers that manage each fabric ASIC. When the fabric card is removed, the containers that manage the fabric ASICs that are part of that fabric card will be stopped. These will be re-started when the fabric card is inserted later. - -## 2.5 Orchagent - -Orchagent creates the switch using the SAI API similar to creating the switch for a forwarding ASIC, except that the switch type will be fabric. When the ASIC is initialized, all the fabric ports are initialized by default. The fabric ports are a subtype of SAI Port object and it can be obtained by getting all the fabric port objects from SAI. Since there are no front panel ports on a fabric ASIC, port_config.ini will be empty and portsyncd will not run. - -On fabric ASICs, OrchDaemon will only monitor and manage fabric ports. It will not maintain cpu port and front panel port related ochres, such as PortsOrch, IntfsOrch, NeighborOrch, VnetOrch, QosOrch, TunnelOrch, and etc. To simplify the change, we will just create FabricOrchDaemon inheriting OrchDaemon for fabric ASICs and this will only run FabricPortsOrch, the module responsible for managing fabric ports. - -## 2.6 Fabric Ports in Forwarding ASICs - -When a forwarding ASIC is initialized, the fabric ports are initialized by default by SAI. Orchagent will run FabricPortsOrch in addition to all the other orchs that needs to be run to manage the forwarding ASIC. Fabric port monitoring and handling is identical to what happens on a Fabric ASIC. - -## 2.7 Cli commands - -``` -> show fabric counters -n [port_id] - -asic2 fabric port counter (number of fabric ports: 192) - -PORT RxCells TxCells Crc Fec Corrected -------------------------------------------------------------------------- - 0 : 71660578 2 0 0 0 - 1 : 71659798 1 0 0 213 - 2 : 0 1 0 0 167 - 3 : 0 2 0 0 193 -``` - -### 2.7.1 Fabric Status - -In a later phase, a `show fabric status` command will be added to show the remote switch ID and link ID for each fabric link of an ASIC. This will be obtained from the SAI_PORT_ATTR_FABRIC_REACHABILITY port attribute of the fabric port. Note that for fabric links that do not have a link partner because of the configuration of the chassis, this will show the status as `down`. The status will also be `down` for fabric links that are down due to some other physical error. To identify links that are down due to error vs links that are not expected to be up because of the chassis connectivity, we need to build up a list of expected fabric connectivity for each ASIC. This can be computed ahead of time based on the vendor configuration and populated in the minigraph. This will be implemented in a later phase. - -# 3 Testing - -Fabric port testing will rely on sonic-mgmt tests that can run on chassis hardware. - -- Test fabric port mapping: To verify the fabric mapping, we can inspect the remote switch ID that are saved in the STATE_DB and match that with the known chassis architecture. - -- Test traffic and counters: Send traffic through the chassis and verify traffic going through fabric ports via counters. - -# 4 Future Work - -- In this proposal, all fabric ports on fabric ASICs or forwarding ASICs that join to form the fabric network will be enabled even when there are no peer ports available. We could provide a config model for the platforms to express the expected fabric connectivity and turn off unnecessary fabric ports. - -- Fabric ports that do not have a peer port will show up as a ‘down’ port. Fabric ports that do have a peer port could also go ‘down’ and there is no current way to differentiate this from a fabric port that does not have a peer port. This can be detected if the config model can express the expected fabric connectivity. - -- Monitor, detect and disable fabric ports that consistently show errors. - From 4db916ea3ad6893b8ddf0ef3e43b3470a3f216c3 Mon Sep 17 00:00:00 2001 From: Eswaran Baskaran Date: Mon, 29 Mar 2021 16:54:26 -0700 Subject: [PATCH 09/12] Add everflow HLD --- doc/voq/everflow.md | 189 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100644 doc/voq/everflow.md diff --git a/doc/voq/everflow.md b/doc/voq/everflow.md new file mode 100644 index 0000000000..2dc52fb286 --- /dev/null +++ b/doc/voq/everflow.md @@ -0,0 +1,189 @@ +# Fabric port support on Sonic + +# High Level Design Document +#### Rev 1 + +# Table of Contents +* [List of Tables](#list-of-tables) +* [List of Figures](#list-of-figures) +* [Revision](#revision) +* [About this Manual](#about-this-manual) +* [Scope](#scope) +* [Definitions/Abbreviations](#definitionsabbreviations) +* [1 Requirements](#1-requirements) +* [2 Design](#2-design) +* [3 Testing](#3-testing) +* [4 Future Work](#4-future-work) + +# List of Tables +* [Table 1: Abbreviations](#definitionsabbreviations) + +# List of Figures + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|--------------------| +| 1 | Aug-28 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Initial Version | +| 1.1 | Sep-1 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Add hotswap handling | +| 2 | Oct-20 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Update counter information | +| 2.1 | Nov-17 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Minor update on container starts | + +# About this Manual + +This document provides an overview of the SONiC support for fabric ports that are present in a VOQ-based chassis. These fabric ports are used to interconnect the forwarding Network Processing Units within the VOQ chassis. + +# Scope + +This document covers: + +- Bring up of fabric ports in a VOQ chassis. +- Monitoring the fabric ports in forwarding and fabric chips. + +This document builds on top of the VOQ chassis architecture discussed [here](https://github.com/Azure/SONiC/blob/master/doc/voq/architecture.md) and the multi-ASIC architecture discussed [here](https://github.com/Azure/SONiC/blob/2f320430c8199132c686c06b5431ab93a86fb98f/doc/multi_asic/SONiC_multi_asic_hld.md). + +# Definitions/Abbreviations + +| | | | +|------|--------------------|--------------------------------| +| SSI | Supervisor SONiC Instance | SONiC OS instance on a central supervisor module that controls a cluster of forwarding instances and the interconnection fabric. | +| NPU | Network Processing Unit | Refers to the forwarding engine on a device that is responsible for packet forwarding. | +| ASIC | Application Specific Integrated Circuit | In addition to NPUs, also includes fabric chips that could forward packets or cells. | +| cell | Fabric Data Units | The data units that traverse a cell-based chassis fabric. | + +# 1 Requirements + +Fabric ports are used in systems in which there are multiple forwarding ASICs are required to be connected. Traffic passes from one front panel port in a forwarding ASIC over a fabric network to one or multiple front panel ports on one or other ASICs. The fabric network is formed using fabric ASICs. Fabric links on the fabric network connect fabric ports on forwarding ASICs to fabric ports on fabric ASICs. + +High level requirements: + +- SONiC needs to form a fabric network among forwarding ASICs, monitor and manage it. Monitoring could include link statistics, error monitoring and reporting, etc. +- SONiC should be able to initialize fabric asics and manage them similar to how forwarding ASICs are managed - using syncd and sairedis calls. + +# 2 Design + +## 2.1 Fabric ASICs + +Fabric asics are used to form a fabric network for connecting forwarding ASICs. For each fabric port on a forwarding ASIC, there is a fabric link in the fabric network connecting to a fabric port on a fabric ASIC. There are typically multiple fabric links between a pair of (NPU, fabric ASIC) to balance traffic. We use the same approach to initializing and managing fabric ASICs as we are doing today for forwarding ASICs. A typical chassis implementation will be to manage all the fabric ASICs in a chassis from the control card or the Supervior Sonic Instance (SSI). We will leverage the work done in the multi-ASIC HLD and instantiate groups of containers for the fabric ASICs. + +For each fabric ASIC, there will be: + +- Database container +- Swss container +- Syncd container + +Unlike forwarding ASICs, fabric ASICs do not have any front panel ports, but only fabric ports. So all the front panel port related containers like lldp, teamd and bgpd can be disabled for fabric ASICs. + +## 2.2 Database Schemas + +``` +DEVICE_METADATA|localhost: { + "switch_type": “fabric” + "switch_id": {{switch_id}} +} +``` + +Each fabric ASIC must be assigned a unique switch_id. The SAI VOQ specification recommends that this number be assigned to be different than the switch_id assigned to the forwarding ASICs in the chassis. + +Fabric port status will be polled periodically and stored in table STATE_DB|FABRIC_PORT_TABLE. Typically, fabric port status about a fabric port includes: + +- Status: Up or down +- If port is down, we may have some more information indicating reason e.g. CRC or misaligned +- If port is up, we should know remote peer information including peer switch_id and peer fabric port. + +``` +STATE_DB:FABRIC_PORT_TABLE:{{fabric_port_name}} + "lane": {{number}} + "status": “up|down” + "crc": “yes” # if status: down + "misaligned": “yes” # if status: down + "remote_switch_id": {{number}} # if status: up + "remote_lane": {{number}} # if status: up +``` + +Fabric port statistics include the following port counters: + +``` + SAI_PORT_STAT_IF_IN_OCTETS, + SAI_PORT_STAT_IF_IN_ERRORS, + SAI_PORT_STAT_IF_IN_FABRIC_DATA_UNITS, + SAI_PORT_STAT_IF_IN_FEC_CORRECTABLE_FRAMES, + SAI_PORT_STAT_IF_IN_FEC_NOT_CORRECTABLE_FRAMES, + SAI_PORT_STAT_IF_IN_FEC_SYMBOL_ERRORS, + SAI_PORT_STAT_IF_OUT_OCTETS, + SAI_PORT_STAT_IF_OUT_FABRIC_DATA_UNITS +``` + +FabricPortsOrch defines the port counters in FLEX_COUNTER_DB and syncd's existing FlexCounters thread periodically collects and saves these counters in COUNTER_DB. “show” cli commands read COUNTER_DB and display statistics information. + +Fabric port also has a couple of queue counters. Similar to the port counters, the queue counters are also polled with FLEX_COUNTER_DB. +``` + SAI_QUEUE_STAT_WATERMARK_LEVEL, + SAI_QUEUE_STAT_CURR_OCCUPANCY_BYTES, + SAI_QUEUE_STAT_CURR_OCCUPANCY_LEVEL +``` + +Note that Linecard Sonic instances will also have STATE_DB|FABRIC_PORT_TABLE as well as port/queue counters because there are fabric ports in forwarding ASICs as well. + +## 2.3 System Initialization + +As part of multi-ASIC support, /etc/sonic/generated_services.conf contains the list of services which will be created for each asic when the system boots up. This is read by systemd-sonic-generator to generate the service files for each container that needs to run. + +Since the fabric ASIC doesn’t need lldp, bgpd and teamd containers to run, systemd-sonic-generator will be modified to not start these services for the fabric ASICs. A per-platform file called `asic_disabled_services` can list the services that are not needed for a given ASIC and systemd-sonic-generator will not generate the service files for these containers. For example, +``` +0,lldp,teamd,bgp +1,lldp,teamd,bgp +2,lldp,teamd,bgp +``` +will not start lldp, teamd and bgp containers for ASICs 0, 1 and 2. + +NOTE: Longer term, we would like to use the FEATURE table to control which containers need to be started for fabric chips. However, that requires multi-ASIC support for the FEATURE table. This will be pursued as a separate project. + +## 2.4 Fabric Card Hotswap + +PMON will be responsible for detecting card presence and hotswap events using the get_change_event API. A new systemd service will be responsbile for turning on/off the service files for the syncd, database and swss containers that manage each fabric ASIC. When the fabric card is removed, the containers that manage the fabric ASICs that are part of that fabric card will be stopped. These will be re-started when the fabric card is inserted later. + +## 2.5 Orchagent + +Orchagent creates the switch using the SAI API similar to creating the switch for a forwarding ASIC, except that the switch type will be fabric. When the ASIC is initialized, all the fabric ports are initialized by default. The fabric ports are a subtype of SAI Port object and it can be obtained by getting all the fabric port objects from SAI. Since there are no front panel ports on a fabric ASIC, port_config.ini will be empty and portsyncd will not run. + +On fabric ASICs, OrchDaemon will only monitor and manage fabric ports. It will not maintain cpu port and front panel port related ochres, such as PortsOrch, IntfsOrch, NeighborOrch, VnetOrch, QosOrch, TunnelOrch, and etc. To simplify the change, we will just create FabricOrchDaemon inheriting OrchDaemon for fabric ASICs and this will only run FabricPortsOrch, the module responsible for managing fabric ports. + +## 2.6 Fabric Ports in Forwarding ASICs + +When a forwarding ASIC is initialized, the fabric ports are initialized by default by SAI. Orchagent will run FabricPortsOrch in addition to all the other orchs that needs to be run to manage the forwarding ASIC. Fabric port monitoring and handling is identical to what happens on a Fabric ASIC. + +## 2.7 Cli commands + +``` +> show fabric counters -n [port_id] + +asic2 fabric port counter (number of fabric ports: 192) + +PORT RxCells TxCells Crc Fec Corrected +------------------------------------------------------------------------- + 0 : 71660578 2 0 0 0 + 1 : 71659798 1 0 0 213 + 2 : 0 1 0 0 167 + 3 : 0 2 0 0 193 +``` + +### 2.7.1 Fabric Status + +In a later phase, a `show fabric status` command will be added to show the remote switch ID and link ID for each fabric link of an ASIC. This will be obtained from the SAI_PORT_ATTR_FABRIC_REACHABILITY port attribute of the fabric port. Note that for fabric links that do not have a link partner because of the configuration of the chassis, this will show the status as `down`. The status will also be `down` for fabric links that are down due to some other physical error. To identify links that are down due to error vs links that are not expected to be up because of the chassis connectivity, we need to build up a list of expected fabric connectivity for each ASIC. This can be computed ahead of time based on the vendor configuration and populated in the minigraph. This will be implemented in a later phase. + +# 3 Testing + +Fabric port testing will rely on sonic-mgmt tests that can run on chassis hardware. + +- Test fabric port mapping: To verify the fabric mapping, we can inspect the remote switch ID that are saved in the STATE_DB and match that with the known chassis architecture. + +- Test traffic and counters: Send traffic through the chassis and verify traffic going through fabric ports via counters. + +# 4 Future Work + +- In this proposal, all fabric ports on fabric ASICs or forwarding ASICs that join to form the fabric network will be enabled even when there are no peer ports available. We could provide a config model for the platforms to express the expected fabric connectivity and turn off unnecessary fabric ports. + +- Fabric ports that do not have a peer port will show up as a ‘down’ port. Fabric ports that do have a peer port could also go ‘down’ and there is no current way to differentiate this from a fabric port that does not have a peer port. This can be detected if the config model can express the expected fabric connectivity. + +- Monitor, detect and disable fabric ports that consistently show errors. + From 4457731fd9dbd1de643fd23cc78d68f7d12d96dc Mon Sep 17 00:00:00 2001 From: Song Yuan Date: Fri, 7 May 2021 17:51:05 -0700 Subject: [PATCH 10/12] Recover Everflow HLD that was accidentally replaced by fabric port HLD --- doc/voq/everflow.md | 159 ++++++++------------------------------------ 1 file changed, 28 insertions(+), 131 deletions(-) diff --git a/doc/voq/everflow.md b/doc/voq/everflow.md index 2dc52fb286..fbf2060b58 100644 --- a/doc/voq/everflow.md +++ b/doc/voq/everflow.md @@ -1,4 +1,4 @@ -# Fabric port support on Sonic +# Everflow support on VOQ Chassis # High Level Design Document #### Rev 1 @@ -23,167 +23,64 @@ # Revision | Rev | Date | Author | Change Description | |:---:|:-----------:|:------------------:|--------------------| -| 1 | Aug-28 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Initial Version | -| 1.1 | Sep-1 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Add hotswap handling | -| 2 | Oct-20 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Update counter information | -| 2.1 | Nov-17 2020 | Ngoc Do, Eswaran Baskaran (Arista Networks) | Minor update on container starts | +| 1 | Dec-1 2020 | Song Yuan, Eswaran Baskaran (Arista Networks) | Initial Version | # About this Manual -This document provides an overview of the SONiC support for fabric ports that are present in a VOQ-based chassis. These fabric ports are used to interconnect the forwarding Network Processing Units within the VOQ chassis. +This document provides an overview of the SONiC support for everflow configuration in a VOQ-based chassis. In a VOQ-based chassis, the everflow configuration is applied on a linecard and the mirror destination could be in a different linecard. We propose a solution where the packet is rewritten with the GRE header in the ingress linecard before it’s sent over to the mirror destination. # Scope -This document covers: - -- Bring up of fabric ports in a VOQ chassis. -- Monitoring the fabric ports in forwarding and fabric chips. - -This document builds on top of the VOQ chassis architecture discussed [here](https://github.com/Azure/SONiC/blob/master/doc/voq/architecture.md) and the multi-ASIC architecture discussed [here](https://github.com/Azure/SONiC/blob/2f320430c8199132c686c06b5431ab93a86fb98f/doc/multi_asic/SONiC_multi_asic_hld.md). +This document builds on top of the VOQ chassis architecture discussed [here](https://github.com/Azure/SONiC/blob/master/doc/voq/architecture.md) and the Everflow design discussed [here](https://github.com/Azure/SONiC/wiki/Everflow-High-Level-Design#3138-mirror-api). The goal of this document is to describe the configuration and design for everflow mirroring sessions for a VOQ-based chassis. # Definitions/Abbreviations | | | | |------|--------------------|--------------------------------| -| SSI | Supervisor SONiC Instance | SONiC OS instance on a central supervisor module that controls a cluster of forwarding instances and the interconnection fabric. | -| NPU | Network Processing Unit | Refers to the forwarding engine on a device that is responsible for packet forwarding. | -| ASIC | Application Specific Integrated Circuit | In addition to NPUs, also includes fabric chips that could forward packets or cells. | -| cell | Fabric Data Units | The data units that traverse a cell-based chassis fabric. | +| FSI | Forwarding SONiC Instance | SONiC OS instance on a linecard module that controls one or more forwarding ASICs. | +| ASIC | Application Specific Integrated Circuit | Refers to the forwarding engine on a device that is responsible for packet forwarding. | +| ERSPAN | Encapsulated Remote Switched Port ANalyzer | Another name for Everflow mirroring sessions | # 1 Requirements -Fabric ports are used in systems in which there are multiple forwarding ASICs are required to be connected. Traffic passes from one front panel port in a forwarding ASIC over a fabric network to one or multiple front panel ports on one or other ASICs. The fabric network is formed using fabric ASICs. Fabric links on the fabric network connect fabric ports on forwarding ASICs to fabric ports on fabric ASICs. - -High level requirements: - -- SONiC needs to form a fabric network among forwarding ASICs, monitor and manage it. Monitoring could include link statistics, error monitoring and reporting, etc. -- SONiC should be able to initialize fabric asics and manage them similar to how forwarding ASICs are managed - using syncd and sairedis calls. +In a VOQ based chassis, the mirror source and destination ports could be on different linecards. This configuration must be accepted and the packet that goes out of the destination port must have the correct encapsulation header based on the configuration. # 2 Design -## 2.1 Fabric ASICs - -Fabric asics are used to form a fabric network for connecting forwarding ASICs. For each fabric port on a forwarding ASIC, there is a fabric link in the fabric network connecting to a fabric port on a fabric ASIC. There are typically multiple fabric links between a pair of (NPU, fabric ASIC) to balance traffic. We use the same approach to initializing and managing fabric ASICs as we are doing today for forwarding ASICs. A typical chassis implementation will be to manage all the fabric ASICs in a chassis from the control card or the Supervior Sonic Instance (SSI). We will leverage the work done in the multi-ASIC HLD and instantiate groups of containers for the fabric ASICs. - -For each fabric ASIC, there will be: - -- Database container -- Swss container -- Syncd container - -Unlike forwarding ASICs, fabric ASICs do not have any front panel ports, but only fabric ports. So all the front panel port related containers like lldp, teamd and bgpd can be disabled for fabric ASICs. - -## 2.2 Database Schemas - -``` -DEVICE_METADATA|localhost: { - "switch_type": “fabric” - "switch_id": {{switch_id}} -} -``` - -Each fabric ASIC must be assigned a unique switch_id. The SAI VOQ specification recommends that this number be assigned to be different than the switch_id assigned to the forwarding ASICs in the chassis. - -Fabric port status will be polled periodically and stored in table STATE_DB|FABRIC_PORT_TABLE. Typically, fabric port status about a fabric port includes: - -- Status: Up or down -- If port is down, we may have some more information indicating reason e.g. CRC or misaligned -- If port is up, we should know remote peer information including peer switch_id and peer fabric port. - -``` -STATE_DB:FABRIC_PORT_TABLE:{{fabric_port_name}} - "lane": {{number}} - "status": “up|down” - "crc": “yes” # if status: down - "misaligned": “yes” # if status: down - "remote_switch_id": {{number}} # if status: up - "remote_lane": {{number}} # if status: up -``` - -Fabric port statistics include the following port counters: - -``` - SAI_PORT_STAT_IF_IN_OCTETS, - SAI_PORT_STAT_IF_IN_ERRORS, - SAI_PORT_STAT_IF_IN_FABRIC_DATA_UNITS, - SAI_PORT_STAT_IF_IN_FEC_CORRECTABLE_FRAMES, - SAI_PORT_STAT_IF_IN_FEC_NOT_CORRECTABLE_FRAMES, - SAI_PORT_STAT_IF_IN_FEC_SYMBOL_ERRORS, - SAI_PORT_STAT_IF_OUT_OCTETS, - SAI_PORT_STAT_IF_OUT_FABRIC_DATA_UNITS -``` +## 2.1 Problem -FabricPortsOrch defines the port counters in FLEX_COUNTER_DB and syncd's existing FlexCounters thread periodically collects and saves these counters in COUNTER_DB. “show” cli commands read COUNTER_DB and display statistics information. +In a regular non-VOQ switch, given the destination IP of the mirror session, mirrororch is responsible for determining the outgoing interface and the destination MAC address by querying routeorch and neighorch. Once these details are determined, mirrororch makes the SAI create_mirror_session API call to create the mirror session. For example, the destination port is set in `SAI_MIRROR_SESSION_ATTR_MONITOR_PORT`, the destination mac for the packet is set in `SAI_MIRROR_SESSION_ATTR_DST_MAC_ADDRESS` and so on. -Fabric port also has a couple of queue counters. Similar to the port counters, the queue counters are also polled with FLEX_COUNTER_DB. -``` - SAI_QUEUE_STAT_WATERMARK_LEVEL, - SAI_QUEUE_STAT_CURR_OCCUPANCY_BYTES, - SAI_QUEUE_STAT_CURR_OCCUPANCY_LEVEL -``` +In addition, mirrororch is also responsible for reacting to changes in the route or nexthop and update the session by making appropriate SAI update calls. -Note that Linecard Sonic instances will also have STATE_DB|FABRIC_PORT_TABLE as well as port/queue counters because there are fabric ports in forwarding ASICs as well. +In a VOQ chassis, the monitor port could be on another linecard and packet rewrite is done in the egress pipeline of that linecard. As the ports of other linecards are only represented as SYSTEM_PORTs in each FSI, we need either changes in SAI or changes in SONiC to support everflow. -## 2.3 System Initialization +### 2.1.1 Proposal -As part of multi-ASIC support, /etc/sonic/generated_services.conf contains the list of services which will be created for each asic when the system boots up. This is read by systemd-sonic-generator to generate the service files for each container that needs to run. +The idea is to fully rewrite the packet in the ingress chip and send it out of a recycle port in the ingress ASIC. After the recycle, the packet goes to the egress ASIC where the packet gets switched out of the mirror destination port. -Since the fabric ASIC doesn’t need lldp, bgpd and teamd containers to run, systemd-sonic-generator will be modified to not start these services for the fabric ASICs. A per-platform file called `asic_disabled_services` can list the services that are not needed for a given ASIC and systemd-sonic-generator will not generate the service files for these containers. For example, -``` -0,lldp,teamd,bgp -1,lldp,teamd,bgp -2,lldp,teamd,bgp -``` -will not start lldp, teamd and bgp containers for ASICs 0, 1 and 2. +![](../../images/voq/Everflow_voq.png) -NOTE: Longer term, we would like to use the FEATURE table to control which containers need to be started for fabric chips. However, that requires multi-ASIC support for the FEATURE table. This will be pursued as a separate project. +The mechanism to enable the recycle port is outside the scope of this document. -## 2.4 Fabric Card Hotswap +## 2.2 Option 1 - Implicit Recycle -PMON will be responsible for detecting card presence and hotswap events using the get_change_event API. A new systemd service will be responsbile for turning on/off the service files for the syncd, database and swss containers that manage each fabric ASIC. When the fabric card is removed, the containers that manage the fabric ASICs that are part of that fabric card will be stopped. These will be re-started when the fabric card is inserted later. +This option can be implemented with minimal changes to SONiC because the bulk of the handling is done inside SAI. The SAI implementation can be enhanced to accept SYSTEM_PORTs as arguments for the `SAI_MIRROR_SESSION_ATTR_MONITOR_PORT` attribute. When a mirror session is created with a destination IP, mirrororch resolves the IP using routeorch and neighorch as before. The following changes are proposed for the VOQ chassis. +Upon receiving mirrororch’s request for a remote neighbor’s information, neighorch must be able to return neighbor information like interface alias. +If the neighbor information returned from neighorch turns out to be a remote neighbor, mirrororch needs to use the SYSTEM_PORT of the remote interface as the mirror destination port. -## 2.5 Orchagent +SAI implementation will program the hardware to send the mirrored packets in this session out of the recycle port. When the packet re-enters the pipeline, the destination MAC of this packet will be the neighbor’s destination MAC address and it will be bridged in the ingress ASIC to the correct egress destination port. SAI will also be responsible for programming the FDB entry in the ingress ASIC appropriately. -Orchagent creates the switch using the SAI API similar to creating the switch for a forwarding ASIC, except that the switch type will be fabric. When the ASIC is initialized, all the fabric ports are initialized by default. The fabric ports are a subtype of SAI Port object and it can be obtained by getting all the fabric port objects from SAI. Since there are no front panel ports on a fabric ASIC, port_config.ini will be empty and portsyncd will not run. +## 2.3 Option 2 - Explicit Recycle -On fabric ASICs, OrchDaemon will only monitor and manage fabric ports. It will not maintain cpu port and front panel port related ochres, such as PortsOrch, IntfsOrch, NeighborOrch, VnetOrch, QosOrch, TunnelOrch, and etc. To simplify the change, we will just create FabricOrchDaemon inheriting OrchDaemon for fabric ASICs and this will only run FabricPortsOrch, the module responsible for managing fabric ports. +This option can be implemented with minimal changes to SAI as long as a recycle port can be created. The recycle port needs to be enabled as an L3 port for this mechanism to work because it requires the packet to be routed after recycle. The following changes are proposed. +Upon receiving mirrororch’s request for a remote neighbor’s information, neighorch must be able to return neighbor information like interface alias. +If the neighbor information returned from neighorch turns out to be a remote neighbor, mirrororch needs to use the recycle port as the mirror destination port and the router mac as the destination MAC address. -## 2.6 Fabric Ports in Forwarding ASICs - -When a forwarding ASIC is initialized, the fabric ports are initialized by default by SAI. Orchagent will run FabricPortsOrch in addition to all the other orchs that needs to be run to manage the forwarding ASIC. Fabric port monitoring and handling is identical to what happens on a Fabric ASIC. - -## 2.7 Cli commands - -``` -> show fabric counters -n [port_id] - -asic2 fabric port counter (number of fabric ports: 192) - -PORT RxCells TxCells Crc Fec Corrected -------------------------------------------------------------------------- - 0 : 71660578 2 0 0 0 - 1 : 71659798 1 0 0 213 - 2 : 0 1 0 0 167 - 3 : 0 2 0 0 193 -``` - -### 2.7.1 Fabric Status - -In a later phase, a `show fabric status` command will be added to show the remote switch ID and link ID for each fabric link of an ASIC. This will be obtained from the SAI_PORT_ATTR_FABRIC_REACHABILITY port attribute of the fabric port. Note that for fabric links that do not have a link partner because of the configuration of the chassis, this will show the status as `down`. The status will also be `down` for fabric links that are down due to some other physical error. To identify links that are down due to error vs links that are not expected to be up because of the chassis connectivity, we need to build up a list of expected fabric connectivity for each ASIC. This can be computed ahead of time based on the vendor configuration and populated in the minigraph. This will be implemented in a later phase. +SAI implementation is the same as a single-chip system. The mirrored packets are rewritten in the ingress ASIC and sent out of the recycle port. When the packet re-enters the pipeline, it gets routed because the destination MAC is the router MAC. The destination IP of this packet is the destination IP configured on the mirror session, so it will be routed by the ingress ASIC. If the destination IP is reachable via multiple nexthops, the ingress ASIC will loadbalance the traffic as needed. # 3 Testing - -Fabric port testing will rely on sonic-mgmt tests that can run on chassis hardware. - -- Test fabric port mapping: To verify the fabric mapping, we can inspect the remote switch ID that are saved in the STATE_DB and match that with the known chassis architecture. - -- Test traffic and counters: Send traffic through the chassis and verify traffic going through fabric ports via counters. +TBD # 4 Future Work - -- In this proposal, all fabric ports on fabric ASICs or forwarding ASICs that join to form the fabric network will be enabled even when there are no peer ports available. We could provide a config model for the platforms to express the expected fabric connectivity and turn off unnecessary fabric ports. - -- Fabric ports that do not have a peer port will show up as a ‘down’ port. Fabric ports that do have a peer port could also go ‘down’ and there is no current way to differentiate this from a fabric port that does not have a peer port. This can be detected if the config model can express the expected fabric connectivity. - -- Monitor, detect and disable fabric ports that consistently show errors. - +TBD From 605687e3cd760bbda5f043c04925b7d77ce2faa2 Mon Sep 17 00:00:00 2001 From: Song Yuan Date: Fri, 7 May 2021 18:00:46 -0700 Subject: [PATCH 11/12] Add figure --- images/voq/Everflow_voq.png | Bin 0 -> 27807 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 images/voq/Everflow_voq.png diff --git a/images/voq/Everflow_voq.png b/images/voq/Everflow_voq.png new file mode 100644 index 0000000000000000000000000000000000000000..6965437c0383b8d8f4a768e0043c3e6b6d1b2a54 GIT binary patch literal 27807 zcmeGERX`nE6E%tgfdGL30fGc~_uwuG?zV7ucXx;2?(XjH?(Xic!SyV*WPkrT=RV(u zd!b?V>N#un^sX8;s)M8?1V4ZJ_6Y<87lfD8x-SO5qJs0h?Y;1!GDo3Fqxyhi-| zQo{WFxKh>@`bMUDARwfXI+~h@LX@NfT3VW#1EUmVpR65Z0t0`^XnJ&YbaxDNbZYi! z#w4h!ZlED-e30t`$(HV{H{q$}u~b6UjRR$mAHUOM2&0qO_|#7CG4)yOhSZaSsFnp? z=nZr$XU97W@#Duna9owfmK=~CbI__E!Jc7YRR*35hX}Xd5ZIx+xIh$;5O-j8H9)H% zq8JcmJx@q@>Ld`O9N7>tjA@`O!O;?R559dvup~dsAgs*-7hvBg8p(^!AqCjgGd!h>rHE{`U5EcMJSS4eX<~ z-VVzR3=rd=>Ty$O_cOpg)z$+D>x+wnkOR-5K)^r|K|TP_K!Lv?0Y)I;f1QJX5CcC! zKt4qIgFpg55rKbF8DRgWf(2xJ`1c%C+=k7O6aTLqY{2vP%QX17 ze`T>X;lu}sOX2ccSnJ_3Qh%qW#pn8ji;K%)t*g%_Bk=Pdao{&jd_!AXOEwxB2L}gg z2L@^jYXcfOR#sLTT6!9KdMaQJDjO$rTP;T_a~p!c3;DMk0X-WXYa>frBMWof_j0wg zE$nPL@$uhp^ylaAdg?hE{dXsGn}3=GY#_~h3Jo1KEzSR%n5~ii{|B-6l)uIPYS-Vl z<9IKOP2NV&n%~0AOwZhw>%Y$B`0GypN8*3={991U$WhM}AYcTPv;nq>i|IS_KT`kq zl>b*rrT-)u7-;`j@_$nPNAkT1Y?9VSz(i{(fsQw za5%X>anSsk8eE_39X3!wKzKle1$gBgK~GYlQ$lBP`aJND7aa6Q0R3YCdi-Wm--5s_ z(wtYJte6NsX#qW|V7@s%-FUu%w)5qM`hl_MaU0t*hqD!S{YIO8>ha6QwL8vprS|<) zm-IV_I^D=2Xe{UmkpB)+F!)HY|Gol~(7=!b{K3NsdBFZ0HJDfExS)uCjs{83xb7!P z$DjXYahLSmnmES+1^bf;*Xs$K6z;rw1O30^?>Rw(`P)6fp*+Eb{yol+GFV#1y$$}{ zjd0@*DERXF1sCN1xaHOb7ff2@pSpXl0|l?21U*6i6eYC51seSS3l$TxdYfS$@1jtQ zizbnL)ZYbAAX@Teh$Ye{GPvYLH(wKj_Jflv6cHLoq}#hjw-Ih+gAc`1&@kwW-Ayj_ zmMe47z~RB>Q>`L%v;F!3rY@4mQB2A#rUC8J-q5Eb)7bB%Dr9*oL3ppjEj zay(*nC~b|866e3cY={Ku&BV}Rp!jc;E^L95=l`GW|N9+6#?k7zJERyRn=jehoZw)- z9!D%M+Q;5jOn?GQMa|zYX=7I<7XHs7bdBOk&Mz9w{k@M=YVL5p7TLo4pqloY9ZNAR zIi1(Uq2@q%aJhAH(#Sw>va8@wVJV88Mku!`qovRH&jtXE3!=s1GFu0p6Y*U-y<7#) zh~A#J>_ovHRDhWGzfGcz54cGvDo5S?w@HAaB3PQ#UsdY#>uz@WRavZay^yKYm0X{# zswnDg^#o1TnJpX252Y}weCKxl-;Ik94zykgPO(&($nJP9Ry>(WSV$<)_H0FIsm4gB z`0E$}0YSDK7}(Te*L~OEDSN%{_7eBb!R80+?MFt90>tA#+uH^|_=-NO>tSxZ?nDOf zREbixKqq9So~r+^L?WHMkv;k`!N59?a@9D>Me82cGavkw7I%Jsx9hMpC%e}!Cuk}K zzCH!;W7%9@#Lr=&WPkIPT@SiHRjX)hW2hBm$2gZjaFSW$LQIITnT z@J>N6k$D@^Xl2X&u73E_F>YqY1{l+&L95%-92KjpW;hSWw|e8ja(bg{%f{TZ&L%*G zkQJFsOAIoNthz|d&D!DbJa>+$zaI=ds!%*=%61(oWJr1^7hYGh@HTez_f(u;tmH0K zFrrmxO^U?sjRPOuTG3@`)7$pz=mIhfYthQoniaabx^kqmf70vp!8W?TIA?ir+0gS< z1ny11I|~gvZ&n?|qthR(Qqx}^XV!5f&y(l(gMHK;$??N8mn)KyWh!}DMIkPf&}^hc zQ^p_q@Hyn5rPgI9Ki6ZpSZzJFBjWk$Zb|-1D_laR*_2D^uKy~BoccG>-}AzYki>F- z%uEISB6;C)P*4x>rheu6V#!)#5nfZEBCUz%%Wt82{89eqVYXONx984=(2eJ#+iCb) zV$on^E1D&W(c`5mR+}qdTMv(n zI>fNItoRw{tjfBfPUrho#dP2-(1?(HY$<%Xm;m2{nM4wp%4xJn9gZO?KAPh+5}ihp zq^+ZGamIR~+wFE7lq(uuY{)N;k5qN?$2~2!}!PLo{acW7g=g;_$X4V|?Q@iD>*9&hSb)vG7aZ_(6qa zz45+_W%a9p7_fxt;Za)(L_mX&S$8=sD7;4 zTnRqnIsxI4P}D=PDM56ShnE)ls~H9w!(r`^eu_bpapoBhgcPdNzMonOf2$@zBDfBk z5j>E#ZWb9Ecde%;7K{E$YOniIDE_28t3O9;xD0dLQ5Bz=B;$>GyEJwN^HqqAA6m#R z3BLv?NC7JxMXlB}aNhn#&F*j(y`a0baKpzWAmiHCS!?rz2_JxcPzn`az|BR1gpz znqDbDUc9N;&3#7oIjKHMI{j9*HK^83KHd5D2l}-;^XxH8mtxUbl(S&zeI%84{H4b&^`dP7`CjSj>#<1g2y+(0;@bi?%y(1RNF$%QOZ@OQ8mYT2hs|Eih-zhTxba948~#L&v9RFU zH34WiNd$9_C&y&Xj~MH=g-h{9xBW0oc2_B9cuYKQH}0SAQtjt<-YT}v(%T&!TUdUX@wcoMd(>pi5g8iKT9i=whMH;7`Z!HD%|0$QUBfhK z(ml=NsKuSI=B!TF-hiwH%NySlfXyZT_3-Y1f$oqoY!~?K3|`v4e|wCbeRrZ*YbtKW zsP$FrG*wA)VQn3!j8lIkvso)$Vau}fq;P7#JyGmzY<>Ew8|P)gv z^Njrq0&M>%Jr|5%fGYptITmyI_OAh9Ruib;v?!OqE%Hsg61ZfE3uUUYJ00=7$mNUr zoTCPV0--h?8E8y(QBfb}PF<>5)lw*<2|y7q=z)r)`kdw&IW;&w3jFEQUNODd;KxpY z7~46%<^6%Q1pQ?hvFO#*6$IX3vL^&CXgKahy$&3wcUu4|ShP%qCzD@h05fuk;6K_2 zHQ~l(Tz3FrVod=S$~k#@yCv^bUhVROfVi39-5}PD@T-)ya3p>!_+i*_-;B>U4b)FIrcE%|z`wwZZ{oC4gr=K_GF58! z;{Lr(%Lp!hr?acYC-}r@da_w=c>1uv9`DYt4joYUise>LqYf4r!4Qfs*;;F^zXVK} z)^4|l_rE0N3!~x^d~9=A_xaC=;u7xIrV2RA3hQM!MV&gExe|5)oX!Hy3k`DIqYWTb z#vh7JrYn=zLeQV&luf*uX1zGmC zKBNve{Dk5S-r^0?s|7py-Fj-g%J57~iNnF)^+8yJ3`br5RCtB2XvT46)A)#vaQCJj z_gA{vGB2^~OQLHL)AxKEZcwQ1#p1d~Zk@e@8*oIMmbm$DEeN0im-Xf1*nHRU`w0hb zV8z>31vG=+kcCOEt)FG${j8MHy&)YsYeBf0;d;C2YgciVk{N4YDdgcF%g;3$SS8Bk zuhsUIG~3M8aYf@%q%iyzixofTUMe}_$eb)_e@&LC*L#g5(RaE~bVf!0Ye)77d2Z4E zNx+az} zB6t7QBT0Z3a;m3r0kiMcu*G%Eq^w6RbfMZZv*@&O?rtdRw_!;nLD8PdZs<|}OW}Vl zZJ~}_pqN@P!(LBt94SGyf_daeNp70t`SQ5<W_(jj{+(o zB)oOo0}k(cvii2fL4}r%b~}*mJZUx(cAJBx{0W5$@5$aA3Bdn9BN+g+Dg)|v#qX>HdQtRA;Drhu2`#}Z^Qb5^V8Kx9&6a;oKa zY;mcktf58g-MHTd63zHzd4CM14PM}IaTd@a${?S-sMbp}yzsIe_^;KRNe<>xy7qlO zy{l7|!mlzcq}5X^!Ot|Nhi)q0-nQO{j{1!{)=xsd{ae`pOw~< z4jiAmdaV)@Pc@YbIye(D^Ti4Ud(%Z+69wwIt%;SVg8gF|+?s?Ksje?5KLv*mzMD_> zf$vBDnM{Jf$@D11-D`TC3hUn_FM>Ah3R#iLpT5JF2eP7!l47e4P!q^VV@DWTU}tYtKpxY#zyVOM8G7ir(G1%(lKBz+;S~6Y@v{Djmy&=p};*bh7$Lse*s(e0V}&NnW8_R6m;B zx0{g3q?KvrRx6x}NqhUw$QEXaSUm8O-x@YQ=q>foMKO~Cs`iHnoMz}8G`R+m&`O^9 zavT$_;rf}D5I3Oz#aDQ;fbf=>ZBJWrr=N`hM0FNcqjx3|I8O2LYbS~Kc&3}n&i$kh zVkYG;!I?8^kCR2RdHJ_ggTN%S_KdyG7s0_>p6WPmkhtou>_6EorV-kAk*)G+* z`<1cNT$sJG1KZP^`&YuL%#C- z@63QG(LPEgLcJlmC^8d!;aK`y-H{aesT^^VLk9hrYjk?yr5eyEGEs$>D4$v$2Hg=c zlg3gAM<7D5K?TmCj;T=gT#&-{^(~c!TD^r-yeQ7r$p!fzEW)X2Eedx|8g?K8PrbJ` zo$-`3Qh&U?#A8D;m7}ZiG{uK{(EPWQnBAwMnM8Pzq0=maR-G%oB#6Ws?u%I;Z zK2oB5%%)J!nO=8@5bz;t>+Yr5p&&kJ^^s^xw}+0^Z7V^!`1h|ASsLQS>irnX zMH`_#%0)u{J<2HgTARqN+!~&4m>*ZjWY@#s#%Vu-SPf|CEAec8Qayo-vd%%bI`0On z0=4}|)tKTp^~jL@@j~k#18Z0+yM3d{)|V^jK)^N3U7=k4Me%H>xarZ8=XuI9K%-kUaRnbe3P@likJhLE^Mnc_O8cN05&tuaRpt z?zPWPdm`4FjPD)3K@uK*(-$y_8TM(Np9f_++J59iK zrG$6~HCW_IF%gFLY0D*^J2G~*BL2LhMjB8GAk8w#bTB1eD-HHrwvAO=Hg&;^@OGo^q^O$mzm1qQ&nE) z=778V-2l+WoLQL0C6g(sQRgeXjTXNKC}>pPcaa|$CiHp;U4w!luI?a&=i}ka*R#xF zzZ^bQ9zAeS!x#V9r~An)1>J{GJ0chHZCE8ks*?0LPI-Ep^?f-L`T~^`zJmuVRRA~8 zPTiFUA887!B>S$fmz=m*AM3~p1SbT2NZYb0gPmd-hu2P}SA*xv)F;>UM=*R1hvV6n zr<~72>#7XbOo^~<3UHod*i1bUBaEhUCim6WF-VDq)?Ah`8BTN>onD%#l7V{L(8%Dn z+5F8nZWE-`1A9SKXOUXiNGoB^wb7Hi&>wRS#fhg+_Ga;(*XS`26*Ws;l_hdz4-LqQ zDy-j%Q^@MMvGH(6gF(WM!p+TWD{_?o zF|fYm;6OVcKgG6IUszEx61OLu3E0#SdzBeZ zW0~DJ*m7z1fTPp$fvXiBJo@|09%Ma62KeX=u#{jx(`kZltqur1xQw3f)y)*l4-mt( zKh~71HrtDrsWurldnkt5vF8%&?kH?8iPaCseKyIFjK8g;C7a0!9x(>?2IvFK`|Lu+lEVx)d*DdS+BzOAbFX!ug0aHLm83N4n8u&#r0cn zh9`=e(J`V-HanabU3$5I&y9+EycE`HKC8$SpIyTmYa>u5R6ym4$4e;5lpj&ca8BQ~ zGg8l&DigPlH8Beq+rp*TUQ09sO6IYRn^ z%_;N8y0(6FmxO7mbEWwC&8`RP(mpDRlUwgu7tA7#%w_7!M7aSt?rPkZWjF0yp>w2= z4*SA!Xf)hzVq?hVMVjo6i9<1%?VA}Rw`kj_1{?`D{J<~x4cyVhFHg%ux!o^D!nvB} z>Va^JGW}i(ichnH>10-xZKKFk&8{d@aYhiv^cSNfTb)khuruEn72|W)8gIups|J3vT~?!O+9%w)<6wD{ zrJl^2-ENc)>nnk2l)84W!{|QfE_P>O-WYiQu({)gt>W3f7SKp2k_DC7 zVJWjw;ZI}4%y|m@;@+Q-<0J*&GNQd$-0tl5pB7b0UmYLNm^M;L7n6e=F9gcr(;!${ zq(&0ydDGb+%OAHxO*z52#WAUX&&s%F2XppMS=G@?Lzy#g$=J1P$9Nj50_=rcKzpg~VASE3o(#?CJzzBgk3UDL+NG6-)A>5$ za5|rY;!XU9)1_R|=ya)eD(eT#U?M}lm~?fT$xMM(Xj;MkAbzaq3?^cNV_QT7U{s-w zOLdNN^g|%?44Cl)I>6+jjzwDvXR*~RS~;3BxyE_-b1PN~udEF%u*P&oFAlMIJE5`_45 zxab7|zs*}YJUP$*2%|5o1L{KgRj$Agnvv_f*m=Hsi>=f0m9)F>*sgqUNSs}&!jsn5 zTl8<8=%}K_N}wHFR5oU{HQ$OUogvOhCk^I~zH61nDFFIYp^bd|5iR97ho$V@BH2pe9~OHU-wslGHhOspCq1f0K|>ka0&foR1p08Nv_`PS@T>9sw# zbErh1>H-86cHPRYEOs>**B8cM_m@h)pJX0H-R8(mh8S;tVgs!~nS3pVLuS*K(~y;7 zioq^H0uU2=O51zayPN&1(UM#?*jw54CihUytl8&8?xM@$FUF!Ht}-$&1*jBi5-RDm zd6CrS;3ZuMbfsM+c`*|bw>lE3e*NdFrrmra=)C)soZD|1N_WJVVSPs?H#YX>Gy^Gq zKJA?40a2XDvX;x7fEg`LH)$p2(yq;T9$wpPFfx2EZMi%H;Y?^3 z8FfZf&4@TtuJAa}7M~mk(_V~gc;0+M$1X3TZhKZJ&UJiMZ4;W1SLF!}=}x}({IY(u zSgkIPjxv`EeqtcGy{i@COxJVrTYea2sHdEJT5YC+TKgj#WymfCci)eFwWUvDBeCEW zaRmgSASOHdwYQ|0efRjVlr1kVwoy%zO!DIoM2aW`eKq*__$%f_wsN@Xhyl$G0vePU zp}$T`XgAmd&p5n0)8u&loYZu`zFie1sq*}Vr z?ApBDU>gIJABs6x@!wrVFeQ>{m+I_OygaSE?)mt)=iQFbmp^zup|Q_b%bFC<-=ixF z!&X3I(y9MWSZeJucJ|;wUX#!Ua~yk9y-hB;KSfn)*N;_CuL?%=#(~2b$2lW3$FLa3 zUXB0DxR*seNqxOrTGu=7v#UEZENKUjmt@7rcvqVC3bG1xc5F?Sf={q}t5ubZm4tZ` zrEg47_zSb4F|g_{@x4sxd_Jey=OThzf7_u5Al4BK4Wg;2S2x3pjn%7sf=p-m#K#Q@ z?*4G-(%VpH<^&1Jd$0=%uK6Xv&T@C_!5P%p_GMP?7{6_+5ce;P!sa8|MzR`RB`UOGdFNm3t@j%?8$t423%mxyUWr%Q_dSA~8u6&? zcCsO(xD+(Y=gNg`H*A%~@{ z9r&A@I<|LDU5I;P8Kv)ULcY@-M1{(#?~ad7=+`$nq$oBP#i+x_tKgby?;x645Rz-S zrv7(-<`M;lO*-hkxzc2r27oXp1=~qvkxbDFfwca$Gc#!1S~65dey%P6Vm0=;{<*AO zv{GOOXV@c~w*)=niUKvpv^;;;Q|lp9S+1Uy=<^`@sNrM0y8IrV!#7M z{n8;rKKC-B)!JDEbr9UQHGi--97m9^Zh)kNLPxld+d`KVrlT`>GDu?F-}DTzj89WMAQFO!asZ)An+0jImXa3oKtO=ep*aMLOrafJxx%Xq!3dOGgHq8+m7N>V{aUruszZ#P!l`mFCais%Jgpj}C+*BN82m^>2 zIrCExX`%0nB^Hpd{7SAQg%;i9R#Jh;bc?53i+C}rOL?nt-kSdo^i*3PY;;2EBPY<@ zdueeHdA^uqq16Ybx-pfjP{Hg!=Q!9xQ-$?54j<4E;NBq9lZWdJ)Qz( z63KQu8ezHzdulT_;g+nYQXks>JNk>spAeXmSYv_If!5*6s|;>etj4PMc6HSnA>+COZs926Z}Yx?}{kz?!1Iz`=9#C7Mi~ zy0_&5eE}Wjcu8(YQThU_ncm@;=qh7{@AE3x&4RvP!@NBEL%N$C&P0Gvuyz68Nd4JL z8x=ff496bNWkWKZXq;ouRTwFZzId#b@o@%xKxno2HYFMSWSAB!Cv7BlK`j zth1H%p1Q~%I0)uN9hs<$Nc$s)Df5?T-tUQ9spLX|Z$Ztb%UidMyOP z=~vF>uXHlLntd=S)q28V*j%{0oe-FLpvM$jH;;~vHBUmQuB}>T95G*uK{2^)dCXT@ z@b-a*-VJLpaU_typ_Z=0w@`|4H#VNam8)N}UN`%Y7i@Fb1tI-gETw6s(VD$=f1G*5 zIfP2%tb`6QucG?hAx_4x-%)CMtmvr(F};+s~s9VO=iA1L$`pxS3R%rqTUZ* zcjC9+gO)={x0|v!FifO#O3~z$wN=#lc>JWcL{@35-rGzRE>~DKg>`(ZRE&V0Kf6CO zxNUtlBg|^I)llcJ{blVPDFA|t;Nydh)docQh~`GqdGh_LN&I!Di{E`*jje^QY(XDe zjQyrWo{6Gk;PWFkHlF4Jk%yMPprIL7uyOkP`m(h4W86V+=ykid{wS$!(A(CiD>I}d z`C2qL>sJBmlJm*n$9jK10}ehAA>|K6UoYxtV)W>PzYChK zrZovR1yqY%Q4}ksk|YbgZEn9PAY`dH?PPFCSD=e-bvQH^?4Qbe^?p2Ywl^i3CJC($ z>UMLnFBCubh|Myl#3Me|{Gxjj1?q6L18sDF0yA}-tp$WZv8mw1i8rIqJ^IV@|foNF>^i~u>iCe(kdF-|z{NN=2({4kRCEo8T><;2c1PerTgEK zSMx=?N04~w*d+7WZ&LQ0g~W~Sr%82jtj1WBp}wV*biXz)URSLJ?V{J@SBLxLihO6}k z0hls;HyPvvJ!jCht#f30U&La`W5LOW)xDk}h&)~q3YTgkef?1;v%(eFLe~x67(@>e zg5({cz3i;|<33kuj>$b7O}8C6wPoK0!%-wcCvCR>%c28YYf_7y-sy7qAY0!s-^qr<)e zXSefunVB^o6^-xZnBS+baeP05g(|at>|VxeO!V{leG=yBXgyn7$Qh+=6Yq$$`!XzR zw%(u)(^t427R7sRKg<8|lRt{$>f~;$CS`Fyoeb^cZT048El^e2n+q{m)cS&+6=wyg zwhjW)EZ9{P)j6VnhZq8J=}eTl)I10+&nk6Jg?-_Nt!7q4nSr<46){5}{=m-|6DnL7 z-10>;CHc`9OXp^+=vE_f*PA|1ru!_c&i5N13*9LYNS*|xFA zU&=}`%P(YkB#rx2M8zf8bTsh$(RegDP8M+n zP44m}P&TWx^A#Fa|GF-E)R6tr8Iv!0Vu>V4Uw)9Egp;@rDgMy-?ZgENW}f5Bh(_nc0}>j$3+o|P9Yk?N6%A}fJ_F2J4y3??CWZfeEFOq z`fdC%R}_`dVa-)0Lr9tokdqKBD;G>Ld0_gpeHdm4T$l&=guSlr*X*^h$u$~|UkE+P zSuOQO-~}p-JPqlUNB}xV1G&}HF-+n_j$-0$Tk~0=DM;Oj6d-MB5$3cON-*Ubok673 zS#mIdlMPz`Zf_t}@qEgsOxM-Am|DMhI0oqIWR1@?Zp@Go5i0|5Na{+~58USSZqMSr}tqJvaNJ1!uDR+w9PM;gbHe1bdDg-{K>VTo~2VVLB{i*pM zco61{E^7Kqe+T-h8(s2an&COJqMOKyUFBkh zGBSlz1VF+M`YP242|!gr{G^n_rWYkXx8GM)uwx{m{qE$vky-Y2!=1G}&g>KH5Xr&k zuBEI87=Mw#Dz-hv)!DGVy{0mOwrIq8)?aIJHJCpQwq&^$lue&eee`B(AY8j^YD(;6 zfPlAxj9OfpIbMtNjfPX3dxP$1nymFMbks@X?*by(D-j7jrQzJBOxnG%U$xb8v&Ry2 z<`)%+u4oR9a&BS8m;MWMyAf4sD{XcvpO7gFdNo3av}Cf}Ox(=pK3+EsRdo;|h=L`w zcuN9u$)&YBrW_5|%~=b$Dp;tK*!jbXm2SR#=_e8#Q!NT%6S^vdR>yNT)e&awtE5$w z4djbJ4TExQ*{yKx72E-?g|H!PiuPn?^8)jn0G+Q))Lyj5rU&;%RzX{H6@V=L4fZ!# zpWknhTRBCc=|GfNaD(lqac=m(Xc|sn~I7nGDar?KB{Hw z+ZkxqV*42uhrJr?;uux4uR$-`vV@YLv!wQybZey`o^jZZxdyq!Q*OC;uW3V)K8$L~R*3#$mP#DUBADWT)ez_G% zXx@W(6(9FxOdzL-utAmKkiyMma$qY1<4gFB(ewBrZ;#Fnk&ez!@oru`bL(l*`)s9k zfIO<@8>acaLSyU0=LhhwK(I&U=!s6=d&&~|Ve-N zr&=^y`kV8v2OHzYBn)SiOCWuyl}Xe8`cR&2aVE)g=%^4SXe^mITpfp&g$014Q@tI) zD4EtA26R~;1Dvfz9KS4=`x|KDkI>bc5fEFXvbf86cwhMyqy^$a%V<|dAcj1^1dxA0 zKdOvI<-Dm!JADH>rm7@AdtCnPH`lW;WhBw-Q_=Qd4CDs-LFteBAGjzyQ{*&UbZ&8o zS1C1E6g@7ryuso`FXR0RCreYxq?~VHb7J<>n(JLf#ab-k!4T7zjlUQN3N>I1ZNW(& z4gdiqA(!sp_$HY}L*9yG!JmwU;A50fqpLhwT3L|Lgy^8tKh!EkLP_;JtvV^6-FGhB zU$kGQ?{s!+mR=_lYv~$zzO*y)2zc%5pdtrw&1AF}F|K}NGTBvRh`iLRyDaCv_KmXh zD4S6$2`^Y>OsY4hnLXkf1YN9jY%A^hyjuUx5PTH|=@zR<0^zzHTU%-X%GBmw)-17O z>R^0`?WMD5dqzZ);Zeb)>x&#dUy!{{GYRHDH=+n2>g_7hpOmY%DDC#-+JQdaz!60` z_|Z~tu1W$}%iPLl%yCKYocw6=Bvq^Q_1#r2L=Q%%ghA(Z;<==Mb&%;+WzetCd8UGf z>L%LSqC-}+V(gloD2$a;mjX+Me1sIeA5}uP5TH}`IeXRC;x2pc?;N|iIKt}$tU z&GAxDroB^)JOdCr9-DH91K3xxzP7iz%((9*!1@T&JqeyMd*wjKx7=4NS6uk)QCg!>H?irRJL^=J-69~!&FCYW==L$Nzh0+W;Qohr`Z zbahl2>NH~^z@_J8tBYKfr#IG)bJGZ z>)e#{-CmaDSBci{sw>?YFZ&F7@V-5~NVZv%C%9{T*GMy4{?ZAqt$R}8b@jXEoTzk3 zsS{(R>@+#oV*XYpM;E>P{AVA|YhCE^tU>A$zH#fBwkKK-hW;^}S}xzNG1sd2+2?VL z;lFst!aq8cm>J!h;fs1N!i&}jp&Cn*=L$<1%KMuVVrTUsKl~K4M8oEk?-R#n=BuFE z&(HRlPzQ)wEqbTtnrG`12ae;oiD0ssP?_q7QTeaAWS7FCGyDiE#6p!w501co@gIoy z;kHB>&3@h&PcNZNFBjcLG2ZL(hlY2h1OJhWbzhXbD_G-gzL{YH}w7Rf?fFog&YWw<$>N(CW^z>Uhls1qEHg5v)!xoL#4M z4uPY;4f5fbXKFCZr}o31S#CUhIdW(E*@3R5o}o#Y_8#M5X**%N8o5=icsoh8?8Qqu?qMzfi3XhDZ8~pW~U2)S(<=r3An5%dI8bK0GO> z5AU&+Mc@MlOop2dDwXog8pYWBLL5xiB-DV%+lyj|Xtf(^*l;|}{NY-Y6UEr!lPoyk zS|Xh-v)GmQG~P)b`i9-%`~X5|6W!{%q7AYVGSlt$TA0!E&VQ}o`4^&lLSMKr>!Qu> zoI)GWa3cMZAgK8=oJb-CWcK97ArJJIX^Ka_{ms0)XY=A(T@ux_1mz_BY}abJS)w}@ z#W*(mDtM~J?R2F}2B^s3VedrPqZ-THrLBa4579DJ13JhSKE|H^gh%?tUek0G&OF8N zOD12KAUohX{#I?R_LNlnGwS^BWe4+NpyT*wP@Fh)A<%0ico{O>n>d}nQzkOM*4YF( zX|S?O@iy@FW};IIi;Q8LP4pI%?JZz-ZAhs~KRO`B4uy}Vk&W*pS1u$>e>hEcst6A> zv3fURWe6+~LhN4_V8=Z4$wHO1q*(17gU()wGg)#>5vomNQY2YW=WafhnO;$68qBkVKO=J`SYT9J;Lmsmp{DE&AxJ)(28DC|*u3cwQZ@o%BF^ zfQFJax0XeMm{$fG@5(+)R5&b~=P}8_Q)^_%z68ek{TROfNL244)#D#VaD!KeT5XvH-<@Kc z`%97TKt$(m?r1Ve`S7%@qBk}T96SzJ3@!@Z!|&mm++@U*5CIab2D1+|js`Ln7;z#;9wChKLQkN~$XxcqpEOTS z@)?JUnhLv|&~6wtCfvD=y53^7%J0Vt@CmLVIj&G?h!~Tfuqr9^>%g`S9acn8EOM zS0fnzb75Bv<<;OHzpouMtCzm!Y**jx4qiw-I1720AS^%RyEVN>l9nIH z$+gmsJ}`L1RDGowF2QLJRd{q?WfuzV#~2#IBH0OFHP9WY(a<|R7u{)=PX7S|6CS7& zkZV)yA4Nh6h%;v;2N-Jt`Q^LUK{`{#vL;xN_?N&Ch+RL$<^c;2oEbq?mEP>`o~%CGy|zgD}6UkU=M73oX=PEg+XVpCKIkg`ooCP~Kh#ej!!Q@y`%&uDs+Zmscp&S4cCQ}^_=XDLYz_J`7c%vtw?tSd(^>~)cjD;{ zDMpcPpF4Q%0NM3jy&^-97WNUWvcyYG(rHH1ylkzz4JJCjn`UKRE)SHdShH2(A+$XB8VGR~Im=GQ*xUC(F`0lAP& z+-VU!+V6fE$}07!v(V%Dox5C^oK1$iCYN;Ard-MgyReS+hVg#MX)s{eAI_0g02)r< zCG8w7F=bePICk7MdRACrce#s<@r)VnbUTI#h03j}(@qF9!|qt>a#QjBvkpXT%yDc^OlE9}58d@L?U&WkT`8P4BJeFtRwey=Kx72x#+m>2 ze66gXF3BaKz*^2O+ZXxM>?*Zv8&+=rj^@;-zP*#Rwjb)!J{5l{HyzVZlutKnx{Y`` z`3uDRnZFQxqx-@l=3Q>1fni0nt0Rm@++X$7=7FIA$F=HMvxT=vp}Q`zKY>ijW#S7h z{i}t|53_eWsiaP`@jIpH@~J*Wn~Xm6R$7-9EaMwBRxU z^#T#d!6zCM$2{9{>I@9j)G!bSGNik0*Wi$HN&X_W;(4WeJgSpZh-SQe1VPK^)W6r{ zM9!CgV!n1RAD=HK+yzqFlxBCD%`oN%H$Ob>{I+nn20YHMofYfjSwtF`#K(WT`6fIh zS2!c1CAu5@?d*52gAHb1CJ8hllNazs6-4&L!inxeug~;mhj}@a-u-)_)Z4JeTY=eL zV-?tG!7OD4yP05sjBgr%1Lv{gxZJ|RHT%k(P%8$K@c4dXZalPf!xTmdFZ|va(^eGk z#qqw))BF7M{)1igU}PGi@s^7sGcrgd|Ni?ghU(S7zOi_ax<{f9TR;ONVB-=g>h8^6 zbc$Hi^M(r-2lP!d8BeAlY{0h*{;&3~`YY=0TN8sc3aE+z}n3g;8v4ndQ&iSsulah>Wi}n6){AzW*Xt)mBIJqV}Vei+_*?Gz0W3yRrXZ`e*zN{k4T{{!}8rkgd5ql#BuY zggmhEYu;2=>a?Z1=Kd`~KhuV2cJc#CtojpQnvyp@eq`UQMPB3nSvNjF7YjH7M23p} zwelqz_L!zscKum-{w26FUE#)m$6$a!r2>2dE~nB(*#GRN0AQ$$+y&ylE>;Fy?8W42 zvA~~2k!+ysNI~rQ?<7ooz$Ln9&?0IT^B4NM28we9)*ius#WxP@;^0iizdQ~uS->v*?FVT8MT+QPP=sj>?_K$`*AqC{ z#igC=MDm|)q{Fh2ZvP3{?;!nu4B7D%?b=|L=^hZmQB~rtq{ocj-voRIjU))@cm$<# z$xtnRO-79#0;w%5r0r8xap%rm(KI#IgjBp$8HU;u#rpT{B#Qiv2A+`~lp5BSwp-_^ z#7!KBk-|%NSsFb}=#`!FIC{)iD0C-^^h%yrS`BtlbgUdFa_hskgohK41_KUhj~5)~ znt29Bl+JG^TxVl{R4imA3+~n=sOrM=JIQz;Rk!9SgaOR;;Fn63dJ?aW3$TF6oo{x2 zqg#+jD`JZm>*maYQGWc9CGKM&HZH5~YYo!qXt-zD?s!oW1&@KS!3n5^TepBT-0QvD z#DPo77!7+k`dV|O%4QT&JD$LG&+W#oL||y4mLvZ(cUj10g!P8cNtxykKqqzE+S~I& z)ZP;EtNes7LrLUeM3t7kN7?hd^`wD%1zHRd(b3U&y(Zt8PHyu*_y`TNB*MoZ-XLO> zL{ajYa97(+#wPA?y=!EiuCXW2R{|h+-Vc1Rc7PcIxh#eZ>u7;CmA5lq3&GXvpSiD} zESw!pJLauGq9Bm_va$i&vtZMQai~a&<4gKI7JAUsK!}YAk;%7Sb4kgoY>0--%$n~8M0B$N9!`f>2*(tM?Ga#Y4l=oXt#jr#Rb-U%8q-^C@;+96 z3;(QYnGktr@#ZO4ZzM?=3;o%g*H($DrDwE8nyCHqK!(KOd9&~7=KU`fmMlp;hOgVJ zt%v1Ls~zV|x*7P*0%G(06y48a86=cv8{F0B()i8g^I_D#f?b%3n+%)0-193el$fOt z^d_o50&kWti(G_O4}fatp82ljqTxuT9!h!Wb+Qn~;=KJaMUaJYq+2Co`6D@p*5{zX z3<)LiLCNk+0v+gO!h=#B@`cO3M*Pa-Q?8*pt@YlGxVTlJ7z*7p&X1R4>nQ zWnR(@mPN!_ZM8imtT8*GYm<5jw-Irim4sOa3DNh>VN7)|hp~)^tvJ1GQNuPT8%V+C zuxntaU(9~Ax-0bBs;6TCOhL9DLXfMsd=-X0Ndtjg75Qr+PEeX~CSem--*Cm^Eq-wn z+U_Dn$ZCq)d41xxOqU=yI~cA`P@wGQakp5z-uRyFcMF15b!5sa?st>umzf3rSRZar zyxJA;0qd3D$L0F9%*vl>73HlFkPgHgUdd;M~Yr<~^b-@gsz1ZPK`z0j5( zUMe3*zroUSew5bQw)&`%v4@B&4(9U<+wN{}B9Hs8^;FPE|I7`@HW%l|rZ@aA8cXNM zwI$7a;T-EzN*OX|dkTiN?|$VZl1iV*hUv!ulN@(&2+@z1NU%-^?$GB_ksXTEk4VuF7(*duQ?hV(-86^@jZyg=P=sA$BKZ z&>NO8L9W18Sx=X`G0qJ{5j>GPc1i%^kss8_I~J&79{X{ROmDRv6~wmgZ|! ztOy0!b9}|N=h|`mz^Yfme6v!j39TmS~g>&=tlVj(GWVOh8@l zkp;MuBwj@s@|P0a&*i5iqkNH^p0{tBlan)?i|S4@H=^khG%$8@Dnplz4N>!1EGn_o zC5c)77xj&MZGWhpbe@-VfC54=XR{&ZD@wtaL5f zcRGlpOT~bJy}4E1&Paos%oJbhnoe#AerbtYXB1VfaUA`P+js6&QgG>M#Q7>3n{|qA zKO|gV&Uf1w5!ZC3qm*8bXxuNa9ZOxCv9%HLV~;tZ_XnRJ^sPq z%0{UL&j8L2LaGbiz6vo;36tl=3dngCkXz$V(86VwfU7D53mQVLB5K(~5eQY~#x)NU z2-OP{uq>nO>`D!Uo1D!rbd%~`LlL3RXM19SO0^Q3C3;VbN6F*nZ;}AqFlc(TcCuev ziYo>(HIH6=mRj$1WheN6xa(wM@=8bem8hi2^i-#Wn+(R4-z0ILtQQ=%H4Y^rmhP zYuvy6bnVMjP2IbOhCF>uO|MmJKJl0quGq5Zx1|T1@cgKjU2U23v8($0fKdb@p=Px% z^kxEuYtR5h2PfXXO;lq|{K4pt$N|t*O|vN9uU`OBOTY8nR_i3Kc+^pvtT> zKTd4cLm}$Hm<&7ahbXTcZ-V$4VJ#oh@`k? z(M!+Vg%=@gGqS#SGOS<}F1h8wzvcTv7GOkode;~MtM%t!o4jlLvSduhIxwXw>kX=H z*~0-=nNBR|A9V?WKVHUp7N%cS*Tu`Q+WlEt`k*7OG_^I?>N*uDAgo2eExp6wPw*;C z{eCJ%6@o+MV{0IGb)76KoG=_ighnpu^2f&LNP%`1r~GIp_G0S@B#P3*oQUq>cu-az z+-W&j_?C24%sK9giWl_Yn(phWm|#Vzl7UR@5NL-`$E!-8l6~h`;l1eAmEI)nkyq*k zR-b7D1J^L_?`e2VNN!Iv+MsrqXx#(LC^6+Cy6^No4{NA4?hLbxh-aDI#Jw;BxCz!vyfdi{qKi{fn{40h zEF7cz>3ns8*au-s6!y_AHbs>eO@`;5KurK2^5^Cx8jDu#^nj?tf-$)7l@a_}%o6Be zY8Wl=i@!U@aYe;cjhv6bcl=bk!TPtZg=hw#hQE248h8pKbnvqY>1U=WLL(wmBG8mq zRv45TlojMjVS9{N$(_;oI0KZ5N13T*`K5?}73-mD`cahZz2$acu%J&GAl{C(e0IMD z>wl=WGjprM3Ua~1><`P{vj;H0d8WDeFFVtY2%uS{I|=ar#i>OBr=D{#o%q+wBmlCg z2pnIu>Zt#s+5h*VKquiaBKk`JXx0SOzIHmP`oB$qfEp2q1ad+&Iq#Lfq=`9pK=-G& zd#C<&LW!462SP^|J{9&~3V{3Wn;9uHAEn1v;d|L7NkL&>D z02lAv*4g^!NAuI*mA>zN^(FqqD4aZy!kM5S{VYGV`Jo&IlC2D8B?5dRA~%o0QOUtf zDXv<_xkRISmrsXVlPjQ{3xmNHZ93ghnfJS6)j$s@D+1BF;Om9<)&s@8A(!==sTxrb zpO>;1XFofK*UGyYzo*h3uMaC2)I56A`~G1+tub7i=j`p>q6d5PA8Hl@zem#up{q9N zNOiEv_L^Iafa(CSTukIKWEZg=+l!!0;4##@1=n^5knMOZx7-N5TUiiiQreWeKm4%P zuV?}fM3s9G%XA^D*l$zS>XDQ@*8ym%N$>3(p9zcnt0azFupHx?8%Cju9BeP`Dk}54=g~fq8CGlrLX#eu&Ijp@!i&6cDs@AIpPkMz(Gd@xwsZ6YO`?NDTKc1>Q{Oz z;Qx%6PIPPy!1|G6DsB`sLK2{#2fcZ=<8PlSgc3zGKHfH99eBn_4W?lv1p&2xhJA%- z$XtiFLn)I}0q_fEKGWvncL{jo&Q6NWH?5T;f(c(T>~D>|$&(3>`yui8TVeZROYN)| zmmt9K&www&g!wJX4ml$glH*$s$H<+B{gAZBv);QKzgVQk@Q)J^H#4g!ISO9gvyvKS@Sd}$3@GGR$j7Zx$AA2U& zf!C36c)tSl27&m`3*=FBvF^~lpTn#6D5Ws z-nx1xa!6@adie*mAc*1&exS9arXt&TdjjLjh}MaFG_OP*pY*8q`9GX!5~%k=t;+ z8p&ze+(@avV=??%lTIg5EbNP%>Q!zOe~(nSI7^umcF@o!;xW5MG6HG<^~sQULOFhy zo7xVq9M2gyOdyc2>Bnlb#4*nqNBEa`nKthx)i;xBL9<9*%0GRVv)PkIANL)h7)cco#B8BVfvoJ= z4R=;R4)zMQ2Q3tb9mLu&LQX<%Gd)}`Qw__ISB_v>+zY0faQ*I&WrA>R)aIR}qRgSd zA7%N3kfls1Z$btQ z30IUP-Arv6C2e^@7kv*fe06`q&mH0f^(<@n&~_!P3ILJP!Hn}Tp8}1P-1qimS8Oy= zg+J-*4*CFk9R+s7d2+W#yRXYQe^GaG|!P$Jy z-l6T_cXwO_$X)#9_(Hz0J!~v6r!*E+3LwSp%7>aC$d=CL;V%>>1-fQZEKT@H$c(#Q ztJ63OyoTwu_aty_DHXT3wYg0mfQkDI0#0sq>>T~Kcdi*{0@STc$48@EUwmwK<7^=e zuBhqq>}$QN`mI~>_S|{ZT+_6;Yh}EtXJ9*)KY{XgD2PPqsc60)K9qanS#Hr;6@G`@ z&Fxrx(L#f%gNKr_;CEhqLarK0!@D!^@+Zij9#ohb$pL1b@i^pLd)sp@rg62ETJr(7 zg4Tt|uW4+TzhY>3Bp^M6$Fq#H57Bu@&+U&Qhngc0Op?P=gj>wT_9!9f&lG$?_F2$2 z=Ke%%u{A|-J{-B;75bFKMt^B~F02~rnQKQ)IS5WvYrBHVciPcpcPk6sJwj#=4oyh1{jEeEO8ac+sh@%eU;qjk6LBi)nUofs5Zb#ThguP~)PAZUO<)AA;yJ z^Sq-zwI0C&%_}>R<9y3zO^+q8IP?X=81`)acT)kQd_ZM_R6Y}m2&9EUR*WsyM3U>q z4OI|aTsu2PZXb|j4*IYv@1Qe$yexoE>`+7=01qd)n4>r(<|o;&tcGG@dCg8894--^ z1|UOwNC2qg=qQ(%^{0GbqzYCs#(Sam2Z;pKqG87o<%6GpVT`8$JpAWr|5rc)_+Uu4 z_4prKliHyq44Y+MXiR>*)|ZmE>o`@VW-3ASOcGvsf>(X>Xd4U!eACaE1$u!9 zIsJc^d;x&-@**tvVU6#AAV@$HK9^(DVdA&xT0R3Cu=V2qb6V31dL@;`- zo#nuL=nPiRoeE@iG zKu_7o@pG9RC`sh^2Gp@j?)tW5GSSt{09;VO z!L&3ZQ$~IMeV-DYYEgcKh?&yIeF#r)(14JY`}jY0~|9(lDgV`HvT% zA5qOzI~}P5@Asc9HGZLA`S8mQOO}Y;#QL;{1EcScVa|5ZO0SFk`By|ByJC3pgPvL8Tr4w*& z*J+bQv@u&jjAk1>?c4ZnwqEiE1qbJSRrwN)3`ro-yIE8Oe1@wWT(*46MB1PiApMVE zUDZ=XU2%pdYKCk0Ae4ti1#T2wEjRlRRMXH&5q69wX00=j&n4d+7W?8Li*obeo^GHc z5Bdi1XhYiQOdGT_RO!+8)&2%Mnk7;I=jn>3BtKT*s~7rGOP<>k4&?{^iemP zM^)L;e#uJk=MIztiu|2vOX*XN5TV~#>jVo7^!Vmn)yK`^fRd3dJm$(_DJ<(6FEutC zdL`ky+Q&KbHny#Mf|ODm^%L!HA5{SWzLe;YIQi6Lp;(Tq*YAYae2bxZx6Xad4d=dm z%qMs*5}Bcd4*3c|5v2aQ&t- zjfC5J_WRgjUm|}zRg|cJnsr~YfX?%P_)s&&q<^l@gB+cjY)&U9!Q#*5SrB$Zv3ShDd;$6XQ!a=ah}MkWga7lybpn=dcAEQ__rCOYj3gF6wnlr5K@ zq8J}I>$2#VSRN-Qe45^_32pRDANY1Zz4c@>b+o@y%;|`3;M&IMM%Bfquam{f4z<&k zm0!4BZBKoq-@Th@xw_&Z$ve5_PCU}$?%^(D!XX>bDglv}Zh7~l@!2r~d}oT^XVQ;v zef*-$75<$%MgLVg`B2vj6f^Qg}pC|PcZ`h+4%==rUxKTsn^zg#-u*%IU&zJZ}_lE#f>zwJk?vCD|EvND k3UH$S&!YeT!n=oaCoa@Dbc2Nj{m-e?l(ZB}<;{Zr3w<8mzW@LL literal 0 HcmV?d00001 From 23d4969e6368e62ed39df986572e4493777ccc2b Mon Sep 17 00:00:00 2001 From: Song Yuan Date: Wed, 12 May 2021 20:21:58 -0700 Subject: [PATCH 12/12] Describe the behavior wheh the mirror destination port is local port. --- doc/voq/everflow.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/doc/voq/everflow.md b/doc/voq/everflow.md index fbf2060b58..74efa148a8 100644 --- a/doc/voq/everflow.md +++ b/doc/voq/everflow.md @@ -71,6 +71,8 @@ If the neighbor information returned from neighorch turns out to be a remote nei SAI implementation will program the hardware to send the mirrored packets in this session out of the recycle port. When the packet re-enters the pipeline, the destination MAC of this packet will be the neighbor’s destination MAC address and it will be bridged in the ingress ASIC to the correct egress destination port. SAI will also be responsible for programming the FDB entry in the ingress ASIC appropriately. +We don't recommend to recycle mirrored packets when the mirror destination port is a local port because we gain nothing from bridging the mirrored package again that would be forwarded locally. + ## 2.3 Option 2 - Explicit Recycle This option can be implemented with minimal changes to SAI as long as a recycle port can be created. The recycle port needs to be enabled as an L3 port for this mechanism to work because it requires the packet to be routed after recycle. The following changes are proposed. @@ -79,6 +81,11 @@ If the neighbor information returned from neighorch turns out to be a remote nei SAI implementation is the same as a single-chip system. The mirrored packets are rewritten in the ingress ASIC and sent out of the recycle port. When the packet re-enters the pipeline, it gets routed because the destination MAC is the router MAC. The destination IP of this packet is the destination IP configured on the mirror session, so it will be routed by the ingress ASIC. If the destination IP is reachable via multiple nexthops, the ingress ASIC will loadbalance the traffic as needed. +When the mirror destination port is a local port, it's recommended to recycle mirror packets too for the following reasons: + +- The implementation of mirrororch is simplified because both remote and local mirror destination ports are handled in the same way. +- The mirrored packets can be load-balanced if the mirror destination IP is reachable via multiple nexthops. + # 3 Testing TBD