diff --git a/devops/tam/dm-tam-correlation.png b/devops/tam/dm-tam-correlation.png new file mode 100644 index 000000000000..03d11ef3667c Binary files /dev/null and b/devops/tam/dm-tam-correlation.png differ diff --git a/devops/tam/drop-monitor-arch.png b/devops/tam/drop-monitor-arch.png new file mode 100644 index 000000000000..38e35e798565 Binary files /dev/null and b/devops/tam/drop-monitor-arch.png differ diff --git a/devops/tam/ifa-arch.JPG b/devops/tam/ifa-arch.JPG new file mode 100644 index 000000000000..925f5a290bb7 Binary files /dev/null and b/devops/tam/ifa-arch.JPG differ diff --git a/devops/tam/ifa-tam-correlation.JPG b/devops/tam/ifa-tam-correlation.JPG new file mode 100644 index 000000000000..c82ef81052ea Binary files /dev/null and b/devops/tam/ifa-tam-correlation.JPG differ diff --git a/devops/tam/sample-ifa-topology.png b/devops/tam/sample-ifa-topology.png new file mode 100644 index 000000000000..b60225b5bba6 Binary files /dev/null and b/devops/tam/sample-ifa-topology.png differ diff --git a/devops/tam/sample-ts-topology.png b/devops/tam/sample-ts-topology.png new file mode 100644 index 000000000000..57d3a7ac4d39 Binary files /dev/null and b/devops/tam/sample-ts-topology.png differ diff --git a/devops/tam/tam-drop-monitor-hld.md b/devops/tam/tam-drop-monitor-hld.md new file mode 100644 index 000000000000..1e34e651c185 --- /dev/null +++ b/devops/tam/tam-drop-monitor-hld.md @@ -0,0 +1,937 @@ +# Drop Monitor + +## Highlevel Design Document + +### Rev 0.2 + +# Table of Contents + +- [Drop Monitor](#drop-monitor) + * [Highlevel Design Document](#highlevel-design-document) + + [Rev 0.1](#rev-01) +- [Table of Contents](#table-of-contents) + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definition-abbreviation) + + [Table 1: Abbreviations](#table-1--abbreviations) +- [1 Feature Overview](#1-feature-overview) + * [1.1 Requirements](#11-requirements) + + [1.1.1 Functional Requirements](#111-functional-requirements) + + [1.1.2 Configuration and Management Requirements](#112-configuration-and-management-requirements) + + [1.1.3 Scalability Requirements](#113-scalability-requirements) + * [1.2 Design Overview](#12-design-overview) + + [1.2.1 Basic Approach](#121-basic-approach) + + [1.2.2 Container](#122-container) + + [1.2.3 SAI Overview](#123-sai-overview) +- [2 Functionality](#2-functionality) + * [2.1 Target Deployment Use Cases](#21-target-deployment-use-cases) + * [2.2 Functional Description](#22-functional-description) +- [3 Design](#3-design) + * [3.1 Overview](#31-overview) + * [3.1.1 DropMonitorMgr](#311-dropmonitormgr) + * [3.2 DB Changes](#32-db-changes) + + [3.2.1 CONFIG DB](#321-config-db) + + [3.2.2 APP DB](#322-app-db) + + [3.2.3 STATE DB](#323-state-db) + + [3.2.4 ASIC DB](#324-asic-db) + + [3.2.5 COUNTER DB](#325-counter-db) + * [3.3 Switch State Service Design](#33-switch-state-service-design) + + [3.3.1 Orchestration Agent](#331-orchestration-agent) + + [3.3.2 Other Process](#332-other-process) + * [3.4 SyncD](#34-syncd) + * [3.5 SAI](#35-sai) + * [3.6 CLI](#36-cli) + + [3.6.1 Data Models](#361-data-models) + + [3.6.2 Configuration Commands](#362-configuration-commands) + - [3.6.2.1 Activating and Deactivating Drop Monitor](#3621-activating-and-deactivating-drop-monitor) + - [3.6.2.2 Setting up Aging interval](#3622-setting-up-aging-interval) + - [3.6.2.3 Setting up flows for drop-monitoring](#3623-setting-up-flows-for-drop-monitoring) + + [3.6.3 Show Commands](#363-show-commands) + - [3.6.3.1 Listing the Drop Monitor attributes](#3631-listing-the-drop-monitor-attributes) + - [3.6.3.1 Listing the Drop Monitor sessions](#3631-listing-the-drop-monitor-sessions) + + [3.6.4 Sample Workflow](#364-sample-workflow) + + [3.6.5 Debug Commands](#365-debug-commands) + + [3.6.6 REST API Support](#366-rest-api-support) +- [4 Flow Diagrams](#4-flow-diagrams) + * [4.1 Config call flow](#41-config-call-flow) +- [5 Error Handling](#5-error-handling) + * [DropMgr](#dropmgr) + * [DropMonitorOrch](#dropmonitororch) +- [6 Serviceability and Debug](#6-serviceability-and-debug) +- [7 Warm Boot Support](#7-warm-boot-support) +- [8 Scalability](#8-scalability) +- [9 Unit Test](#9-unit-test) + * [CLI](#cli) +- [Broadcom Internal Information : To be removed before publishing externally.](#broadcom-internal-information---to-be-removed-before-publishing-externally) + * [Key notes](#key-notes) + * [Specific Limitations](#specific-limitations) + * [Supported Drop Reasons](#supported-drop-reasons) + * [Qualified Drop Reasons](#qualified-drop-reasons) + * [Drop Report format](#drop-report-format) + + +## List of Tables + +[Table 1: Abbreviations](#table-1-abbreviations) + +## Revision + +| Rev | Date | Author | Change Description | +|---|-----------|------------------|-----------------------------------| +| 0.1 | 10/15/2019 | Shirisha Dasari | Initial Version | +| 0.2 | 07/01/2020 | Bandaru Viswanath | Major update to accomodate enhancements to use new TAM infrastructure, DB schmas and UI | + +## About This Manual + +This document provides general information about the Drop Monitor feature implementation in SONiC. + +## Scope + +The term Inband Telemetry refers to the set of technologies of the Switch, that embed useful metadata as part of packets that are transiting the Switch. SAI community defined a Telemetry and Monitoring specification that defines a unified interface - [TAM](https://github.com/opencomputeproject/SAI/tree/master/doc/TAM) - for provisioning and monitoring these technologies. Inband Flow Analyzer feature is one such Inband Telemetry technologies. + +The design as specified in this document uses the common [SONiC TAM](tam.hld) infrastructure. + +This document describes the high level design of Drop Monitor feature in SONiC. + +## Definition/Abbreviation + +### Table 1: Abbreviations + +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| SAI | Switch Abstraction Interface | +| TAM | Telemetry and Monitoring | + +# 1 Feature Overview + +The Drop Monitor feature in SONiC allows the user to setup packet-drop monitoring sessions for specific flows. A Collector, identified by an IP address and associated transport parameters, can be configured on the switch to send packet drop-reports. + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +1.0 Drop Monitor feature allows user to configure a Drop Monitor session on a given switch and send the drop-reports to a specified collector. Drop Monitor session is defined by flow classifiers that are used to identify a flow that needs to be monitored for packet drops. + +2.0 Drop Monitor provisioning as listed below. + +2.1 TAM switch identifier to uniquely identify a switch in network and drop-reports carry this information. + +2.2 ACL configuration to identify a flow-group to look for packet drops. + +2.3 TAM sampler configuration to indicate the sampling rate for the flow. + +2.4 TAM collector configuration that can be attached to Drop Monitor session to send drop reports. + +2.5 An aging-interval configuration. If the Drop Monitor feature doesn't notice packet drops for this duration, it considers packet drops to have stopped. + +3.0 When the first packet of the flow is dropped by the switch, a "Drop-start" report is sent to the collector. This report contains the event type (Drop-start), first 128 bytes of the packet dropped, flow details and the drop reasons for the packet drop. + +3.1 "Drop-active" report is sent to the collector when the drop reason for a flow changes, while the aging-interval has not expired. This report contains the event type (Drop-active), flow details and drop reasons for the packet drop. Note that this report is sent only when the reason for the packet drop for this flow changes. + +3.2 "Drop-stop" report is sent to the collector when the drops are not seen on the flow for a duration of aging interval. This report contains the event type (Drop-stop), flow details and the last noted drop reason for packet drops for this flow. + +4.0 Drop Monitor feature is by default de-activated on the switch, and requires an explicit user intervention to activate it in the configuration. + +4.1 If the Drop Monitor feature can't be supported/activated on the switch, any attempt to activate the feature returns appropriate failure. + +5.0 The Drop Monitor feature is supported on physical and LAG ports. + +5.1 The Drop Monitor feature supports dynamic-port-breakout and aliasing features. + +6.0 The Drop Monitor feature will be a part of the TAM container along with other TAM features. + +### 1.1.2 Configuration and Management Requirements + +The TAM Drop Monitor feature supports the new management framework and KLISH CLI for configuration. The following configuration and status information is available. + +- To activate / de-activate the feature +- To create/clear appropriate Drop Monitor configuration on a per-flow-group basis and switch-wide. +- To display current status and statistics for the Drop Monitor on a per flow-group basis. + +### 1.1.3 Scalability Requirements + +- Number of Drop Monitor sessions that can be supported is proportional to the availability of resources in hardware such as ACLs. No specific constraints are imposed. +- Only a single collector is supported. + +## 1.2 Design Overview + +### 1.2.1 Basic Approach + +The Drop Monitor feature is newly developed. + +### 1.2.2 Container + +A container called 'tam' exists as a container for all TAM applications, including the this application. + +### 1.2.3 SAI Overview + +The SAI TAM spec specifies the TAM APIs to be used to configure the TAM functionality. Please refer to SAI-Proposal-TAM2.0-v2.0.docx in [https://github.com/opencomputeproject/SAI/tree/master/doc/TAM](https://github.com/opencomputeproject/SAI/tree/master/doc/TAM) for more details. + +# 2 Functionality + +## 2.1 Target Deployment Use Cases + +Drop Monitor is used to monitor the network for faults and bottlenecks, it aids in detecting data plane faults and isolating them to a given device or location and assists in network planning. Some of the key usecases are - + +- Idenitfying switches and flows that are undergoing packet-drops +- Identifying packet-drop patterns + +## 2.2 Functional Description + +The drop monitor feature detects and reports drops on a per-flow basis to a configured external collector. Dropped packets of the flow are sampled (based upon the sampling rate configuration) to the drop-monitor application. + +Once a flowgroup (matching criteria) is configured for monitoring drops, the drop monitor application reports the following drop events on the flow to the collector in the protobuf format: + + - Drop start: Sent when drops are observed on the flow for the first time. The report contains the event, flow keys, first 128 bytes of the packet and the last observed drop reasons. + + - Drop active: Sent whenever the drop reason on the flow changes. The report contains the event, flow keys, and the drop reasons. + + - Drop stop: Sent when no sampled dropped packets are observed for the aging interval. The report contains event, flow keys and the last observed drop reasons. + + +# 3 Design + +## 3.1 Overview + +***Drop Monitor Architecture*** + +![architecture](drop-monitor-arch.png) + +The architecture of the drop monitor feature in SONiC is illustrated in the diagram above. + +1. CLI/gNMI/REST interface is used to configure the drop monitor feature functionality. Configuration is saved to CONFIG_DB. + +2. DropMonitorMgr picks the configuration from CONFIG_DB. + +3. DropMonitorMgr populates the APPL_DB drop monitor tables appropriately. + +4. DropMonitorOrch picks the configuration from APPL_DB, interacts with aclOrch for ACL related information. + +5. DropMonitorOrch programs the ASIC_DB via SAIREDIS calls. + +6. Syncd picks the configuration from ASIC_DB and configures SAI. + +7. When a drop is detected on the flow configured in step 1, a "drop-start" event is sent to the configured collector. + +8. When the drop reason on the flow changes, a "drop-active" event is sent to the configured collector. + +9. Once drops on the flow have not been seen for configured "aging-interval", a "drop-stop" event is sent to the collector. + +## 3.1.1 DropMonitorMgr + +The DropMonitorMgr runs in the TAM docker and is used to pass drop monitor configuration arriving from UI to dropMonitorOrch. DropMonitorMgr validates and collates information from CONFIG_DB tables TAM_COLLECTOR_TABLE, TAM_DROP_MONITOR_SESSIONS_TABLE, and TAM_SAMPLERATE_TABLE. APPL_DB TAM_DROP_MONITOR_FOW_TABLE is populated with a valid flow configuration once all the required data is available. + +The DropMonitorMgr configures the source IP address to be used in drop reports to the system IP address. 9073 is configured as the source port number to be used in drop reports. + +## 3.2 DB Changes + +### 3.2.1 CONFIG DB + +TAM\_DROPMONITOR\_TABLE + + ;Defines TAM Drop Monitor switch-wide configuration in CONFIG_DB + + key = global ; Only one instance and + ; has a fixed key ”global". + aging-interval = 1 * 5DIGIT ; Aging interval in seconds + + Example: + > keys *TAM_DROPMONITOR_TABLE* + 1) "TAM_DROPMONITOR_TABLE|global" + + > hgetall "TAM_DROPMONITOR_TABLE|global" + + 1) "aging-interval" + 2) 3600 + +TAM\_DROPMONITOR\_SESSIONS\_TABLE + + ;Defines TAM Drop Monitor session configuration in CONFIG_DB + + key = name ; name is collector name and should be unique. + flowgroup = 1*255VCHAR ; Flow group reference + collector = 1*255VCHAR ; Collector Reference + sample-rate = 1*255VCHAR ; Sampler reference + + Example: + > keys *TAM_DROPMONITOR_SESSIONS* + + 1) "TAM_DROPMONITOR_SESSIONS_TABLE|dm1" + + > hgetall "TAM_DROPMONITOR_SESSIONS_TABLE|dm1" + + 1) "flowgroup" + 2) "websrvrflows" + 3) "collector" + 4) "dmcol1" + 5) "sample-rate" + 6) "aggressive" + + +### 3.2.2 APPL DB + +TAM\_DROPMONITOR\_TABLE + + ;Contains DROPMONITOR feature status + + key = global ; Only one instance and + ; has a fixed key ”global" + op-switch-id = 1 * 5DIGIT ; Currently used switch-id + + Example: + > keys *TAM_DROPMONITOR* + 1) "TAM_DROPMONITOR_TABLE:global" + + > HGETALL "TAM_DROPMONITOR_TABLE:global" + 1) "op-switch-id" + 2) 54325 + +TAM\_DROP\_MONITOR\_FLOW\_TABLE + + ;Defines a TAM drop monitor flow configuration. + + key = name ; Flow name. Should be + unique + acl-table-name = table-name ; Parameter to map acl table to + the flow. + acl-rule-name = rule-name ; Parameter to map acl rule to + the flow. + dst-ipaddress-type = "ipv4" / "ipv6" ; Collector IP address type. + dst-ipaddress = ipv4_address / ipv6_address ; Collector IP address. + dst-port = 1 * 4DIGIT ; Collector UDP port number + src-ipaddress = ipv4_address / ipv6_address ; Source IP address to be used in the drop + event reports sent to collector. + src-port = 1 * 4DIGIT ; Source port number to be used in the + drop event reports sent to the + collector. + sampling-rate = 1 * 5DIGIT ; Sampling rate. 1 out of “rate” + dropped packets of flow are + sampled for processing. + aging-interval = 1 * 5DIGIT ; Optional. Aging interval for + drop- + monitoring in seconds. Determines + the interval for which the system + waits to declare that a flow is + no longer being dropped. + flowgroup-id = flowgroup-id ; Flow group ID. Must be unique per flow. + + Example: + > keys *TAM_DROP_MONITOR* + 1) "TAM_DROP_MONITOR_FLOW_TABLE:flow1" + + > HGETALL "TAM_DROP_MONITOR_FLOW_TABLE:flow1" + 1) "acl-table-name" + 2) "table1" + 3) "acl-rule-name" + 4) "rule1" + 5) "ipaddress-type" + 6) "ipv4" + 7) "dst-ipaddress" + 8) "10.10.10.40" + 9) "dst-port" + 10) "9070" + 11) "src-ipaddress" + 12) "10.10.10.20" + 13) "src-port" + 14) "9070" + 15) "sampling-rate" + 16) "1000" + 17) "aging-interval" + 18) "10" + 17) "flowgroup-id" + 18) "1" + + +### 3.2.3 STATE DB + +N/A + +### 3.2.4 ASIC DB + +N/A + +### 3.2.5 COUNTER DB + +N/A + + +## 3.3 Switch State Service Design + +### 3.3.1 Orchestration Agent + +A new orchestration agent class, DropMonitorOrch is added to convert the incoming drop monitor configuration from APPL_DB to ASIC_DB configuration. DropMonitorOrch subscribes to the TAM_DROPMONITOR_FLOW_TABLE and TAM_DROPMONITOR_TABLE and converts the incoming flow configuration into drop-monitor SAI configuration. + +DropMonitorOrch maintains data pertaining to all the currently configured flows and the associated TAM object bindings. TAM object bindings are re-used wherever possible. + +Interaction with aclOrch is required to retrieve the SAI object ID associated with the drop-monitor flow ACL rule. + +Drop monitor uses sampling rate configuration for sampling the dropped packets of the flow. The sampling configuration in terms of rate is converted into a SAI sample packet object and provided to the TAM drop-monitor object for configuration. + +DropMonitorOrch checks for support for the Drop Monitor feature in the silicon using SAI capability API and sets the field `drop_monitor_supported` to `True` in the `SWITCH_TABLE` of APPL_DB under the key `switch`. + +### 3.3.2 Other Process + +N/A + +## 3.4 SyncD + +N/A + +## 3.5 SAI + +The SAI TAM API spec defines all TAM APIs supported in SAI. Please refer to [SAI-Proposal-TAM2.0-v2.0.doc](https://github.com/opencomputeproject/SAI/tree/master/doc/TAM) for more details. + +***Below diagram provides details about various TAM objects needed to support Drop Monitor, and their correlation*** + +![TAM objects correlation](dm-tam-correlation.png) + +## 3.6 CLI + +### 3.6.1 Data Models + +The user facing data model is based on OpenConfig compatible TAM yang model. The backend data model (SONiC YANG) will use the formats in CONFIG_DB & APPL_DB. See above sections. + +### 3.6.2 Configuration Commands + +#### 3.6.2.1 Activating and Deactivating Drop Monitor + +The command syntax for activating/de-activating the feature on the switch is as follows: + +``` +sonic (config-tam-dm)# [no] enable +``` + +Deactivating Drop Monitor will purge all Drop Monitor configuration from the switch. + +#### 3.6.2.2 Setting up Aging interval + +The command syntax for setting up the aging interval for Drop Monitoring is as follows: + +``` +sonic (config-tam-dm)# [no] aging-interval +``` +| **Attribute** | **Description** | +|--------------------------|-------------------------------------| +| `aging-interval` | An integer, representing duration in seconds. Range is 1-3600, Default value is 5seconds | + +The default value for the aging interval is 5 seconds. + +#### 3.6.2.3 Setting up flows for drop-monitoring + +A Drop Monitoring session associated a previously defined flow-group as described below. + +- The Drop Monitor session must have a unique name for referencing. +- The flow-group must be previously created with the `flow-group` command (under `config-tam` hierarchy). For drop-monitoring, the flow-group must be associated with an interface. +- The sampling-rate can be set, by referencing a previously created sampler, created with the `sampler` command (under `config-tam` hierarchy). +- A collector must be associated with the session, where the drop-reports will be sent. The collector must be previously created with the `collector` command (under `config-tam` hierarchy).. + +When a sesssion that is previously created is removed (with the `no` command), the associated flows are no longer monitored for drops by the switch. + +The following attribtes are supported for drop-monitor sessions. + +| **Attribute** | **Description** | +|--------------------------|-------------------------------------| +| `name` | A string that uniquely identifies the Drop Monitor session | +| `flowgroup` | Specifies the name of *flow-group* | +| `collector` | Specifies the name of the *collector* | +| `sample-rate` | Specifies the name of the *sampler* | + + +The command syntax for creating /removing the sessions are as follows: + +``` +sonic(config-tam-dm)# session flowgroup collector [sample-rate ] + +sonic (config-tam-dm)# no session +``` + +### 3.6.3 Show Commands + +#### 3.6.3.1 Listing the Drop Monitor attributes + +The following command lists the switch-wide attributes that are in use. + +``` +sonic # show tam drop-monitor +``` +Sample usage shown below. + +``` +sonic # show tam drop-monitor + +Status : Active +Switch ID : 2020 +Aging Interval : 60 + +``` + +#### 3.6.3.1 Listing the Drop Monitor sessions + +The following command lists the details for all drop-monitor sessions or for a specific session. Note that only explicitly configured tuples in the associated flow-group are displayed. + +``` +sonic # show tam drop-monitor sessions [] +``` + +Sample usage shown below. + +``` +sonic # show tam drop-monitor sessions + +Name Flow Group Collector Sampler +----------- ---------------- -------------- ------------- +http_236 tcp_port_236 Col_i19 aggresive +http_239 tcp_port_239 Col_i19 aggresive +http_241 tcp_port_241 Col_i19 aggresive + +sonic # show tam drop-monitor sessions http_236 + +Session : http_236 +Flow Group Name : tcp_port_236 + Id : 4025 + Priority : 100 + SRC IP : 13.92.96.32 + DST IP : 7.72.235.82 + DST L4 Port : 236 + Ingress Intf : Ethernet20 +Collector : Col_i19 +Sampler : aggresive +Packet Count : 7656 + +``` + +### 3.6.4 Sample Workflow + +This section provides a sample Drop Monitor workflow using CLI, for monitoring the packet drops as described below. + +> Need to monitor all packet drops on this switch, for all the flows destined for the webserver running at 20.20.1.1. A Drop Monitor collector for analysing the metadata is running at 20.20.20.4:9091 (UDP). Not every packet drop needs monitored, but one in 100 is acceptable. If a flow didn't see any drop for 5sec, it can be assumed that the active drop window may have ended. The flows are ingressing onto switch1 on port 'Ethernet44' + +Note that there would be other system wide configuration that is not covered in the sample workflow below, but would be required for the packet forwarding/routing. + +``` + +; setup switch-wide configuration + +sonic (config-tam)# switch-id 1234 + +; setup the sample-rate + +sonic (config-tam)# sampler websamp rate 100 + +; create the flowgroup + +sonic (config-tam)# flow-group websrvflows dst-ip 20.20.1.1 dst-l4-port 80 protocol 6 + +# associate the ingress interface to the flowgroup + +sonic (config) # interface Ethernet 44 +sonic (config-if-Ethernet44)# flow-group websrvflows + +; setup the collector + +collector dmcol1 type ipv4 ip 20.20.20.4 port 9091 protocol UDP + +; Enable Drop Monitor on the switch + +sonic (config-tam-dm)# enable + +; Setup aging interval + +sonic (config-tam-dm)# aging-interval 5 + +; Create the drop-monitoring session + +sonic(config-tam-dm)# session webflowmonitor flowgroup websrvflows sample-rate websamp collector dmcol1 + +``` + +### 3.6.5 Debug Commands +N/A + +### 3.6.6 REST API Support + +#### Yang-based REST API based on the openconfig-tam module defintions + +##### Setting up the aging-interval + +* Method : PATCH +* URI : /restconf/data/openconfig-tam:tam/dropmonitor/global +* Data format +```json +{ + "openconfig-tam:global": { + "config": { + "aging-interval": 0 + } + } +} + +``` + +##### Obtaining the current aging-interval + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/dropmonitor/global +* Response format +```json +{ + "openconfig-tam:global": { + "config": { + "aging-interval": 0 + }, + "state": { + "aging-interval": 0 + } + } +} +``` + +##### Resetting the aging-interval to defaults + +* Method : DELETE +* URI : /restconf/data/openconfig-tam:tam/dropmonitor/global/config/aging-interval + +##### Obtaining the all of the sessions + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/dropmonitor/dropmonitor-sessions +* Response format +```json +{ + "openconfig-tam:dropmonitor-sessions": { + "dropmonitor-session": [ + { + "name": "string", + "config": { + "name": "string", + "flowgroup": "string", + "collector": "string", + "sample-rate": "string" + }, + "state": { + "name": "string", + "flowgroup": "string", + "collector": "string", + "sample-rate": "string" + } + } + ] + } +} + +``` + +##### Obtaining a specific session + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/dropmonitor/dropmonitor-sessions/dropmonitor-session={name} +* Response format +```json +{ + "openconfig-tam:dropmonitor-session": [ + { + "name": "string", + "config": { + "name": "string", + "flowgroup": "string", + "collector": "string", + "sample-rate": "string" + }, + "state": { + "name": "string", + "flowgroup": "string", + "collector": "string", + "sample-rate": "string" + } + } + ] +} +``` + +##### Creating a session + +* Method : PATCH +* URI : /restconf/data/openconfig-tam:tam/dropmonitor/dropmonitor-sessions +* Data format +```json +{ + "openconfig-tam:dropmonitor-sessions": { + "dropmonitor-session": [ + { + "name": "string", + "config": { + "name": "string", + "flowgroup": "string", + "collector": "string", + "sample-rate": "string" + } + } + ] + } +} +``` + +##### Deleting a session + +* Method : DELETE +* URI : /restconf/data/openconfig-tam:tam/dropmonitor/dropmonitor-sessions/dropmonitor-session={name} + + + # 4 Flow Diagrams + +## 4.1 Config call flow + +All the configuration is stored in the CONFIG_DB via the management framework. + +# 5 Error Handling + +## DropMgr + +Any mismatch in the CONFIG_DB and APPL_DB schema for drop-monitor flows will be flagged/logged appropriately. + +## DropMonitorOrch + +SAI errors will be captured via the error handling framework and handled appropriately. All errors arising out of processing of incoming drop-monitor flow configuration will be captured via syslogs. + +# 6 Serviceability and Debug + +Logging of errors and invalid parameters will be done at each level allowing a hierarchical view of failures arising out of mis-configuration or system errors. + +# 7 Warm Boot Support + +No special handling is done for the warm boot case. The TAM configuration is restored from the Config DB and TAM functionality will continue to work as it is through a warm boot. + +# 8 Scalability + +N/A + +# 9 Unit Test + +## CLI + +TBD + +# Broadcom Internal Information : To be removed before publishing externally. + +## Key notes + +* Drop Monitor feature is supported via the MOD firmware running on embedded R5 cores on TD3-X7, TH2 and TH3 Broadcom silicon. Additional details on the MOD firmware, including specific list of drop-reasons, can be had by requesting (via docSafe) for the document number StrataXGS-MOD-AN100. + +* Drop Monitor feature is an *advanced* feature that is not available in all the Broadcom SONiC packages. + +## Specific Limitations + +Drop Monitor feature in SONiC inherits the limitations of the underlying firmware and the hardware. These are listed below. + +1. Only a single collector is supported +2. Drop Monitor flows must be IPv4 flows +3. Drop Monitor is supported on TD3-X7, TH2 and TH3 platforms only. + +## Supported Drop Reasons + +The drop reasons supported are as below:- + + +| Drop reasons | Drop reasons | Drop reasons | Drop reasons | Drop reasons | Drop reasons | +|-------------------------------|-----------------------------|----------------------------|------------------|---------------------|-------------------------| +| L3_SOURCE_MISS | L3_DEST_MISS | MCAST_MISS | IP_MCAST_MISS | UNKNOWN_VLAN | L3_HEADER_MISMATCH | +| DOS_ATTACK | MARTIAN_ADDR | TUNNEL_ERROR | PARITY_ERROR | L3_MTU_FAIL | HIGIG_HDR_ERROR | +| MCAST_IDX_ERROR | CLASS_BASED_MOVE | L3_ADDR_BIND_FAIL | MPLS_LABEL_MISS | MPLS_INVALID_ACTION | MPLS_INVALID_PAYLOAD | +| TUNNEL_OBJECT_VALIDATION_FAIL | MPLS_SEQUENCE_NUMBER | L2_NON_UNICAST_MISS | NIV_PRIO_DROP | NIV_RPF_FAIL | UNKNOWN_SUBTENDING_PORT | +| TUNNEL_ADAPT_LOOKUP_MISS | PACKET_FLOW_SELECT_MISS | TUNNEL_DECAP_ECN_ERROR | FAILOVER_DROP | OTHER_LOOKUP_MISS | INVALID_TPID | +| TUNNEL_TTL_ERROR | MPLS_ILLEGAL_RESERVED_LABEL | L3_HEADER_ERROR | L2_HEADER_ERROR | TTL1 | TTL | +| FCOE_ZONE_CHECK_FAIL | IPMC_INTERFACE_MISMATCH | MPLS_TTL | MPLS_UNKNOWN_ACH | OAM_ERROR | L2_GRE_SIP_MISS | +| L2_GRE_VPNID_MISS | BFD_ERROR | CONGESTION_CNM_PROXY_ERROR | VXLAN_SIP_MISS | VXLAN_VPNID_MISS | NIV_INTERFACE_MISS | +| NIV_TAG_INVALID | NIV_TAG_DROP | NIV_UNTAG_DROP | TRILL_INVALID | TRILL_MISS | TRILL_RPF_FAIL | +| TRILL_TTL | NAT_ERROR | TCP_UDP_NAT_MISS | ICMP_NAT_MISS | NAT_FRAGMENT | | + +## Qualified Drop Reasons + +The following drop-reaons are explicitly qualified on SONiC. + +| **Drop Reasons** | **Description** | +|------------------|----------------------------------| +| L3_DEST_MISS | Missing L3 route/host entry | +| UNKNOWN_VLAN | Vlan of packet unknown | +| MARTIAN_ADDR | Packet with source IP=0.0.0.0 | +| DOS_ATTACK | Packet with SIP=DIP | +| L3_MTU_FAIL | MTU > configured MTU | +| L3_ADDR_BIND_FAIL| L3 Packet with incorrect SMAC | +| INVALID_TPID | Packet with an invalid TPID | +| L3_HEADER_ERROR | Error in L3 header | +| TTL1 | Packet with TTL=0 | +| TTL | Packet with TTL=1 | +| PARITY_ERROR | Parity error | + +## Drop Report format + +The drop-report format is silicon specific. When used with the MOF firmware, the drop reports are encoded in protobuf format provided below. + +```protobuf + +/* + * Copyright (c) 2017 Broadcom. The term "Broadcom" refers + * to Broadcom Limited and/or its subsidiaries. + + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + syntax = "proto2"; + + message Event { + required uint64 timestamp = 1; + + oneof EventType { + MirrorOnDrop mod_event = 2; + FlowLearning flow_event = 3; + RouteChangeDetection rcd_event = 4; + NetworkLatencyMonitoring nlm_event = 5; + } + } + + message MirrorOnDrop { + enum MOD_STATE { + DROP_INVALID = 0; + DROP_START = 1; + DROP_ACTIVE = 2; + DROP_STOP = 3; + } + + optional MOD_STATE mod_state = 1; + } + + message FlowLearning { + enum FL_STATE { + FL_INVALID = 0; + FL_LEARN = 1; + FL_AGING = 2; + FL_EXPORT = 3; + FL_TABLE_FULL = 4; /* Atomic event, no other information need to be send */ + } + + optional FL_STATE fl_state = 1; + } + + message RouteChangeDetection { + optional uint32 dummy = 1; + } + + message NetworkLatencyMonitoring { + optional uint32 dummy = 1; + } + + message Flow { + enum DROP_REASON { + DROP_INVALID = 0; + L3_SOURCE_MISS = 1; + L3_DEST_MISS = 2; + MCAST_MISS = 3; + IP_MCAST_MISS = 4; + UNKNOWN_VLAN = 5; + L3_HEADER_MISMATCH = 6; + DOS_ATTACK = 7; + MARTIAN_ADDR = 8; + TUNNEL_ERROR = 9; + PARITY_ERROR = 10; + L3_MTU_FAIL = 11; + HIGIG_HDR_ERROR = 12; + MCAST_IDX_ERROR = 13; + CLASS_BASED_MOVE = 14; + L3_ADDR_BIND_FAIL = 15; + MPLS_LABEL_MISS = 16; + MPLS_INVALID_ACTION = 17; + MPLS_INVALID_PAYLOAD = 18; + TUNNEL_OBJECT_VALIDATION_FAIL = 19; + MPLS_SEQUENCE_NUMBER = 20; + L2_NON_UNICAST_MISS = 21; + NIV_PRIO_DROP = 22; + NIV_RPF_FAIL = 23; + UNKNOWN_SUBTENDING_PORT = 24; + TUNNEL_ADAPT_LOOKUP_MISS = 25; + PACKET_FLOW_SELECT_MISS = 26; + TUNNEL_DECAP_ECN_ERROR = 27; + FAILOVER_DROP = 28; + OTHER_LOOKUP_MISS = 29; + INVALID_TPID = 30; + TUNNEL_TTL_ERROR = 31; + MPLS_ILLEGAL_RESERVED_LABEL = 32; + L3_HEADER_ERROR = 33; + L2_HEADER_ERROR = 34; + TTL1 = 35; + TTL = 36; + FCOE_ZONE_CHECK_FAIL = 37; + IPMC_INTERFACE_MISMATCH = 38; + MPLS_TTL = 39; + MPLS_UNKNOWN_ACH = 40; + OAM_ERROR = 41; + L2_GRE_SIP_MISS = 42; + L2_GRE_VPNID_MISS = 43; + BFD_ERROR = 44; + CONGESTION_CNM_PROXY_ERROR = 45; + VXLAN_SIP_MISS = 46; + VXLAN_VPNID_MISS = 47; + NIV_INTERFACE_MISS = 48; + NIV_TAG_INVALID = 49; + NIV_TAG_DROP = 50; + NIV_UNTAG_DROP = 51; + TRILL_INVALID = 52; + TRILL_MISS = 53; + TRILL_RPF_FAIL = 54; + TRILL_TTL = 55; + NAT_ERROR = 56; + TCP_UDP_NAT_MISS = 57; + ICMP_NAT_MISS = 58; + NAT_FRAGMENT = 59; + NAT_MISS = 60; + } + + optional uint32 proto = 1; + optional uint32 sip = 2; + optional uint32 dip = 3; + + optional uint32 l4_sport = 4; + optional uint32 l4_dport = 5; + optional uint32 vnid = 6; + optional uint32 inner_proto = 7; + optional uint32 inner_sip = 8; + optional uint32 inner_dip = 9; + + optional uint32 inner_l4_sport = 10; + optional uint32 inner_l4_dport = 11; + optional bytes custom_key = 12; + optional uint32 group_id = 13; /* Packet type is derived directly from group ID */ + optional bytes packet = 14; + + optional uint64 packet_count = 15; + optional uint64 byte_count = 16; + + optional DROP_REASON drop_reason_1 = 17; + optional DROP_REASON drop_reason_2 = 18; + + } + + message EventPair { + repeated Event event = 1; + repeated Flow flow = 2; + } + + message GenEvent { + required string system_id = 1; + optional uint32 component_id = 2; + optional uint32 sub_component_id = 3; + repeated EventPair eventpair = 4; + optional string hostname = 5; + } +``` diff --git a/devops/tam/tam-hld.md b/devops/tam/tam-hld.md new file mode 100644 index 000000000000..f0b9e383970b --- /dev/null +++ b/devops/tam/tam-hld.md @@ -0,0 +1,1070 @@ +# Telemetry and Monitoring + +## Highlevel Design Document + +### Rev 0.1 + +# Table of Contents + +- [Telemetry and Monitoring](#telemetry-and-monitoring) + - [Highlevel Design Document](#highlevel-design-document) + - [Rev 0.1](#rev-01) +- [Table of Contents](#table-of-contents) + - [List of Tables](#list-of-tables) + - [Revision](#revision) + - [About This Manual](#about-this-manual) + - [Scope](#scope) + - [Definition/Abbreviation](#definitionabbreviation) + - [Table 1: Abbreviations](#table-1-abbreviations) +- [1 Feature Overview](#1-feature-overview) + - [1.1 Requirements](#11-requirements) + - [1.1.1 Functional Requirements](#111-functional-requirements) + - [1.1.2 Configuration and Management Requirements](#112-configuration-and-management-requirements) + - [1.1.3 Scalability Requirements](#113-scalability-requirements) + - [1.2 Design Overview](#12-design-overview) + - [1.2.1 Basic Approach](#121-basic-approach) + - [1.2.2 Container](#122-container) + - [1.2.3 SAI Overview](#123-sai-overview) +- [2 Functionality](#2-functionality) + - [2.1 Target Deployment Use Cases](#21-target-deployment-use-cases) + - [2.2 Functional Description](#22-functional-description) +- [3 Design](#3-design) + - [3.1 Overview](#31-overview) + - [3.2 DB Changes](#32-db-changes) + - [3.2.1 CONFIG DB](#321-config-db) + - [3.2.2 APP DB](#322-app-db) + - [3.2.3 STATE DB](#323-state-db) + - [3.2.4 ASIC DB](#324-asic-db) + - [3.2.5 COUNTER DB](#325-counter-db) + - [3.3 Daemons](#33-daemons) + - [3.4 Switch State Service Design](#34-switch-state-service-design) + - [3.4.1 Orchestration Agent](#341-orchestration-agent) + - [3.4.2 Other Process](#342-other-process) + - [3.5 SyncD](#35-syncd) + - [3.6 SAI](#36-sai) + - [3.7 CLI](#37-cli) + - [3.7.1 Data Models](#371-data-models) + - [3.7.2 Configuration Commands](#372-configuration-commands) + - [3.7.2.1 Setting up switch-wide TAM attributes](#3721-setting-up-switch-wide-tam-attributes) + - [3.7.2.2 Setting up Collectors](#3722-setting-up-collectors) + - [3.7.2.2 Setting up Flow Groups](#3722-setting-up-flow-groups) + - [3.7.2.2 Setting up Samplers](#3722-setting-up-samplers) + - [3.7.3 Show Commands](#373-show-commands) + - [3.7.3.1 Listing the Switch-wide TAM attributes](#3731-listing-the-switch-wide-tam-attributes) + - [3.7.3.2 Listing the Collectors](#3732-listing-the-collectors) + - [3.7.3.3 Listing the Samplers](#3733-listing-the-samplers) + - [3.7.3.4 Listing the Flow groups](#3734-listing-the-flow-groups) + - [3.7.3.3 Listing the TAM features](#3733-listing-the-tam-features) + - [3.7.5 Debug Commands](#375-debug-commands) + - [3.7.6 REST API Support](#376-rest-api-support) +- [4 Flow Diagrams](#4-flow-diagrams) + - [4.1 Config call flow](#41-config-call-flow) +- [5 Error Handling](#5-error-handling) + - [CLI](#cli) +- [6 Serviceability and Debug](#6-serviceability-and-debug) +- [7 Warm Boot Support](#7-warm-boot-support) +- [8 Scalability](#8-scalability) +- [9 Unit Test](#9-unit-test) + - [CLI](#cli) + +## List of Tables + +[Table 1: Abbreviations](#table-1-abbreviations) + +## Revision + +| Rev | Date | Author | Change Description | +|---|-----------|------------------|-----------------------------------| +| 0.1 | 08/18/2020 | Bandaru Viswanath | New draft for SONiC TAM Infrastructure | + +## About This Manual + +This document provides general information about the TAM infrastructure implementation in SONiC. + +## Scope + + SAI community defined a Telemetry and Monitoring specification that defines a unified interface - [TAM](https://github.com/opencomputeproject/SAI/tree/master/doc/TAM) - for provisioning and monitoring various telemetry and monitoring technologies / application. + +This document describes the high level design of TAM infrastructure in SONiC. The design is intended to help implement TAM applications that use and share the common TAM infrastructure. + +## Definition/Abbreviation + +### Table 1: Abbreviations + +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| SAI | Switch Abstraction Interface | +| TAM | Telemetry and Monitoring | + +# 1 Feature Overview + +For an introduction to TAM, reader is encouraged to consult the SAI specification referred earlier. In general, TAM deals with dataplane telemetry applications that are supported by the silicon. The SONiC NOS is responsible for provisioning the dataplane application behavior, using the appropriate SAI API. + +There are many common aspects among the TAM features, as defined in the TAM specification. For example, almost all of them will require a controller to be setup, which is the intended receiver for the telemetry information. Likewise, when the applications may allow a per-flow (or a group of flows) configuration. The TAM infrastructure implements these common aspects so that applications can share the design and implementation without having to re-implement from scratch. More details on provisioning are avaiable in subsequent sections. + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +1.0 TAM infrastructure must allow configuration of common dataplane specific switch-wide attributes such as switch-identifier to be used in data-export, enterprise-id to be used in IPFIX export records etc. + +2.0 TAM infrastructure must allow configuring different types of collectors. It must support - IPv4 or IPv6, TCP or UDP for the collectors. The collectors shall be named to be referenced by individual features. + +3.0 TAM infrastucture must allow a common specification of a flow (or flow-group) definition that can be named and refererenced by individual features. + +4.0 TAM infrastucture must allow a common specification of a sampling rate definition on a given interface that can be named and refererenced by individual features. + +5.0 TAM infrastucture must support activating and deactivating individual features. + +### 1.1.2 Configuration and Management Requirements + +The TAM infrastructure supports the new management framework and KLISH CLI for configuration. + +### 1.1.3 Scalability Requirements + +In general, TAM applications are scaled based on availability of resources in hardware such as ACLs. Application specific scalability constraints may be imposed by corresponding applications. + +## 1.2 Design Overview + +### 1.2.1 Basic Approach + +The TAM infrastructure is newly developed. + +### 1.2.2 Container + +A container called 'tam' is introducted as a holder for all TAM applications, including the infrastructure. + +### 1.2.3 SAI Overview + +The SAI TAM spec specifies the TAM APIs to be used to configure the TAM functionality. Please refer to SAI-Proposal-TAM2.0-v2.0.docx in https://github.com/opencomputeproject/SAI/tree/master/doc/TAM for more details. + +# 2 Functionality + +## 2.1 Target Deployment Use Cases + +TAM infrastructure is not an independent feature. + +## 2.2 Functional Description + +TAM infrastructure is not an independent feature. + +# 3 Design + +## 3.1 Overview + +## 3.2 DB Changes + +### 3.2.1 CONFIG DB + +TAM\_SWITCH\_TABLE + +This table holds the current user configuration for the switch-wide attributes. + + + ;Defines TAM Global configuration in CONFIG_DB + + key = global ; Only one instance and + ; has a fixed key ”global". + switch-id = 1 * 5DIGIT ; Switch Id to be used in reports + enterprise-id = 1 * 5DIGIT ; IPFIX + + Example: + > keys *TAM_SWITCH* + 1) "TAM_SWITCH_TABLE|global" + + > HGETALL "TAM_SWITCH_TABLE|global" + 1)"switch-id” + 2)54325 + 3)"enterprise-id" + 4)22334 + + +TAM\_COLLECTORS\_TABLE + +This table holds the current user configuration for the Collectors. + + + ;Defines TAM Collector configuration in CONFIG_DB + + key = name ; name is collector name and should be unique. + ipaddress = ipv4_prefix ; Collector IP address + port = port_num ; Port Number + ipaddress-type = “ipv4”/”ipv6” ; IP type + protocol = ”TCP"/”UDP” ; Transport protocol to be used to reach collector + ; UDP is default + + Example: + > keys *TAM_COLLECTORS* + 1) "TAM_COLLECTOR_TABLE|collector1" + + > HGETALL "TAM_COLLECTORS_TABLE|collector1" + 1) "ipaddress" + 2) "10.20.2.1" + 3) "port" + 4) "6050" + 5) "ipaddress-type" + 6) "ipv4" + 7) "protocol" + 8) "UDP" + +TAM\_FEATURES\_TABLE + +This table holds the current user configuration for each of the TAM features. + + + ;Defines TAM Features configuration in CONFIG_DB + + key = name ; Feature name and should be among + ; IFA, DROPMONITOR, TAILSTAMPING + status = ACTIVE" / INACTIVE" ; Activate or Turnoff the feature + + Example: + > keys *TAM_FEATURES* + 1)"TAM_FEATURES_TABLE|IFA" + 2)"TAM_FEATURES_TABLE|DROPMONITOR" + 3)"TAM_FEATURES_TABLE|TAILSTAMPING" + + > hgetall "TAM_FEATURES_TABLE|ifa” + 1) "status" + 2) "ACTIVE" + +TAM\_SAMPLINGRATE\_TABLE + +This table holds sampling rates used across the TAM features. + + + ;Defines TAM Sample-Rate configuration in CONFIG_DB + + key = name ; Sampler name and should be unique + + sampling-rate = 1 * 5DIGIT ; Sampling rate. One packet in every ‘rate’ packets + ; will be sampled + + Example: + > keys *TAM_SAMPLINGRATE_TABLE* + 1)"TAM_SAMPLINGRATE_TABLE|aggressive" + 2)"TAM_SAMPLINGRATE_TABLE|lazy" + + > hgetall "TAM_SAMPLINGRATE_TABLE|aggressive” + 1) "sampling-rate" + 2) 1000 + + +TAM\_FLOWGROUP\_TABLE +This table holds various flow-group references that are setup to be used with the TAM features. + + ;Defines TAM Flow Group configuration in CONFIG_DB + + key = name ; Flowgroup name and should be unique + id = 1 * 5DIGIT ; Unique identification for a flow-group + table-name = 1*255VCHAR ; ACL Table reference + + Example: + > keys *TAM_FLOWGROUP_TABLE* + 1)"TAM_FLOWGROUP_TABLE|websrv1" + 2)"TAM_FLOWGROUP_TABLE|storagecluster2" + + > hgetall "TAM_FLOWGROUP_TABLE|storagecluster2" + 1) "id" + 2) 1000 + 3) "table-name" + 4) "TAM" + + +### 3.2.2 APPL DB + +TAM\_APPL\_SWITCH\_TABLE + +This table holds the current operational switch-wide attributes for the TAM features. + + ;Operational TAM Global Status in STATE_DB + + key = global ; Only one instance and + ; has a fixed key ”global" + op-switch-id = 1 * 5DIGIT ; Currently used switch-id + op-enterprise-id = 1 * 5DIGIT ; Currently used enterprise-id + + Example: + > keys *TAM_APPL_SWITCH* + 1) "TAM_APPL_SWITCH_TABLE|global" + + > HGETALL "TAM_APPL_SWITCH_TABLE|global" + 1)”op-switch-id” + 2)54325 + 3)”op-enterprise-id" + 4)22334 + +### 3.2.3 STATE DB + +TAM\_STATE\_FEATURES\_TABLE + +This table holds the current operational status for the TAM features. + + + ;Reflects TAM Features status in STATE_DB + key = name ; Feature name and should be among + ; IFA, DROPMONITOR, TAILSTAMPING + op-status = ACTIVE" / INACTIVE" / "UNSUPPORTED" / "INSUFFICIENT_RESOURCES" + ; current status of the feature + + Example: + > keys *TAM_STATE_FEATURES* + 1)"TAM_STATE_FEATURES_TABLE|IFA" + 2)"TAM_STATE_FEATURES_TABLE|DROPMONITOR" + 3)"TAM_STATE_FEATURES_TABLE|TAILSTAMPING" + + > hgetall "TAM_FEATURES_TABLE|IFA” + 1) "op-status" + 2) "UNSUPPORTED" + + +### 3.2.4 ASIC DB + +N/A + +### 3.2.5 COUNTER DB + +N/A + +## 3.3 Daemons + +A new daemon called `tammgrd` is introduced, which is responsible for managing the common attributes for all the TAM features. This daemon listens for changes on the `TAM_SWITCH_TABLE` in the CONFIG_DB and `TAM_STATE_FEATURES_TABLE` in the STATE_DB and effects the changes in global attributes only when all features are inactive. The `TAM_APPL_SWITCH_TABLE` in APPL_DB reflects the currently operational attributes. + +This daemon is also responsible for setting the initial/default values for the global attributes in the absense of any user configuration. The default values are derived from the system mac-address. + +## 3.4 Switch State Service Design + +### 3.4.1 Orchestration Agent + +Orchestration Agent determines the support for the individual TAM features based on the underlysing SAI capabilities. The support status is reflected in the SWITCH_TABLE as part of the APPL_DB. Individual TAM features use this status to determine whether the configuration should be procecced or not. Likewise, the Management infrastructure uses this to determine whether to allow the user configuration or not. + +### 3.4.2 Other Process + +N/A + +## 3.5 SyncD + +N/A + +## 3.6 SAI + +The SAI TAM API spec defines all TAM APIs supported in SAI. Please refer to SAI-Proposal-TAM2.0-v2.0.docx in https://github.com/opencomputeproject/SAI/tree/master/doc/TAM for more details. + +## 3.7 CLI + +### 3.7.1 Data Models + +The user facing data model is based on OpenConfig compatible TAM yang model. The backend data model (SONiC YANG) will use the formats in CONFIG_DB, APPL_DB and STATE_DB. See above sections. + +### 3.7.2 Configuration Commands + +#### 3.7.2.1 Setting up switch-wide TAM attributes + +Two switch-wide attributes are supported by the TAM infrastructure. Any changes to the switch-wide attributes are effected only when none of TAM features are active. + +| **Attribute** | **Description** | +|--------------------------|-------------------------------------| +| `switch-id` | A 32-bit idenitifier that uniquely identifies the switch, and is used in the telemetry reports. When not configured, the last 16-bits from the system mac address are used. | +| `enterprise-id` | A 32-bit identifier that is used in the IPFIX telemetry reports. When not configured, the last 16-bits from the system mac address are used. | + +The command syntax for setting up the switch-wide TAM attributes are as follows: + +``` +sonic (config-tam)# [no] switch-id +sonic (config-tam)# [no] enterprise-id +``` + +#### 3.7.2.2 Setting up Collectors + +A collector is typically a machine reachable from the switch, where the telemetry reports are sent. The collectors are application specific, in the sense that the collectors must understand the data format of the telemetry reports they receive. A collector may be capable of processing reports from one or more applications. + +TAM infrastructure allows us to create a collector by specifying the reachability information and associating a reference to this collector. This reference, a name, can be specified as a collector by individual feature configurations. + +A collector, while being associated with any TAM application, can't be removed. The `no` command fails citing existing association. + +The following collector attribtes are supported. + +| **Attribute** | **Description** | +|--------------------------|-------------------------------------| +| `name` | A string that uniquely identifies the collector, and will be referrenced from other configurations | +| `ip-address-type` | Specifies the version of IP protocol to be used for reaching the collector. Valid values are `"IPv4"` and `"IPv6"` | +| `ip-address` | The IP address of the collector | +| `protocol` | Specifies the transport protocol to be used for reaching the collector. Valid values are `"UDP"` and `"TCP"`. When not specified, `"UDP"` is used as default. | +| `port` | Port number on which the Collector is listening for the reports. | + +The command syntax for setting up the collectors are as follows: + +``` +sonic (config-tam)# collector type {ipv4 | ipv6} ip port [protocol { UDP | TCP }] + +sonic (config-tam)# no collector +``` + +#### 3.7.2.2 Setting up Flow Groups + +A flow-group is a packet match criterion that defines a set of flows that are of interest. For example, `all packets destined for a the webserver at 10.10.1.1` is an example flow group. Multiple flows can match a given flow-group criterion. Typically, a flow-group is defined with a combination of L2 and L3 header fields, with some fields specified as wild-cards. + +TAM infrastructure allows us to create a flow-group by specifying the tuple information and associating a reference to this flow-group from TAM applications. A flow-group can be removed with the `no` form of the command. Flow Groups that are actively referenced in other applications can't be removed. + +The following flow-group attribtes are supported. Except the `name` attribute, all the other attributes are optional. When not specified, they default to the equivalent of a wildcard in the packet matching. + + +| **Attribute** | **Description** | +|--------------------------|-------------------------------------| +| `name` | A string that uniquely identifies the flow-group, and will be referrenced from other configurations | +| `src-mac` | Source MAC address of the packets belonging to the flow-group | +| `dst-mac` | Destination MAC address of the packets belonging to the flow-group | +| `ethertype` | Ethertype of the packets belonging to the flow-group | +| `src-ip` | Source IP address of the packets belonging to the flow-group | +| `dst-ip` | Destination IP address of the packets belonging to the flow-group | +| `l4-src-port` | Source Port (L4) of the packets belonging to the flow-group | +| `l4-dst-port` | Destination Port (L4) of the packets belonging to the flow-group | +| `protocol` | Protocol field of the ip-header of the packets belonging to the flow-group | +| `priority` | Priority of the flow-group, among the other flow-groups that are created. Range is 1 - 1024, Default value is 100 | + + +The command syntax for setting up the flow-groups is as follows: + +``` +sonic (config-tam)# flow-group [src-mac ] [dst-mac ] [ethertype ] [src-ip ] [dst-ip ] [src-l4-port ] [dst-l4-port ] [protocol ] [priority ] + +sonic (config-tam)# no flow-group +``` + +#### 3.7.2.3 Attaching a Flow Group to an interface + +Created flow-groups can be attached to an interface. This is accomplished via the interface mode as shown below. The flow-group can be attached to more than one interface. + +``` +sonic (config) # interface +sonic (config-if-EthernetXY)# [no] flow-group +``` + +#### 3.7.2.4 Setting up Samplers + +TAM infrastructure supports setting up a sampling configuration and refer to this sampler configuration from the application which support sampling for the traffic. + +A sampler, while being associated with any TAM application, can't be removed. The `no` command fails citing existing association. + +The following sampling attribtes are supported. + +| **Attribute** | **Description** | +|--------------------------|-------------------------------------| +| `name` | A string that uniquely identifies the sampler, and will be referrenced from other configurations | +| `sample-rate` | Sampling rate, one in every `sample-rate` packets will be sampled | + +The command syntax for setting up the samplers are as follows: + +``` +sonic (config-tam)# sampler rate + +sonic (config-tam)# no sampler + +``` + +### 3.7.3 Show Commands + +#### 3.7.3.1 Listing the Switch-wide TAM attributes + +The following command lists all the switch-wide TAM attributes currently in use. + +``` +sonic # show tam switch +``` +An example invocation is as below + +``` +sonic # show tam switch + +TAM Device information +---------------------- +Switch ID : 23456 +Enterprise ID : 1234 +``` + +#### 3.7.3.2 Listing the Collectors + +The following command lists the details for all collectors or for a specific collector. + +``` +sonic # show tam collectors [] +``` +Sample usage shown below. + +``` +sonic # show tam collectors + +Name IP Address Port Protocol +---------------- -------------- ------ -------- +IFA_Col_i19 192.168.78.121 7071 UDP +MOD_Col_m16 192.168.78.123 7076 UDP + +sonic-cli# show tam collectors IFA_Col_i19 + +Name : IFA_Col_i19 +IP Address : 192.168.78.121 +Port : 7071 +Protocol : UDP +``` + +#### 3.7.3.3 Listing the Samplers + +The following command lists the details for all samplers or for a specific sampler. + +``` +sonic # show tam samplers [] +``` +Sample usage shown below. + +``` +sonic # show tam samplers + +Name Sample Rate +----------- ----------- +sflow_low 1 +aggresive 2000 + +sonic-cli# show tam samplers aggresive + +Name : aggresive +Sample Rate : 2000 +``` + +#### 3.7.3.4 Listing the Flow groups + +The following command lists the details for all flow-groups or for a specific flow-group. Note that only explicitly configured tuples are displayed. + +``` +sonic # show tam flow-groups [] +``` +Sample usage shown below. + +``` +sonic # show tam flow-groups + +Flow Group Name : udp_port_239 + Id : 4025 + Priority : 100 + SRC IP : 10.72.195.23 + DST L4 Port : 239 + Ingress Intf : Ethernet20 +Packet Count : 10584 + +Flow Group Name : udp_port_241 + Id : 4022 + Priority : 99 + SRC Port : 1906 + DST L4 Port : 241 +Packet Count : 8654367 + +sonic # show tam flow-groups udp_port_239 + +Flow Group Name : udp_port_239 + Id : 4025 + Priority : 100 + SRC IP : 10.72.195.23 + DST L4 Port : 239 +Packet Count : 10584 +``` + +#### 3.7.3.3 Listing the TAM features + +The following command lists the current status for all TAM features or for a specific feature. + +``` +sonic # show tam features [ { ifa | drop-monitor | tail-stamping }] +``` +Sample usage shown below. + +``` +sonic # show tam features + +Name Status +----------- --------- +ifa Active +drop-monitor Active + +sonic-cli# show tam features ifa + +Name : ifa +Status : Active +``` + +### 3.7.5 Debug Commands +N/A + +### 3.7.6 REST API Support + +#### Yang-based REST API based on the openconfig-tam module defintions + +##### Obtaining the status of all TAM feautures on the switch + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/features-state +* Response format +```json +{ + "openconfig-tam:features-state": { + "feature": [ + { + "feature-ref": "string", + "state": { + "feature-ref": "string", + "op-status": "ACTIVE" + } + } + ] + } +} +``` + +##### Obtaining a status of a specific TAM feature + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/features-state/feature={feature-ref} +* Response format +```json +{ + "openconfig-tam:feature": [ + { + "feature-ref": "string", + "state": { + "feature-ref": "string", + "op-status": "ACTIVE" + } + } + ] +} +``` + +##### Activating/De-activating a specific TAM feature + +* Method : PATCH +* URI : /restconf/data/openconfig-tam:tam/features/feature={feature-ref} +* Data format +```json +{ + "openconfig-tam:feature": [ + { + "feature-ref": "string", + "config": { + "feature-ref": "string", + "status": "ACTIVE" + } + } + ] +} +``` + +##### Obtaining TAM switch-wide attributes + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/switch +* Response format +```json +{ + "openconfig-tam:switch": { + "config": { + "switch-id": 0, + "enterprise-id": 0 + }, + "state": { + "switch-id": 0, + "enterprise-id": 0, + "op-switch-id": 0, + "op-enterprise-id": 0 + } + } +} +``` + +##### Setting-up TAM switch-wide attribute : switch-id + +* Method : PATCH +* URI : /restconf/data/openconfig-tam:tam/switch/config/switch-id +* Data format +```json +{ + "openconfig-tam:switch-id": 0 +} +``` +##### Setting-up TAM switch-wide attribute : enterprise-id + +* Method : PATCH +* URI : /restconf/data/openconfig-tam:tam/switch/config/enterprise-id +* Data format +```json +{ + "openconfig-tam:enterprise-id": 0 +} +``` + +##### Resetting the TAM switch-wide attributes to defaults + +* Method : DELETE +* URI : /restconf/data/openconfig-tam:tam/switch/config/switch-id +* URI : /restconf/data/openconfig-tam:tam/switch/config/enterprise-id + +##### Obtaining all of the collectors + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/collectors +* Response format +```json +{ + "openconfig-tam:collectors": { + "collector": [ + { + "name": "string", + "config": { + "name": "string", + "ip": "string", + "port": 0, + "protocol": "UDP", + "encapsulation": "NONE" + }, + "state": { + "name": "string", + "ip": "string", + "port": 0, + "protocol": "UDP", + "encapsulation": "NONE" + } + } + ] + } +} +``` + +##### Obtaining a specific collector + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/collectors/collector={name} +* Response format +```json +{ + "openconfig-tam:collector": [ + { + "name": "string", + "config": { + "name": "string", + "ip": "string", + "port": 0, + "protocol": "UDP", + "encapsulation": "NONE" + }, + "state": { + "name": "string", + "ip": "string", + "port": 0, + "protocol": "UDP", + "encapsulation": "NONE" + } + } + ] +} +``` +##### Creating a collector + +* Method : PATCH +* URI : /restconf/data/openconfig-tam:tam/collectors +* Data format +```json +{ + "openconfig-tam:collectors": { + "collector": [ + { + "name": "string", + "config": { + "name": "string", + "ip": "string", + "port": 0, + "protocol": "UDP", + "encapsulation": "NONE" + } + } + ] + } +} +``` +##### Deleting a collector + +* Method : DELETE +* URI : /restconf/data/openconfig-tam:tam/collectors/collector={name} + +##### Obtaining all of the Samplers + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/samplers +* Response format +```json +{ + "openconfig-tam:samplers": { + "sampler": [ + { + "name": "string", + "config": { + "name": "string", + "sampling-rate": 0 + }, + "state": { + "name": "string", + "sampling-rate": 0 + } + } + ] + } +} +``` + +##### Obtaining a specific sampler + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/samplers/sampler={name} +* Response format +```json +{ + "openconfig-tam:sampler": [ + { + "name": "string", + "config": { + "name": "string", + "sampling-rate": 0 + }, + "state": { + "name": "string", + "sampling-rate": 0 + } + } + ] +} +``` +##### Creating a Sampler + +* Method : PATCH +* URI : /restconf/data/openconfig-tam:tam/samplers +* Data format +```json +{ + "openconfig-tam:samplers": { + "sampler": [ + { + "name": "string", + "config": { + "name": "string", + "sampling-rate": 0 + } + } + ] + } +} +``` +##### Deleting a Sampler + +* Method : DELETE +* URI : /restconf/data/openconfig-tam:tam/samplers/sampler={name} + + +##### Obtaining all of the flow-groups + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/flowgroups +* Response format +```json +{ + "openconfig-tam:flowgroups": { + "flowgroup": [ + { + "name": "string", + "config": { + "name": "string", + "id": 0, + "priority": 0, + "ip-version": "UNKNOWN" + }, + "state": { + "name": "string", + "id": 0, + "priority": 0, + "ip-version": "UNKNOWN" + }, + "l2": { + "config": { + "source-mac": "string", + "source-mac-mask": "string", + "destination-mac": "string", + "destination-mac-mask": "string", + "ethertype": "string" + }, + "state": { + "source-mac": "string", + "source-mac-mask": "string", + "destination-mac": "string", + "destination-mac-mask": "string", + "ethertype": "string" + } + }, + "ipv4": { + "config": { + "source-address": "string", + "destination-address": "string", + "dscp": 0, + "protocol": "string", + "hop-limit": 0 + }, + "state": { + "source-address": "string", + "destination-address": "string", + "dscp": 0, + "protocol": "string", + "hop-limit": 0 + } + }, + "ipv6": { + "config": { + "source-address": "string", + "source-flow-label": 0, + "destination-address": "string", + "destination-flow-label": 0, + "dscp": 0, + "protocol": "string", + "hop-limit": 0 + }, + "state": { + "source-address": "string", + "source-flow-label": 0, + "destination-address": "string", + "destination-flow-label": 0, + "dscp": 0, + "protocol": "string", + "hop-limit": 0 + } + }, + "transport": { + "config": { + "source-port": "string", + "destination-port": "string", + "tcp-flags": [ + "string" + ] + }, + "state": { + "source-port": "string", + "destination-port": "string", + "tcp-flags": [ + "string" + ] + } + } + } + ] + } +} +``` +##### Creating a flow-group + +* Method : PATCH +* URI : /restconf/data/openconfig-tam:tam/flowgroups +* Data format +```json +{ + "openconfig-tam:flowgroups": { + "flowgroup": [ + { + "name": "string", + "config": { + "name": "string", + "id": 0, + "priority": 0, + "ip-version": "UNKNOWN" + }, + "l2": { + "config": { + "source-mac": "string", + "source-mac-mask": "string", + "destination-mac": "string", + "destination-mac-mask": "string", + "ethertype": "string" + } + }, + "ipv4": { + "config": { + "source-address": "string", + "destination-address": "string", + "dscp": 0, + "protocol": "string", + "hop-limit": 0 + } + }, + "ipv6": { + "config": { + "source-address": "string", + "source-flow-label": 0, + "destination-address": "string", + "destination-flow-label": 0, + "dscp": 0, + "protocol": "string", + "hop-limit": 0 + } + }, + "transport": { + "config": { + "source-port": "string", + "destination-port": "string", + "tcp-flags": [ + "string" + ] + } + } + } + ] + } +} +``` +##### Deleting a flow-group + +* Method : DELETE +* URI : /restconf/data/openconfig-tam:tam/flowgroups/flowgroup={name} + + # 4 Flow Diagrams + +## 4.1 Config call flow + +All the configuration is stored in the CONFIG_DB by the management framework. + +# 5 Error Handling + +## CLI + +* CLI configuration sanity will be enforced by the CLI handler & CVL and any invalid configuration is rejected. An error is displayed to the user notifying the reason for rejection of the configuration. + +# 6 Serviceability and Debug + +N/A + +# 7 Warm Boot Support + +No special handling is done for the warm boot case. The TAM configuration is restored from the Config DB and TAM functionality will continue to work as it is through a warm boot. + +# 8 Scalability + +N/A + +# 9 Unit Test + +## CLI + +The CLI testcases are included as part of individual feature testcases. + +# Broadcom Internal Information : To be removed before publishing externally. + +## Revision History + +* This document has been derived from the earlier (Arlo+/Buzznik) TAM feature HLDs. The common TAM functionality has been brought into a single place in this HLD. + +## Key notes + +* The INSUFFICIENT_RESOURCES status for a TAM feature is introduced primarily to cover the need where firmware based telemetry features can't be activated because there are no free embedded-cores. This status remains unused in Buzznik+ timeframe since the number of cores on any slicion are equal/greater than the applications, but will need to be addressed in subsequent releases. + +## Specific Limitations + diff --git a/devops/tam/tam-inband-flow-analyser.md b/devops/tam/tam-inband-flow-analyser.md new file mode 100644 index 000000000000..d4e328b96f45 --- /dev/null +++ b/devops/tam/tam-inband-flow-analyser.md @@ -0,0 +1,762 @@ +# IFA : Inband Flow Analyzer + +## Highlevel Design Document + +### Rev 0.3 + +# Table of Contents + +- [IFA : Inband Flow Analyzer](#ifa---inband-flow-analyzer) + * [Highlevel Design Document](#highlevel-design-document) + + [Rev 0.1](#rev-01) +- [Table of Contents](#table-of-contents) + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definition-abbreviation) + + [Table 1: Abbreviations](#table-1--abbreviations) +- [1 Feature Overview](#1-feature-overview) + * [1.1 Requirements](#11-requirements) + + [1.1.1 Functional Requirements](#111-functional-requirements) + + [1.1.2 Configuration and Management Requirements](#112-configuration-and-management-requirements) + + [1.1.3 Scalability Requirements](#113-scalability-requirements) + * [1.2 Design Overview](#12-design-overview) + + [1.2.1 Basic Approach](#121-basic-approach) + + [1.2.2 Container](#122-container) + + [1.2.3 SAI Overview](#123-sai-overview) +- [2 Functionality](#2-functionality) + * [2.1 Target Deployment Use Cases](#21-target-deployment-use-cases) + * [2.2 Functional Description](#22-functional-description) +- [3 Design](#3-design) + * [3.1 Overview](#31-overview) + * [3.2 DB Changes](#32-db-changes) + + [3.2.1 CONFIG DB](#321-config-db) + + [3.2.2 APP DB](#322-app-db) + + [3.2.3 STATE DB](#323-state-db) + + [3.2.4 ASIC DB](#324-asic-db) + + [3.2.5 COUNTER DB](#325-counter-db) + * [3.3 Daemons](#33-daemons) + * [3.4 Switch State Service Design](#34-switch-state-service-design) + + [3.4.1 Orchestration Agent](#341-orchestration-agent) + + [3.4.2 Other Process](#342-other-process) + * [3.5 SyncD](#35-syncd) + * [3.6 SAI](#36-sai) + * [3.7 CLI](#37-cli) + + [3.7.1 Data Models](#371-data-models) + + [3.7.2 Configuration Commands](#372-configuration-commands) + - [3.7.2.1 Activating and Deactivating IFA](#3721-activating-and-deactivating-ifa) + - [3.7.2.2 Setting up flows for monitoring with IFA](#3722-setting-up-flows-for-monitoring-with-ifa) + + [3.7.3 Show Commands](#373-show-commands) + - [3.7.3.1 Listing the IFA attributes](#3731-listing-the-ifa-attributes) + - [3.7.3.1 Listing the IFA sessions](#3731-listing-the-ifa-sessions) + + [3.7.4 Sample Workflow](#374-sample-workflow) + - [Ingress Node configuration](#ingress-node-configuration) + - [Intermediate/Transit Node configuration](#intermediate-transit-node-configuration) + - [Egress Node configuration](#egress-node-configuration) + + [3.7.5 Debug Commands](#375-debug-commands) + + [3.7.6 REST API Support](#376-rest-api-support) +- [4 Flow Diagrams](#4-flow-diagrams) + * [4.1 Config call flow](#41-config-call-flow) +- [5 Error Handling](#5-error-handling) + * [CLI](#cli) + * [IFAOrch](#ifaorch) +- [6 Serviceability and Debug](#6-serviceability-and-debug) +- [7 Warm Boot Support](#7-warm-boot-support) +- [8 Scalability](#8-scalability) +- [9 Unit Test](#9-unit-test) + * [CLI](#cli-1) +- [Broadcom Internal Information : To be removed before publishing externally.](#broadcom-internal-information---to-be-removed-before-publishing-externally) + * [Key notes](#key-notes) + * [Specific Limitations](#specific-limitations) + +## List of Tables + +[Table 1: Abbreviations](#table-1-abbreviations) + +## Revision + +| Rev | Date | Author | Change Description | +|---|-----------|------------------|-----------------------------------| +| 0.1 | 06/14/2019 | Naveen Kumar Aketi | Initial version | +| 0.2 | 10/16/2019 | Naveen Kumar Aketi | Version 0.2 as per new design | +| 0.3 | 08/18/2020 | Bandaru Viswanath | Major update to accomodate enhancements to use new TAM infrastructure, DB schmas, UI and IFA 2.0 support | + +## About This Manual + +This document provides general information about the Inband Flow Analyzer (IFA) feature implementation in SONiC. + +## Scope + +The term Inband Telemetry refers to the set of technologies of the Switch, that embed useful metadata as part of packets that are transiting the Switch. SAI community defined a Telemetry and Monitoring specification that defines a unified interface - [TAM](https://github.com/opencomputeproject/SAI/tree/master/doc/TAM) - for provisioning and monitoring these technologies. Inband Flow Analyzer feature is one such Inband Telemetry technologies. + +The design as specified in this document uses the common [SONiC TAM](tam.hld) infrastructure. + +This document describes the high level design of Inband Flow Analyzer feature in SONiC. + +## Definition/Abbreviation + +### Table 1: Abbreviations + +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| SAI | Switch Abstraction Interface | +| TAM | Telemetry and Monitoring | +| IFA | Inband Flow Analyzer | + +# 1 Feature Overview + +Inband Flow Analyzer (IFA) records flow specific information from switches across a network for specific flows. It is described at the [IETF draft](https://datatracker.ietf.org/doc/draft-kumar-ippm-ifa). The protocol defines an IFA header to mark the flow and direct the collection of analyzed metadata per marked packet per hop across the network. Some of the text in the document is taken from the above IETF draft, for contextual purposes. + +IFA performs flow analysis, and possible actions on the flow data, inband. Once a flow is enabled for analysis, an Ingress node makes a copy of the flow or samples the live traffic flow, or tags a live traffic flow for analysis and data collection. Copying of a flow is done by sampling or cloning the flow. These new packets are representative packets of the original flow and possess the exact same characteristics as the original flow. This means that IFA packets traverse the same path in the network and same queues in the networking element as the original packet would. + +The intermediate nodes keep appending their own metadata to the IFA-tagged packets before forwarding them. The terminating node is responsible for terminating the IFA flow by summarizing/extracting the metadata of the entire path and sending it to a designated Collector. A network monitoring application on a Collector can analyze the telemetry information provided by IFA feature and provide full visibility of a network by providing metrics such as latency, packet loss and full network path that packet travels. A complete description of the functionality can be obtained from the IETF draft mentioned above. + +The IFA feature in SONiC allows the user to setup flow monitoring sessions for specific flows. The switch may be marked as an ingress node, a transit node or a terminating node for a flow. A Collector, identified by an IP address and associated transport parameters, can be configured on the terminating nodes to send the extracted metadata. More details on provisioning are avaiable in subsequent sections. + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +1.0 IFA feature allows user to configure IFA session on a given switch and send the telemetry metadata to a specified collector. IFA session is defined by flow classifiers that are used to identify a flow that needs to be monitored for inband telemetry. + +1.1 IFA feature is accomplished by configuring IFA session on various nodes that act as ingress, intermediate and egress nodes. The IFA role is per flow on a Swich. For example, any given switch can act as an ingress node for one flow and as an intermediate node for another flow. + +1.2 Ingress nodes sample the flow and tags the sampled packets for analysis and data collection, and insert own metadata. + +1.3 Intermediate node is responsible for identifying IFA tagged flows and insertion of telemetry metadata. + +1.4 Egress node is responsible for terminating the IFA flow by summarizing the telemetry data of the entire path and sending it to collector as well as dropping the sampled packets. + +2.0 IFA provisioning as listed below. + +2.1 TAM switch identifier to uniquely identify a switch in network and IFA header of the reports carry this information. + +2.2 Enterprise Id to be used for the IPFIX export records. + +2.3 ACL configuration to identify a flow-group to insert IFA metadata. + +2.4 TAM sampler configuration to indicate the sampling rate for the flow. + +2.5 TAM collector configuration that can be attached to IFA flow on egress device to forward collected telemetry data. + +3.0 IFA feature is by default de-activated on the switch, and requires an explicit user intervention to activate it in the configuration. + +3.1 If the IFA feature can't be supported/activated on the switch, any attempt to activate the IFA feature returns appropriate failure. + +4.0 The IFA feature is supported on physical and LAG ports. + +4.1 The IFA feature supports dynamic-port-breakout and aliasing features. + +5.0 The IFA feature will be a part of the TAM container along with other TAM features. + +### 1.1.2 Configuration and Management Requirements + +The TAM IFA feature supports the new management framework and KLISH CLI for configuration. The following configuration and status information is available. + +- To activate / de-activate IFA feature +- To create/clear appropriate IFA configuration for ingress, intermediate and egress nodes, on a per-flow group basis. +- To display current status and statistics for the IFA on a per flow-group basis. + +### 1.1.3 Scalability Requirements + +- Number of IFA sessions that can be supported is proportional to the availability of resources in hardware such as ACLs. No specific constraints are imposed. +- Only a single collector is supported. + +## 1.2 Design Overview + +### 1.2.1 Basic Approach + +The IFA feature is newly developed. + +### 1.2.2 Container + +A container called 'tam' exists as a container for all TAM applications, including the IFA application. + +### 1.2.3 SAI Overview + +The SAI TAM spec specifies the TAM APIs to be used to configure the TAM functionality. Please refer to [SAI-Proposal-TAM2.0-v2.0.docx](https://github.com/opencomputeproject/SAI/tree/master/doc/TAM) for more details. + +# 2 Functionality + +## 2.1 Target Deployment Use Cases + +IFA is used to proactively monitor the network for faults and performance bottlenecks, it aids in detecting data plane faults and isolating them to a given device or location and assists in planning network capacity and projecting future load. Some of the key usecases are - + +- SLA conformance that require end-to-end path latency for targeted flows +- Idenitfying switches and flows that are undergoing congestion +- Identifying network bandwidth usage patterns +- Packet path and path change analysis + +## 2.2 Functional Description + +The IFA feature in SONiC allows the user to setup flow monitoring sessions for specific flows. The switch may be marked as an ingress node, a transit node or a terminating node for a flow. + +When IFA feature is activated, the switch will behave as intermediate node and perform IFA intermediate node actions. What it means is that the switch will append its own metadata to all IFA-tagged flows transiting the switch. Non IFA-tagged flows are not impacted. There may be multiple IFA intermediate nodes in the path for the IFA-tagged flows. + +When the switch is configured as an ingress node for any flow (or set of flows), the switch - + + - Makes a copy of a sampled packet from the flow based upon the configured sampling rate. + - Adds IFA tag to the sampled packet + - Adds its own metadata to the packet. + +The sampled packet is then forwarded along the same path as the original packet. + +When the switch is configured as an egress node for any flow (or set of flows), the switch - + +- Collectes the IFA sampled packets belonging to the flow +- Adds its own metadata +- Summarizes the telemetry data for the entire path, encapsulates the telemetry data as payload and sends it to collector. + +The sampled packet is then dropped at the egress node. + +At all other non-IFA activated nodes, IFA sampled packet is forwarded as a normal packet in the network. + +***Example IFA topology illustrating the packet paths and metadata*** + +![IFA topology](sample-ifa-topology.png) + +# 3 Design + +## 3.1 Overview + +***IFA Architecture*** + +![IFA architecture](ifa-arch.JPG) + +The above diagram illustrates the architecture of the IFA feature within SONiC. + +Below is the call flow sequence specified in above architecture diagram + +1 IFA and ACL configuration from UI is saved to CONFIG DB. + +2 IFA Manager reads IFA configuration from CONFIG DB, processes and validates the same. + +3 IFA Manager updates valid IFA configuration to APPL DB. + +4 ACL ORCH reads ACL configuration from CONFIG DB. + +5 IFA ORCH reads IFA configuration from APPL DB and creates/deletes TAM INT IFA objects. + +6 IFA ORCH checks for ACL table and ACL rule specified in IFA configuration. + +7 ACL ORCH will make SAI API calls to update ASIC DB. + +8 ACL ORCH notifies IFA ORCH about ACL rule creation/deletion. + +9 IFA ORCH attaches/detaches the TAM INT IFA object to/from ACL ENTRY. + +10, 11, 12 SYNCD reads IFA and ACL information from ASIC DB and configures ASIC accordingly. + +A CLI reads IFA ACL counters from COUNTER DB and displays it as output of show commands. + +## 3.2 DB Changes + +### 3.2.1 CONFIG DB + +TAM\_IFA\_SESSIONS\_TABLE + + ;Defines TAM IFA configuration in CONFIG_DB + + key = name ; name is ifa flow name and should be unique. + flowgroup = 1*255VCHAR ; Flow group reference + collector = 1*255VCHAR ; Collector Reference + sample-rate = 1*255VCHAR ; Sampler reference + node-type = "INGRESS"/"EGRESS"/"INTERMEDIATE" + ; IFA Node type, INTERMEDIATE is the default + + Example: + > keys *TAM_IFA_SESSIONS* + 1) "TAM_IFA_SESSIONS_TABLE|ifa1" + + > hgetall "TAM_IFA_SESSIONS_TABLE|ifa1" + 1) "flowgroup" + 2) "websrvrflows" + 3) "collector" + 4) "ifacol1" + 5) "sample-rate" + 6) "aggressive" + 7) "node-type" + 8) "ingress" + +### 3.2.2 APPL DB + +TAM\_IFA\_TABLE + +This table holds the current operational attributes for the IFA feature. + + ;Operational IFA Global Status in APPL_DB + + key = global ; Only one instance and + ; has a fixed key ”global" + op-switch-id = 1 * 5DIGIT ; Currently used switch-id + op-enterprise-id = 1 * 5DIGIT ; Currently used enterprise-id + + Example: + > keys *TAM_IFA_TABLE* + 1) "TAM_IFA_TABLE|global" + + > HGETALL "TAM_IFA_TABLE|global" + 1)”op-switch-id” + 2)54325 + 3)”op-enterprise-id" + 4)22334 + +TAM\_IFA\_FLOW\_TABLE + + ;Defines a TAM INT IFA flow configuration + + key = name ; name is flow name and should be + unique + acl-table-name = table-name ; Parameter to map to acl table to + the flow. + acl-rule-name = rule-name ; Parameter to map to acl rule to + the flow. + dst-ipaddress-type = "ipv4" / "ipv6" ; Optional parameter that indicates + collector IP address type. + dst-ipaddress = ipv4_prefix / ipv6_prefix ; Optional parameter that indicates + collector IP address. + dst-port = 1 * 4DIGIT ; Optional parameter that indicates + collector UDP port number + src-ipaddress = ipv4_prefix / ipv6_prefix ; Optional parameter that indicates + source IP address. + src-port = 1 * 4DIGIT ; Optional parameter that indicates + source port number + sampling-rate = 1 * 5DIGIT ; Optional parameter that indicates + the sampling rate for the packets + belonging to the flow that need + to be tagged with IFA meta-data. + One packet in every + ‘sampling-rate’ packets will be + tagged with IFA Metadata. + Note: This parameter is + applicable for switches + configured as the type + ingress-node, and will be ignored + for the other types. If this + field is not specified, it + indicates a 1:1 sampling. + Range is 1 to 10000. A value of 1 + indicates that all packets + belonging to this flow will be + sampled. + + Example: + > KEYS *TAM_INT_IFA_FLOW* + 1) "TAM_INT_IFA_FLOW_TABLE:F1" + 2) "TAM_INT_IFA_FLOW_TABLE:F2" + + > HGETALL TAM_INT_IFA_FLOW_TABLE:F1 + 1) "acl-table-name" + 2) "T1" + 3) "acl-rule-name" + 4) "R1" + 5) "dst-ipaddress-type" + 6) "ipv4" + 7) "dst-ipaddress" + 8) "10.20.30.40" + 9) "dst-port" + 10) "2233" + 11) "src-port" + 12) "9070" + 13) "src-ipaddress" + 14) "10.52.141.231" + + > HGETALL TAM_INT_IFA_FLOW_TABLE:F2 + 1) "acl-table-name" + 2) "T2" + 3) "acl-rule-name" + 4) "R2" + 5) "sampling-rate" + 6) "1000" + +### 3.2.3 STATE DB + +N/A + +### 3.2.4 ASIC DB + +N/A + +### 3.2.5 COUNTER DB + +N/A + +## 3.3 Daemons + +IFA manager daemon runs as part of TAM docker. IFA manager processes IFA configuration from CONFIG DB, validates for consistency and completeness of IFA configuration and updates valid IFA configuration to APPL DB. + +## 3.4 Switch State Service Design + +### 3.4.1 Orchestration Agent + +A new orchestration agent class, IFAOrch is added to convert the incoming IFA config to ASIC configuration. IFAOrch subscribes to the IFA tables of APPL DB and converts the configuration to the SAI TAM API call sequence described in section 3.6. + +IFAOrch maintains data pertaining to all the currently configured IFA entities and the associated TAM object bindings. TAM object bindings are re-used wherever possible. + +IFAOrch checks for support for the IFA feature in the silicon using SAI capability API and sets the field `tam_int_ifa_supported` to `True` in the `SWITCH_TABLE` of APPL_DB under the key `switch`. + +### 3.4.2 Other Process + +N/A + +## 3.5 SyncD + +N/A + +## 3.6 SAI + +The SAI TAM API spec defines all TAM APIs supported in SAI. Please refer to SAI-Proposal-TAM2.0-v2.0.docx at [TAM](https://github.com/opencomputeproject/SAI/tree/master/doc/TAM) for more details. + +***Below diagram provides details about various TAM objects needed to support IFA and their correlation*** + +![IFA TAM objects correlation](ifa-tam-correlation.JPG) + +## 3.7 CLI + +### 3.7.1 Data Models + +The user facing data model is based on OpenConfig compatible TAM yang model. The backend data model (SONiC YANG) will use the formats in CONFIG_DB & STATE_DB. See above sections. + +### 3.7.2 Configuration Commands + +#### 3.7.2.1 Activating and Deactivating IFA + +The command syntax for activating/de-activating the IFA feature on the switch is as follows: + +``` +sonic (config-tam-ifa)# [no] enable +``` +As noted earlier, the IFA activated switches act as intermediate nodes for all IFA-tagged flows transiting the switch. + +Deactivating IFA will purge all IFA configuration from the switch. + +#### 3.7.2.2 Setting up flows for monitoring with IFA + +A IFA monitoring session associated a previously defined flow-group, with IFA as described below. + +- The IFA session must have a unique name for referencing. +- The flow-group must be previously created with the `flow-group` command (under `config-tam` hierarchy). +- The switch can be setup to act as an ingress node or as an egress node for this session. +- On ingress nodes, the sampling-rate can be set, by referencing a previously created sampler, created with the `sampler` command (under `config-tam` hierarchy). On ingress nodes, the flow-group must be associated with an interface. +- On the egress nodes, a collector must be associated with the flow, where the extracted metadata will be sent. The collector must be previously created with the `collector` command (under `config-tam` hierarchy).. + +When a sesssion that is previously created is removed (with the `no` command), the associated flows are no longer processed for IFA as an ingress node or as an egress-node by the switch. + +The following attribtes are supported for IFA sessions. + +| **Attribute** | **Description** | +|--------------------------|-------------------------------------| +| `name` | A string that uniquely identifies the IFA session | +| `flowgroup` | Specifies the name of *flow-group* | +| `collector` | Specifies the name of the *collector*, valid and mandatory for egress nodes | +| `sample-rate` | Specifies the name of the *sampler*, valid and mandatory for ingress nodes | +| `node-type` | Specifies the IFA role of the switch for the flow. Valid values are `"ingress"` and `"egresss"`| + +The command syntax for creating /removing the sessions are as follows: + +``` +sonic(config-tam-ifa)# session flowgroup [collector ] [sample-rate ] node-type {ingress | egress } + +sonic (config-tam-ifa)# no session +``` + +### 3.7.3 Show Commands + +#### 3.7.3.1 Listing the IFA attributes + +The following command lists the switch-wide attributes that are in use. + +``` +sonic # show tam ifa +``` +Sample usage shown below. + +``` +sonic # show tam ifa + +Status : Active +Version : 2.0 +Switch ID : 2020 +Enterprise ID : 2345 + +``` + +#### 3.7.3.1 Listing the IFA sessions + +The following command lists the details for all ifa sessions or for a specific session. Note that only explicitly configured tuples in the associated flow-group are displayed. + +``` +sonic # show tam ifa sessions [] +``` + +Sample usage shown below. + +``` +sonic # show tam ifa sessions + +Name Flow Group Collector Sampler Node Type +----------- ---------------- -------------- ------------- ---------- +http_236 tcp_port_236 - aggresive Ingress +http_239 tcp_port_239 - - Intermediate +http_241 tcp_port_241 IFA_Col_i19 - Egress + +sonic # show tam ifa sessions http_236 + +Session : http_236 (Ingress) +Flow Group Name : tcp_port_236 + Id : 4025 + Priority : 100 + SRC IP : 13.92.96.32 + DST IP : 7.72.235.82 + DST L4 Port : 236 + Ingress Intf : Ethernet20 +Collector : None +Sampler : aggresive +Packet Count : 7656 + +``` +### 3.7.4 Sample Workflow + +This section provides a sample IFA workflow using CLI, for monitoring the traffic as described below. + +> Need to monitor all flows to the webserver running at 20.20.1.1. A IFA collector for analysing the metadata is running at 20.20.20.4:9090 (UDP). Not every packet needs monitored, but one in 1000 is acceptable. The flows are ingressing onto switch1 on port 'Ethernet44' + +Note that there would be other system wide configuration that is not covered in the sample workflow below, but would be required for the packet forwarding/routing. + +#### Ingress Node configuration + +``` +; setup switch-wide configuration + +sonic (config-tam)# switch-id 1234 +sonic (config-tam)# enterprise-id 4434 + +; setup the sample-rate + +sonic (config-tam)# sampler websamp rate 1000 + +; create the flowgroup + +sonic (config-tam)# flow-group websrvflows dst-ip 20.20.1.1 dst-l4-port 80 protocol 6 + +; associate the ingress interface to the flowgroup + +sonic (config) # interface Ethernet 44 +sonic (config-if-Ethernet44)# flow-group websrvflows + +; Enable IFA on the switch + +sonic (config-tam-ifa)# enable + +; Create the IFA monitoring session + +sonic(config-tam-ifa)# session webflowmonitor flowgroup websrvflows sample-rate websamp node-type ingress + +``` + +#### Intermediate/Transit Node configuration + +``` +; setup switch-wide configuration + +sonic (config-tam)# switch-id 1235 +sonic (config-tam)# enterprise-id 4434 + +; Enable IFA on the switch + +sonic (config-tam-ifa)# enable + +``` + +#### Egress Node configuration + +``` +; setup switch-wide configuration + +sonic (config-tam)# switch-id 1236 +sonic (config-tam)# enterprise-id 4434 + +; setup the collector + +sonic (config-tam)# collector ifacol1 type ipv4 ip 20.20.20.4 port 9090 protocol UDP + +; create the flowgroup + +sonic (config-tam)# flow-group websrvflows dst-ip 20.20.1.1 dst-l4-port 80 protocol 6 + +; Enable IFA on the switch + +sonic (config-tam-ifa)# enable + +; Create the IFA monitoring session + +sonic(config-tam-ifa)# session webflowmonitor flowgroup websrvflows collector ifacol1 node-type egress + +``` + +### 3.7.5 Debug Commands +N/A + +### 3.7.6 REST API Support + +The following REST API are supported - + +#### Yang-based REST API based on the openconfig-tam module defintions + +##### Obtaining the all of the sessions + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/ifa-sessions +* Response format +```json +{ + "openconfig-tam:ifa-sessions": { + "ifa-session": [ + { + "name": "string", + "config": { + "name": "string", + "flowgroup": "string", + "collector": "string", + "sample-rate": "string", + "node-type": "INGRESS" + }, + "state": { + "name": "string", + "flowgroup": "string", + "collector": "string", + "sample-rate": "string", + "node-type": "INGRESS" + } + } + ] + } +} +``` + +##### Obtaining a specific session + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/ifa-sessions/ifa-session={name} +* Response format +```json +{ + "openconfig-tam:ifa-session": [ + { + "name": "string", + "config": { + "name": "string", + "flowgroup": "string", + "collector": "string", + "sample-rate": "string", + "node-type": "INGRESS" + }, + "state": { + "name": "string", + "flowgroup": "string", + "collector": "string", + "sample-rate": "string", + "node-type": "INGRESS" + } + } + ] +} +``` + +##### Creating a session + +* Method : PATCH +* URI : /restconf/data/openconfig-tam:tam/ifa-sessions +* Data format +```json +{ + "openconfig-tam:ifa-sessions": { + "ifa-session": [ + { + "name": "string", + "config": { + "name": "string", + "flowgroup": "string", + "collector": "string", + "sample-rate": "string", + "node-type": "INGRESS" + } + } + ] + } +} +``` + +##### Deleting a session + +* Method : DELETE +* URI : /restconf/data/openconfig-tam:tam/ifa-sessions/ifa-session={name} + + # 4 Flow Diagrams + +## 4.1 Config call flow + +All the configuration is stored in the CONFIG_DB via the management framework. + +# 5 Error Handling + +## IFA/TAM Application + +- Any configuration errors/dependency failures are logged into Syslog and are ignored. + +## CLI + +- CLI configuration sanity will be enforced by the CLI handler & CVL and any invalid configuration is rejected. An error is displayed to the user notifying the reason for rejection of the configuration. + +## IFAOrch + +- Any error occurring in the orchestration agent is logged appropriately via SWSS logging. +- Errors or failures of SAI APIs will be logged by IFAOrch. +- On failure of a SAI TAM API in the config sequence of section 3.6, the previously configured steps will be rolled back i.e previously created intermediate TAM objects for IFA etc will be destroyed. + +# 6 Serviceability and Debug + +N/A + +# 7 Warm Boot Support + +No special handling is done for the warm boot case. The TAM configuration is restored from the Config DB and TAM functionality will continue to work as it is through a warm boot. + +# 8 Scalability + +N/A + +# 9 Unit Test + +## CLI + +TBD + +# Broadcom Internal Information : To be removed before publishing externally. + +## Key notes + +* With the addition of IFA 2.0 support, IFA 1.1 support is removed in Broadcom-SONIC. + +* IFA 2.0 protocol is not documented in this document. It may be requested via docSafe for the document number 56870-AN500-D1. + +* IFA 2.0 feature is very similar to the IFA 1.1 feature supported in 2.x and 3.0 release of Broadcom SONiC. They are almost identical from a provisioning perspetive. However, the protocol is very different. + +* IFA 2.0 feature is an *advanced* feature that is not available in all the Broadcom SONiC packages. + +* IFA 2.0 feature requires premium Cancun (for TD3-X7) and the firmware, both of which are packaged into the SONIC packages. + +## Specific limitations + +IFA 2.0 feature in SONiC inherits the limitations of the underlying firmware and the hardware. These are listed below. + +1. Only a single collector is supported +2. IFA enabled flows that are tunneled are not supported +3. IFA enabled flows must be L3 & IPv4 flows +4. IFA is supported on TD3-X7 platforms only. +5. The flow path may not exceed 32 nodes + +## Additional constraints + +1. The sampled packet contains about 128bytes of the original packet, and will be sent to the collector along with the aggregated metadata from individual nodes in the flow path. diff --git a/devops/tam/tam-tail-stamping-hld.md b/devops/tam/tam-tail-stamping-hld.md new file mode 100644 index 000000000000..208e81ceae81 --- /dev/null +++ b/devops/tam/tam-tail-stamping-hld.md @@ -0,0 +1,663 @@ +# Tailstamping + +## Highlevel Design Document + +### Rev 0.2 + +# Table of Contents + +- [Tailstamping](#tailstamping) + * [Highlevel Design Document](#highlevel-design-document) + + [Rev 0.1](#rev-01) +- [Table of Contents](#table-of-contents) + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definition-abbreviation) + + [Table 1: Abbreviations](#table-1--abbreviations) +- [1 Feature Overview](#1-feature-overview) + * [1.1 Requirements](#11-requirements) + + [1.1.1 Functional Requirements](#111-functional-requirements) + + [1.1.2 Configuration and Management Requirements](#112-configuration-and-management-requirements) + + [1.1.3 Scalability Requirements](#113-scalability-requirements) + * [1.2 Design Overview](#12-design-overview) + + [1.2.1 Basic Approach](#121-basic-approach) + + [1.2.2 Container](#122-container) + + [1.2.3 SAI Overview](#123-sai-overview) +- [2 Functionality](#2-functionality) + * [2.1 Target Deployment Use Cases](#21-target-deployment-use-cases) + * [2.2 Functional Description](#22-functional-description) +- [3 Design](#3-design) + * [3.1 Overview](#31-overview) + * [3.2 DB Changes](#32-db-changes) + + [3.2.1 CONFIG DB](#321-config-db) + + [3.2.2 APP DB](#322-app-db) + + [3.2.3 STATE DB](#323-state-db) + + [3.2.4 ASIC DB](#324-asic-db) + + [3.2.5 COUNTER DB](#325-counter-db) + * [3.3 Daemons](#33-daemons) + * [3.4 Switch State Service Design](#34-switch-state-service-design) + + [3.4.1 Orchestration Agent](#341-orchestration-agent) + + [3.4.2 Other Process](#342-other-process) + * [3.5 SyncD](#35-syncd) + * [3.6 SAI](#36-sai) + * [3.7 CLI](#37-cli) + + [3.7.1 Data Models](#371-data-models) + + [3.7.2 Configuration Commands](#372-configuration-commands) + - [3.7.2.1 Activating and Deactivating Tailstamping](#3721-activating-and-deactivating-tailstamping) + - [3.7.2.2 Setting up flows for monitoring with Tailstamping](#3722-setting-up-flows-for-monitoring-with-tailstamping) + + [3.7.3 Show Commands](#373-show-commands) + - [3.7.3.1 Listing the TS attributes](#3731-listing-the-ts-attributes) + - [3.7.3.1 Listing the Tailstamping sessions](#3731-listing-the-tailstamping-sessions) + + [3.7.4 Sample Workflow](#374-sample-workflow) + + [3.7.5 Debug Commands](#375-debug-commands) + + [3.7.6 REST API Support](#376-rest-api-support) +- [4 Flow Diagrams](#4-flow-diagrams) + * [4.1 Config call flow](#41-config-call-flow) +- [5 Error Handling](#5-error-handling) + * [CLI](#cli) + * [TSOrch](#tsorch) +- [6 Serviceability and Debug](#6-serviceability-and-debug) +- [7 Warm Boot Support](#7-warm-boot-support) +- [8 Scalability](#8-scalability) +- [9 Unit Test](#9-unit-test) + * [CLI](#cli-1) +- [Broadcom Internal Information : To be removed before publishing externally.](#broadcom-internal-information---to-be-removed-before-publishing-externally) + * [Key notes](#key-notes) + * [Specific Limitations](#specific-limitations) + +## List of Tables + +[Table 1: Abbreviations](#table-1-abbreviations) + +## Revision + +| Rev | Date | Author | Change Description | +|---|-----------|------------------|-----------------------------------| +| 0.1 | 10/29/2019 | Naveen Kumar Aketi | Initial version | +| 0.2 | 08/18/2020 | Bandaru Viswanath | Major update to accomodate enhancements to use new TAM infrastructure, DB schmas and UI | + +## About This Manual + +This document provides general information about the Tailstamping feature implementation in SONiC. + +## Scope + +The term Inband Telemetry refers to the set of technologies of the Switch, that embed useful metadata as part of packets that are transiting the Switch. SAI community defined a Telemetry and Monitoring specification that defines a unified interface - [TAM](https://github.com/opencomputeproject/SAI/tree/master/doc/TAM) - for provisioning and monitoring these technologies. Tailstamping feature is one such Inband Telemetry technologies. + +The design as specified in this document uses the common [SONiC TAM](tam.hld) infrastructure. + +This document describes the high level design of Tailstamping feature in SONiC. + +## Definition/Abbreviation + +### Table 1: Abbreviations + +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| SAI | Switch abstraction interface | +| TAM | Telemetry and monitoring | +| TS | Tailstamping | +| CRC | Cyclic Redundancy Check | +| MAC | Media Access Control | + +# 1 Feature Overview + +Tailstamping feature attaches arrival and departure timestamps along with a switch identifier to a frame at one or more switches along a flow path across the network for a given set of flows matching the user provided criterion. Time-stamps can be gathered and analyzed by a designated collector. A network monitoring application can analyze the time-stamps and measure the latency of a flow across the network, trace packets across a path, detect hotspots points and validate arrival sequence. + +The Tailstamping feature in SONiC allows the user to setup flow monitoring sessions for specific flows. + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +1.0 Tailstamping feature allows user to enable time stamping on a per flow and per port basis. + +1.1 Ingress and Egress timestamps are added at the end of frames. + +1.2 Ingress and Egress timestamps are in 1588 Time-of-Day format (UTC time). + +1.3 The presence of any tailstamping headers in the packet must not affect forwarding or processing of the packet in downstream devices, including those that do not support the timestamping feature. + +1.4 Timestamping packet format is not covered by any standards and is Broadcom proprietary. + +2.0 Tailstamping feature needs configuration as mentioned below. + +2.1 TAM swith identifier to uniquely identify a device in network and to insert as part of the metadata. + +2.2 TAM flowgroup configuration to identify a flow that should be tagged with tailstamping metadata. + +3.0 UI commands available to configure Tailstamping configuration. + +3.1 UI commands available to show Tailstamping configuration, status and statistics. + +3.2 UI commands available to clear Tailstamping configuration. + +4.0 The maximum number of Tailstamping sessions are platform dependent. + +4.1 The maximum packet length supported by Tailstamping is platform dependent. + +4.2 Frames received by CPU are ingress time-stamped if the frame matches the user provided flow criterion. + +5.0 Tailstamping feature is by default not active on the switch, and requires an explicit user intervention to activate it in the configuration. + +5.1 If the feature can't be supported/activated on the switch, any attempt to activate the feature returns appropriate failure. + +6.0 The Tailstamping feature is supported on physical and LAG ports. + +6.1 The Tailstamping feature supports dynamic-port-breakout and aliasing features. + +7.0 The Tailstamping feature will be a part of the TAM container along with other TAM features. + +### 1.1.2 Configuration and Management Requirements + +The TAM Tailstamping feature supports the new management framework and KLISH CLI for configuration. The following configuration and status information is available. + +- To activate / de-activate Tailstamping feature +- To create/clear appropriate Tailstamping configuration on a per-flow group basis. +- To display current status and statistics for the Tailstamping on a per flow-group basis. + +### 1.1.3 Scalability Requirements + +- Number of Tailstamping sessions that can be supported is proportional to the availability of resources in hardware such as ACLs. No specific constraints are imposed. + +## 1.2 Design Overview + +### 1.2.1 Basic Approach + +The Tailstamping feature is newly developed. + +### 1.2.2 Container + +A container called 'tam' exists as a container for all TAM applications, including the TS application. + +### 1.2.3 SAI Overview + +The SAI TAM spec specifies the TAM APIs to be used to configure the TAM functionality. Please refer to SAI-Proposal-TAM2.0-v2.0.docx in [https://github.com/opencomputeproject/SAI/tree/master/doc/TAM](https://github.com/opencomputeproject/SAI/tree/master/doc/TAM) for more details. + +# 2 Functionality + +## 2.1 Target Deployment Use Cases + +Tailstamping is used to proactively monitor the network for faults and performance bottlenecks, it aids in detecting data plane faults and isolating them to a given device or location and assists in planning network capacity and projecting future load. Some of the key usecases are - + +- SLA conformance that require end-to-end path latency for targeted flows +- Idenitfying switches and flows that are undergoing congestion +- Identifying network bandwidth usage patterns +- Packet path and path change analysis + +The mechanism to direct the tail-stamped packets to a collector is not part of this feature. User should use appropriate means to direct the tail-timestamped packets to a collector. For example, user can configure a ACL matching the tail-timestamped packets and configure the ACL action as mirror/redirect. + +## 2.2 Functional Description + +The Tailstamping feature in SONiC allows the user to setup flow monitoring sessions for specific flows. The switch adds the tailstamping metadata for the matching flows, towards the end of the ethernet packet, right infront of the CRC. + + +***Example topology illustrating the packet paths and tailstamping metadata*** + +![Tailstamping topology](sample-ts-topology.png) + +# 3 Design + +## 3.1 Overview + +***Tailstamping Architecture*** + +![architecture](ts_arch.JPG) + +The above diagram illustrates the architecture of the Tailstamping feature within SONiC. + +Below is the call flow sequence specified in above architecture diagram : + +1 TS and Flowgroup configuration from CLI is saved to CONFIG DB. + +2 TS Manager reads configuration from CONFIG DB, processes and validates the configuration. + +3 ACL Manager reads ACL configuration from CONFIG DB, processes and validates ACL configuration. + +4 TS Manager updates valid TS configuration to APPL DB. + +5 ACL Manager updates valid ACL configuration to APPL DB. + +6 TS ORCH reads TS configuration from APPL DB and creates/deletes TAM INT TS objects. + +7 ACL ORCH reads ACL configuration from APPL DB. + +8 ACL ORCH notifies TS ORCH about ACL rule creation/deletion. + +9 TS ORCH checks for ACL table and ACL rule specified in TS configuration. + +10 TS ORCH attaches/detaches the TAM INT (IFA1_TS) object to/from ACL ENTRY. + +11 ACL ORCH will make SAI API calls to update ASIC DB. + +12,13,14 SYNCD reads TS and ACL information from ASIC DB and configures ASIC accordingly. + +A CLI reads TS ACL counters from COUNTER DB and displays it as output of show commands. + +## 3.2 DB Changes + +### 3.2.1 CONFIG DB + +TAM\_TAILSTAMPING\_SESSIONS\_TABLE + + ;Defines TAM TS configuration in CONFIG_DB + + key = name ; name is collector name and should be unique. + flowgroup = 1*255VCHAR ; Flow group reference + + Example: + > keys *TAM_TAILSTAMPING_SESSIONS* + + 1) "TAM_TAILSTAMPING_SESSIONS_TABLE|sla1" + + > hgetall "TAM_TAILSTAMPING_SESSIONS_TABLE|sla1" + + 1) "flowgroup" + 2) "slaflows" + +### 3.2.2 APPL DB + +TAM\_TS\_TABLE + +;Operational TS Global Status in APPL_DB + + key = global ; Only one instance and + ; has a fixed key ”global" + op-switch-id = 1 * 5DIGIT ; Currently used switch-id + + Example: + > keys *TAM_TS_TABLE* + 1) "TAM_TS_TABLE|global" + + > HGETALL "TAM_TS_TABLE|global" + 1)”op-switch-id” + 2)54325 + + +TAM\_TS\_FLOW\_TABLE + + ;Contains TS flow configuration + + key = name ; name is flow name and should be unique. + acl-table-name = table-name ; Parameter to map to acl table to the flow. + acl-rule-name = rule-name ; Parameter to map to acl rule to the flow. + + Example: + > KEYS *TAM\_TS\_FLOW_TABLE* + 1) "TAM_TS_FLOW_TABLE:F1" + + > HGETALL TAM_TS_FLOW_TABLE:F1 + 1) "acl-table-name" + 2) "T1" + 3) "acl-rule-name" + 4) "R1" + +### 3.2.3 STATE DB + +N/A + +### 3.2.4 ASIC DB + +The ASIC DB is updated by SAI REDIS upon invocation of SAI REDIS APIs by TSOrch. + +### 3.2.5 COUNTER DB + +N/A + +## 3.3 Daemons + +TS manager daemon runs as part of TAM docker. TS manager processes TS configuration from CONFIG DB, validates for consistency and completeness of TS configuration and updates valid configuration to APPL DB. + +## 3.4 Switch State Service Design + +### 3.4.1 Orchestration Agent + +A new orchestration agent class, TSOrch is added to convert the incoming config to ASIC configuration. TSOrch subscribes to the TS tables of APPL DB and converts the configuration to the SAI TAM API call sequence described in section 3.6. + +TSOrch maintains data pertaining to all the currently configured TS entities and the associated TAM object bindings. TAM object bindings are re-used wherever possible. + +TSOrch checks for support for the Tailstamping feature in the silicon using SAI capability API and sets the field `tam_int_ifa_ts_supported` to `True` in the `SWITCH_TABLE` of APPL_DB under the key `switch`. + +### 3.4.2 Other Process + +N/A + +## 3.5 SyncD + +N/A + +## 3.6 SAI + +The SAI TAM API spec defines all TAM APIs supported in SAI. Please refer to SAI-Proposal-TAM2.0-v2.0.docx in [https://github.com/opencomputeproject/SAI/tree/master/doc/TAM](https://github.com/opencomputeproject/SAI/tree/master/doc/TAM) for more details. + +Tailstamping feature is accomplished by attaching a SAI TAM INT object to SAI ACL entry. Below diagram provides details about how the SAI ACL object and SAI TAM objects are stitched. + + +***Below diagram provides details about various TAM objects needed to support TS and their correlation*** + +![TS TAM objects correlation](ts-tam-correlation.png) + +## 3.7 CLI + +### 3.7.1 Data Models + +The user facing data model is based on OpenConfig TAM yang model (TBD). The backend data model (SONiC YANG) will use the formats in CONFIG_DB & APPL_DB. See above sections. + +### 3.7.2 Configuration Commands + +#### 3.7.2.1 Activating and Deactivating Tailstamping + +The command syntax for activating/de-activating the Tailstamping feature on the switch is as follows: + +``` +sonic (config-tam-ts)# [no] enable +``` + + +Deactivating TS will purge all associated configuration from the switch. + +#### 3.7.2.2 Setting up flows for monitoring with Tailstamping + +A Tailstamping monitoring session associated a previously defined flow-group, with TS as described below. + +- The monitoring session must have a unique name for referencing. +- The flow-group must be previously created with the `flow-group` command (under `config-tam` hierarchy). + +When a sesssion that is previously created is removed (with the `no` command), the associated flows are no longer processed for TS by the switch. + +The following attribtes are supported for Tailstamping sessions. + +| **Attribute** | **Description** | +|--------------------------|-------------------------------------| +| `name` | A string that uniquely identifies the TS session | +| `flowgroup` | Specifies the name of *flow-group* | + +The command syntax for creating /removing the sessions are as follows: + +``` +sonic(config-tam-ts)# session flowgroup + +sonic (config-tam-ts)# no session +``` + +### 3.7.3 Show Commands + +#### 3.7.3.1 Listing the TS attributes + +The following command lists the switch-wide attributes that are in use. + +``` +sonic # show tam tail-stamping +``` +Sample usage shown below. + +``` +sonic # show tam tail-stamping + +Status : Active +Switch ID : 2020 + +``` + +#### 3.7.3.1 Listing the Tailstamping sessions + +The following command lists the details for all tail-stamping sessions or for a specific session. Note that only explicitly configured tuples in the associated flow-group are displayed. + +``` +sonic # show tam tail-stamping sessions [] +``` + +Sample usage shown below. + +``` +sonic # show tam tail-stamping sessions + +Name Flow Group +----------- ---------------- +http_236 tcp_port_236 +http_239 tcp_port_239 +http_241 tcp_port_241 + +sonic # show tam tail-stamping sessions http_236 + +Session : http_236 +Flow Group Name : tcp_port_236 + Id : 4025 + Priority : 100 + SRC IP : 13.92.96.32 + DST IP : 7.72.235.82 + DST L4 Port : 236 +Packet Count : 7656 + +``` + +### 3.7.4 Sample Workflow + +This section provides a sample Tailstamping workflow using CLI, for monitoring the traffic as described below. + +> Gather flow metadata for network probe packets sent from 10.10.1.1:8080 to 20.4.5.2:7070 (tcp) + +Note that there would be other system wide configuration that is not covered in the sample workflow below, but would be required for the packet forwarding/routing. + +``` +; setup switch-wide configuration + +sonic (config-tam)# switch-id 1234 + +; create the flowgroup + +sonic (config-tam)# flow-group probeflow src-ip 10.10.1.1 src-l4-port 8080 dst-ip 20.4.5.2 dst-l4-port 7070 protocol 6 + +; Enable Tailstamping on the switch + +sonic (config-tam-ts)# enable + +; Create the Tailstamping monitoring session + +sonic(config-tam-ts)# session probemonitor flowgroup probeflow + +``` + + +### 3.7.5 Debug Commands +N/A + +### 3.7.6 REST API Support + +The following REST API are supported - + +#### Yang-based REST API based on the openconfig-tam module defintions + +##### Obtaining the all of the sessions + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/tailstamping-sessions +* Response format +```json +{ + "openconfig-tam:tailstamping-sessions": { + "tailstamping-session": [ + { + "name": "string", + "config": { + "name": "string", + "flowgroup": "string" + }, + "state": { + "name": "string", + "flowgroup": "string" + } + } + ] + } +} +``` + +##### Obtaining a specific session + +* Method : GET +* URI : /restconf/data/openconfig-tam:tam/tailstamping-sessions/tailstamping-session={name} +* Response format +```json +{ + "openconfig-tam:tailstamping-session": [ + { + "name": "string", + "config": { + "name": "string", + "flowgroup": "string" + }, + "state": { + "name": "string", + "flowgroup": "string" + } + } + ] +} +``` + +##### Creating a session + +* Method : PATCH +* URI : /restconf/data/openconfig-tam:tam/tailstamping-sessions +* Data format +```json +{ + "openconfig-tam:tailstamping-sessions": { + "tailstamping-session": [ + { + "name": "string", + "config": { + "name": "string", + "flowgroup": "string" + } + } + ] + } +} + +``` + +##### Deleting a session + +* Method : DELETE +* URI : /restconf/data/openconfig-tam:tam/tailstamping-sessions/tailstamping-session={name} + + # 4 Flow Diagrams + +## 4.1 Config call flow + +All the configuration is stored in the CONFIG_DB via the management framework. + +# 5 Error Handling + +## Tailstamping/TAM Application + +- Any configuration errors/dependency failures are logged into Syslog and are ignored. + +## CLI + +* CLI configuration sanity will be enforced by the CLI handler & CVL and any invalid configuration is rejected. An error is displayed to the user notifying the reason for rejection of the configuration. + +## TSOrch + +- Any error occurring in the orchestration agent is logged appropriately via SWSS logging. +- Errors or failures of SAI APIs will be logged by TSOrch. +- On failure of a SAI TAM API in the config sequence of section 3.6, the previously configured steps will be rolled back i.e previously created intermediate TAM objects for TS etc will be destroyed. + +# 6 Serviceability and Debug + +N/A + +# 7 Warm Boot Support + +No special handling is done for the warm boot case. The TAM configuration is restored from the Config DB and TAM functionality will continue to work as it is through a warm boot. + +# 8 Scalability + +N/A + +# 9 Unit Test + +## CLI + +* Verify CLI command to configure TAM INT IFA IFA-TS feature enable. +* Verify CLI command to configure TAM INT IFA IFA-TS feature disable. +* Verify ACL configuration with packet action type as int_insert for IPv4 type ACL. +* Verify ACL configuration with packet action type as int_insert for IPv6 type ACL. +* Verify CLI command to configure IFA-TS flow with ACL table name and ACL rule name. +* Verify CLI clear command to clear TAM INT IFA IFA-TS flow. +* Verify CLI show command to show TAM INT IFA IFA-TS status. +* Verify CLI show command to show TAM INT IFA IFA-TS statistics for all flows. +* Verify CLI show command to show TAM INT IFA IFA-TS statistics for a specific flows. +* Verify CLI show command to show TAM INT IFA IFA-TS flow. + +## REST + +TBD + +## Tailstamping Application + +* Verify if IFA-TS configuration from CONFIG DB is received by IFA-TS manager. +* Verify if TSOrch is able to create IFA-TS table entries in APPL DB successfully. +* Verify if TSOrch is able to delete IFA-TS table entries in APPL DB successfully. + +## TSOrch + +* Verify if IFA-TS configuration from APPL DB is received by TSOrch. +* Verify if TSOrch is able to create TAM objects for IFA-TS configuration via SAI TAM APIs successfully. +* Verify if TSOrch is able to delete existing IFA-TS configuration via SAI TAM APIs successfully. +* Verify if TSOrch is able to use existing TAM objects for IFA-TS config. +* Verify if TSOrch rolls back config in a clean way if there is a SAI API failure. +* Verify if TSOrch is able to set TAM INT attribute for IFA-TS ACL. +* Verify if TSOrch is able to reset TAM INT attribute for IFA-TS ACL. + +## Functional Tests + +* Verify if IPv4 traffic matching IFA-TS flow is time stamped. +* Verify if IPv6 traffic matching IFA-TS flow is time stamped. +* Verify if IFA-TS headers are inserted in correct order. +* Verify that there is no crash encountered at any of the layers with an invalid IFA-TS configuration. +* Verify that an invalid configuration is rejected gracefully at appropriate layers. +* Verify that IFA-TS configuration is restored after warmboot. +* Verify that switched traffic matching IFA-TS flow is time stamped. +* Verify that routing traffic matching IFA-TS flow is time stamped. + +## Negative Tests + +* Verify if CLI throws error when a user tries to create a duplicate IFA-TS flow. +* Verify if CLI returns error if CLI is unable to write the IFA-TS config to config DB. +* Verify if CLI returns entry not found when a clear command is issued on non-existent flow. +* Verify if IFA-TS manager logs an error on receipt of an incorrect IFA-TS table entries from CONFIG DB. +* Verify if TSOrch logs an error on receipt of an incorrect IFA-TS table entries from APPL DB. +* Verify if TSOrch logs an error if it is unable to read IFA-TS table data from APPL DB. +* Verify if TSOrch logs all errors encountered during processing of the incoming IFA-TS config request. +* Verify if TSOrch logs any errors arising out of SAI API failure. +* Verify if TSOrch logs an error when no further IFA-TS configuration can be configured to hardware. +* Verify if feature not supported is returned when IFA-TS feature is not supported by underlying silicon. + +# Broadcom Internal Information : To be removed before publishing externally. + +## Key notes + +1. Tailstamping feature in the dataplane is completely supported in the silicon. + +2. Switches which have Tailstamping feature can be used as intermediate nodes in an IFA deployments, where the ingress and egress nodes run IFA. The IFA firmware on the egress nodes is capable of extracting the TS metadata from the end of the packet and present a unified metadata stack to the collector. + +3. Tailstamping feature is an *advanced* feature that is not available in all the Broadcom SONiC packages. + +4. Tailstamping metadata format varies slightly based on silicon, while being compatible with each other. The specific formats are documented in appropriate silicon documentation. + +## Specific Limitations + +Tailstamping feature inherits the usage constraints from the underlying hardware. These are listed below. + +1. Only TCP and UDP traffic can be timestamped. +2. Timestamp is not part of the L3 packet. Any checks that assume the L3+ packet length field represents the total frame length will not be accurate. +3. Systems that need to subject packets to such checks must disable timestamping for the corresponding system, port or flow. +4. Header length fields or checksum fields(e.g UDP checksum) will not be updated upon insertion of the timestamp. +5. IEEE 802.3 frames(e.g SNAP LLC) are not supported. +6. No switches across the timestamping path should do pad stripping or otherwise adjust frame content based on the IP header payload/total length fields for Ethernet II frames. +7. Packets/Features using the HiGig2 extension header cannot coexist with packet timestamping. diff --git a/devops/tam/ts-tam-correlation.png b/devops/tam/ts-tam-correlation.png new file mode 100644 index 000000000000..ef064d4005bc Binary files /dev/null and b/devops/tam/ts-tam-correlation.png differ diff --git a/devops/tam/ts_arch.JPG b/devops/tam/ts_arch.JPG new file mode 100644 index 000000000000..690d91000c99 Binary files /dev/null and b/devops/tam/ts_arch.JPG differ