Skip to content

Commit

Permalink
Merge pull request sonic-net#270 from BRCM-SONIC/vxlan_stats
Browse files Browse the repository at this point in the history 
VXLAN HLD Update for Tunnel Statistics support.
  • Loading branch information
@adyeung
adyeung authored Sep 21, 2021
2 parents ae0168d + 8e378ce commit 3c945ad
Showing 1 changed file with 189 additions and 18 deletions.
207 changes: 189 additions & 18 deletions L3/EVPN/EVPN VXLAN HLD.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# EVPN VXLAN HLD

#### Rev 1.2
#### Rev 1.3

# Table of Contents

Expand All @@ -22,10 +22,11 @@
- [4 Feature Design](#4-feature-design)
- [4.1 Overview](#41-design-overview)
- [4.2 DB Changes](#42-db-changes)
- [CONFIG DB](#config_db-changes)
- [APP DB](#app_db-changes)
- [STATE DB](#state_db-changes)
- [COUNTER_DB](#counter_db-changes)
- [CONFIG DB](#421-config_db-changes)
- [APP DB](#422-app_db-changes)
- [STATE DB](#423-state_db-changes)
- [COUNTER_DB](#424-counter_db-changes)
- [FLEX_COUNTER_DB](#425-flex_counter_db-changes)
- [4.3 Modules Design and Flows](#43-modules-design-and-flows)
- [4.3.1 Tunnel Creation](#431-tunnel-auto-discovery-and-creation)
- [4.3.2 Tunnel Deletion](#432-tunnel-deletion)
Expand All @@ -44,9 +45,12 @@
- [5.1 Click CLI](#51-click-based-cli)
- [5.1.1 Configuration Commands](#511-configuration-commands)
- [5.1.2 Show Commands](#512-show-commands)
- [5.1.3 Clear Commands](#513-clear-commands)
- [5.2 SONiC CLI](#52-sonic-cli)
- [5.2.1 Configuration Commands](#521-configuration-commands)
- [5.2.2 Show Commands](#522-show-commands)
- [5.2.3 Validations](#523-validations)
- [5.2.4 Clear Commands](#524-clear-commands)
- [5.3 Openconfig URI](#53-openconfig-uri)
- [6 Serviceability and Debug](#6-serviceability-and-debug)
- [7 Warm reboot Support](#7-warm-reboot-support)
Expand All @@ -55,6 +59,7 @@
- [8.2 Underlays supported](#82-underlays-supported)
- [8.3 VxLAN QoS Mode](#83-vxlan-qos-mode)
- [8.4 Unit Test Plans](#84-unit-test-plans)
- [9 References ](#9-references)

# List of Tables

Expand All @@ -73,7 +78,7 @@
| 0.8 | 24/04/2020 | Nikhil Kelapure | Updated Warm-reboot section |
| 1.1 | 04/14/2020 | Mohanarajan Selvaraj | VXLAN QoS Changes |
| 1.2 | 04/24/2020 | Rajesh Sankaran | VLAN interface underlay support |
| 1.3 | 05/14/2021 | Rajesh Sankaran | VXLAN stats support |
| 1.3 | 09/02/2021 | Rajesh Sankaran | VXLAN stats support |

# Definition/Abbreviation

Expand Down Expand Up @@ -182,11 +187,10 @@ in SONiC.
3. Support configuration of a global VLAN-VNI map.
4. Support configuration of L3VNI association with VRF.
5. Support display of packet tx and rx stats.
6. Support display of octet tx and rx stats.
7. Support display of rx and tx bit rates.
8. Support per tunnel and all tunnel stats display.
9. Support clearing of stats per tunnel and all tunnels.
10.Support Click, KLISH, REST interfaces for stats.
6. Support display of rx and tx bit rates.
7. Support per tunnel and all tunnel stats display.
8. Support clearing of stats per tunnel and all tunnels.
9.Support Click, KLISH, REST interfaces for stats.



Expand Down Expand Up @@ -416,6 +420,26 @@ vni = 1*8DIGIT ; VNI associated with VRF
```

**FLEX_COUNTER_TABLE**

Producer: config manager

Consumer: Flexcounterorch

Description:

- Contains the polling interval in msec for VXLAN tunnels.

Schema:

```
; Existing table
key = FLEX_COUNTER_TABLE|TUNNEL ;
; field = value
POLL_INTERVAL = number ;uint32
FLEX_COUNTER_STATUS = enable/disable ; enable or disable tunnel counters
```

### 4.2.2 APP_DB Changes

**EVPN_NVO_TABLE**
Expand Down Expand Up @@ -573,6 +597,8 @@ operstatus = "oper_up"/"oper_down"

### 4.2.4 COUNTER_DB changes

**COUNTERS TABLE**

Producer: syncd

Consumer: show scripts
Expand All @@ -594,7 +620,80 @@ SAI_TUNNEL_STAT_OUT_OCTETS = number ;uint64
SAI_TUNNEL_STAT_OUT_PACKETS = number ;uint32
```

**COUNTERS_BACKUP TABLE**

This table is a backup of the counters table with the backup taken everytime the
counters are cleared. In addition to the same fields as the counters table it
also holds the LAST_CLEAR_TIMESTAMP field which captures the last time the counters
were cleared.

**COUNTERS_TUNNEL_NAME_MAP TABLE**

Producer: VxlanOrch

Consumer: show scripts

Description:

- Mapping between tunnel sip,dip to tunnel oid.

Schema:

```
; New table
key = COUNTERS_TUNNEL_NAME_MAP ;
; field = value
sip:dip = tunnel_vid ;uint64
```

**RATES TABLE**

Producer: syncd

Consumer: show scripts

Description:

- Per Tunnel Rx/Tx packets and octet rate counters are added.
- Per Tunnel Last recorded Rx/Tx packet and octet counters

Schema:

```
; Existing table
key = RATES:tunnel_vid ; tunnel oid
; field = value
SAI_TUNNEL_STAT_IN_OCTETS_last = number ;uint64
SAI_TUNNEL_STAT_IN_PACKETS_last = number ;uint32
SAI_TUNNEL_STAT_OUT_OCTETS_last = number ;uint64
SAI_TUNNEL_STAT_OUT_PACKETS_last = number ;uint32
RX_BPS = number ;uint32
RX_PPS = number ;uint32
TX_BPS = number ;uint32
TX_PPS = number ;uint32
```

### 4.2.5 FLEX_COUNTER_DB changes

**FLEX_COUNTER_GROUP TABLE**

Producer: Flexcounterorch

Consumer: syncd

Description:

- Contains the polling interval for VXLAN tunnels.

Schema:

```
; Existing table
key = FLEX_COUNTER_GROUP_TABLE:TUNNEL_STAT_COUNTER ;
; field = value
POLL_INTERVAL = number ;uint32 in msec
FLEX_COUNTER_STATUS = enable/disable ; enable or disable tunnel counters
```

## 4.3 Modules Design and Flows

Expand Down Expand Up @@ -1014,6 +1113,8 @@ Kernel uses a single flag for both ARP & ND suppression, We support only one CLI
### 4.3.11 Support for Tunnel Statistics

With the L2 and L3 support over VXLAN it is useful to support tunnel statistics.
The support for Tunnel statistics shall follow the existing counter architecture.
The calculation of tunnel counter rates shall be as described in the Reference section.

#### 4.3.11.1 Counters supported.

Expand All @@ -1022,13 +1123,16 @@ These will be stored in the counters DB for each tunnel.

#### 4.3.11.2 Changes to SwSS

- Flexcounter group for tunnels added with poll interval of 1 second and stat mode being STAT_MODE_READ. This is called as part of VxlanOrch constructor.
- Flexcounter group for tunnels added with default poll interval of 3 second and stat mode being STAT_MODE_READ. This is called as part of VxlanOrch constructor.
- VxlanOrch adds and removes tunnels in the FLEX_COUNTER_DB when a tunnel gets created or destroyed. Addition to the FLEX_COUNTER_DB will be done after all the SAI calls to create the tunnel. Removal from the DB will be done before SAI calls to delete the tunnel.
- Handles polling interval changes by processing the FLEX_COUNTER_TABLE in the CONFIG_DB.

#### 4.3.11.3 Changes to syncd

- Flex counter and Flex group counter processing to also include tunnels.
- Include Tunnel counters as part of FlexCounter::collectCounters call.
- The existing get_tunnel_stats API shall be used to fetch the stats from the sai driver.
- Handle polling interval changes for the tunnel group.

### 4.3.12 VXLAN QoS mode

Expand Down Expand Up @@ -1118,6 +1222,9 @@ Linux kernel version 4.9.x used in SONiC requires backport of a few patches to s
- config neigh_suppress enable <vid>
- config neigh_suppress disable <vid>
- vid represents the vlan_id. By default ARP/ND suppression is disabled.
6. VXLAN counter polling interval
- counterpoll tunnel interval <poll_interval>
- poll_interval is in msec. Range is 3000-30000. Default is 3000.
```

Expand Down Expand Up @@ -1172,11 +1279,11 @@ Linux kernel version 4.9.x used in SONiC requires backport of a few patches to s
+---------+---------+-------------------+--------------+
Total count : 2
5. show vxlan remote_mac <remoteip/all> <vlanid/all>
5. show vxlan remotemac <remoteip/all> <vlanid/all>
- lists all the MACs learnt from the specified remote ip or all the remotes for the specified/all vlans. (APP DB view)
- VLAN, MAC, RemoteVTEP, VNI, Type are the columns.
show vxlan remote_mac all
show vxlan remotemac all
+---------+-------------------+--------------+-------+--------+
| VLAN | MAC | RemoteVTEP | VNI | Type |
+=========+===================+==============+=======+========+
Expand All @@ -1194,7 +1301,7 @@ Linux kernel version 4.9.x used in SONiC requires backport of a few patches to s
+---------+-------------------+--------------+-------+--------+
Total count : 6
show vxlan remote_mac 3.3.3.3
show vxlan remotemac 3.3.3.3
+---------+-------------------+--------------+-------+--------+
| VLAN | MAC | RemoteVTEP | VNI | Type |
+=========+===================+==============+=======+========+
Expand All @@ -1205,11 +1312,11 @@ Linux kernel version 4.9.x used in SONiC requires backport of a few patches to s
Total count : 2
6. show vxlan remote_vni <remoteip/all>
6. show vxlan remotevni <remoteip/all>
- lists all the VLANs learnt from the specified remote ip or all the remotes. (APP DB view)
- VLAN, RemoteVTEP, VNI are the columns
show vxlan remote_vni all
show vxlan remotevni all
+---------+--------------+-------+
| VLAN | RemoteVTEP | VNI |
+=========+==============+=======+
Expand All @@ -1219,14 +1326,45 @@ Linux kernel version 4.9.x used in SONiC requires backport of a few patches to s
+---------+--------------+-------+
Total count : 2
show vxlan remote_vni 3.3.3.3
show vxlan remotevni 3.3.3.3
+---------+--------------+-------+
| VLAN | RemoteVTEP | VNI |
+=========+==============+=======+
| Vlan101 | 3.3.3.3 | 1001 |
+---------+--------------+-------+
Total count : 1
7. show vxlan counters <remoteip/all>
- lists all the tunnel statistics
- lists also the configured polling interval
Polling Interval : 3 seconds
SIP DIP RX_OK RX_BPS RX_PPS TX_OK TX_BPS TX_PPS
------ --- -------- -------- -------- ------- -------- --------
2.2.2.2 3.3.3.3 0 0 0 0 0 0
2.2.2.2 3.3.3.4 0 0 0 0 0 0
8. counterpoll show enhanced to display the tunnel polling interval.
counterpoll show
Type Interval (in ms) Status
-------------------------- ------------------ --------
QUEUE_STAT default (10000) enable
PORT_STAT default (1000) enable
PORT_BUFFER_DROP default (60000) enable
RIF_STAT default (1000) enable
TUNNEL_STAT default (3000) enable
BUFFER_POOL_WATERMARK_STAT default (10000) enable
PG_WATERMARK_STAT default (10000) disable
```

#### 5.1.3 Clear Commands


```
1. sonic-clear tunnelcounters
clears counters for all the tunnels.
```

Expand Down Expand Up @@ -1257,6 +1395,9 @@ Linux kernel version 4.9.x used in SONiC requires backport of a few patches to s
- switch(config) interface vlan <vlan-id-xxx>
- switch(conf-if-vlanxxx)# [no] neigh-suppress
- This command will suppress both ARP & ND flooding over the tunnel when Vlan is extended.
6. VXLAN counter polling interval
- switch(config) counters vxlan interval <polling_interval>
- polling_interval is in seconds from 3-30 sec.
```

Expand Down Expand Up @@ -1290,6 +1431,14 @@ Linux kernel version 4.9.x used in SONiC requires backport of a few patches to s
```

#### 5.2.4 Clear commands

```
1. clear counters vxlan <remoteip/all>
Clears counters for a specific or all tunnels.
```

### 5.3 Openconfig URI


Expand Down Expand Up @@ -1384,6 +1533,21 @@ GET support - state
curl -v -X GET -u admin:YourPaSsWoRd https://10.59.132.165/restconf/data/openconfig-vxlan:vxlan/state/vxlan-vni-peer-infos -k
curl -v -X GET -u admin:YourPaSsWoRd https://10.59.132.165/restconf/data/openconfig-vxlan:vxlan/state/vxlan-tunnel-infos/vxlan-tunnel-info -k
VXLAN Counter commands
-----------------------
Show Command
curl -v -X GET -u admin:YourPaSsWoRd "https://x.x.x.x/restconf/operations/openconfig-counters-ext:vxlan-tunnel-counters" -H "Content-Type: application/yang-data+json" -d "{\"openconfig-counters-ext:input\":{\"tunneliface\":\"all\"}}" -k
Clear Command
curl -v -X POST -u admin:YourPaSsWoRd "https://x.x.x.x/restconf/operations/openconfig-interfaces-ext:clear-vxlan-tunnel-counters" -H "Content-Type: application/yang-data+json" -d "{\"openconfig-interfaces-ext:input\":{\"tunnel\":\"all\"}}" -k
Config Command
curl -v -X PATCH -u admin:YourPaSsWoRd "https://x.x.x.x/restconf/data/openconfig-counters-ext:vxlan-tunnel-counters/vxlan-tunnel-counter=time_value/config" -H "Content-Type: application/yang-data+json" -d "{\"openconfig-counters-ext:config\":{\"vxlan-tunnel-counter-interval\":time_value}}" -k
```

## 6 Serviceability and Debug
Expand Down Expand Up @@ -1611,3 +1775,10 @@ Queuing on tunnel termination is to queue 0 by default. User should bind a DSCP
7. Move remote MAC to local by programming same entry in STATE_FDB_TABLE and verify Linux and FRR are updated
8. Move local MAC entry to remote by replacing fdb entry in Linux and verify VXLAN_FDB_TABLE and STATE_FDB_TABLE are updated.

## 9 References

- [SONiC VXLAN HLD](https://github.com/Azure/SONiC/blob/master/doc/vxlan/Vxlan_hld.md)
- [SONiC Counter Rates](https://github.com/Azure/SONiC/tree/master/doc/rates-and-utilization)
- [RFC 7432](https://tools.ietf.org/html/rfc7432)
- [RFC 8365](https://tools.ietf.org/html/rfc8365)

0 comments on commit 3c945ad

Please sign in to comment.