Contrib. Project: Latency Gadget: delay_mb

Introduction

The delay_mb is a latency control gadget, which provides a bump-in-the-wire device. The device provides two basic functions:

1. Latency variation
2. Rate control

The project is provided as a contrib project under NetFPGA-SUME release 1.4.0. The project is provided "as is" and is not maintained. It is recommended for users to change the design and adapt IP parameters according to their needs.

Architecture

The delay_mb project provides a bump-in-the-wire which can be connected between two devices: the first device will should be connected to NetFPGA-SUME Port 0, and the second device to NetFPGA-SUME Port 1. All traffic from Port 0 is sent to Port 1, and all traffic from Port 1 is sent to Port 0.

The basic infrastructure for this project is taken from the reference_switch project, which means that the clocking, resets, registers access and 10G ports are identical to the reference switch.

The nf_datapath module is modified: gone are the input_arbiter, output_port_lookup and output_queues. From port 0 to port 1 there are two connected modules: first delay and then rate_limiter. From port 0 to port 1 there is one connected module: delay. This means that latency control is bi-directional and rate limiting is uni directional.

Inputs and outputs to all modules are using the standard AXI-S interface, 256b wide.

In the current project, the modules are mapped as follows:

1. Port 0 to Port 1 delay module: 0x44010000

2. Port 1 to Port 0 delay module: 0x44020000

3. Port 0 to Port 1 rate_limiting: 0x44030000

Delay module

The delay module is basically just a large FIFO (32K entries of 32B). The latency control is provided by time stamping every entering packet and checking the time at the head of the FIFO before reading the packet. A single register (0x4401001c or 0x4402001c) sets the latency, with 5ns resolution. For example, setting a value of 0xA will provide an additional latency of 50ns. As the register is 32b wide, one can configure it to provide latency at the order of seconds, however due to the FIFO depth it is not supposed to sustain full 10Gbps for longer than 700us (theoretical upper bound). At lower data rates the module was shown to support latency in the order of milliseconds.

Rate Limiter module

The rate limiter is also a FIFO-based module, which provides pacing of data out of the FIFO (8K entires of 32B). The pacing of data is done using two registers:

1. RATE_BASE (0x4403001C) - Sets a base value for the rate counter, defining a cycle length.

2. RATE_VALID (0x44030020) - Sets the valid value for the rate counter, defining the number of clocks data is valid in every cycle.

The basic idea is that data is read out of the FIFO every VALID clock, and that there a BASE number of clocks in every cycles. This means that if RATE_BASE=2 and RATE_VALID=1, one of every 2 clocks data will be read from the FIFO. If RATE_BASE=200 and RATE_VALID=100, data will be read from the FIFO every 100 of 200 clocks. Note that this provides control over the burstiness of data, but that only constant bit rate (CBR) is supported.

An important note when using the module:

The core clock of NetFPGA-SUME is 200MHz. This provides a potential bandwidth of up to 51.2Gbps. With 10Gbps, this means that setting RATE_VALID=1 and RATE_BASE=5 will provide full data rate of close to it. Recommended values for example data rates will be as follows (value in bracket indicates best value measured in an iperf test, which is not highly accurate):

1. 5Gbps - RATE_BASE=10, RATE_VALID=1 (RATE_BASE=48, RATE_VALID=5)

2. 1Gbps - RATE_BASE=50, RATE_VALID=1 (RATE_BASE=48, RATE_VALID=1)

3. 100Mbps - RATE_BASE=500, RATE_VALID=1 (RATE_BASE=483, RATE_VALID=1)

Testing

The delay_mb project uses the standard nf_test.py testing infrastructure. The test supported under the project is taken from the reference_switch simple broadcast (both_simple_broadcast), and is run the same way. The test does not validate timing.