-
Notifications
You must be signed in to change notification settings - Fork 15
Ethernet Packet Format
This page documents the Ethernet packet format for communicating with the FPGA boards. Note that Ethernet is only used to interface to the first FPGA board, which then acts as a "bridge" (or "hub") to the other FPGA boards connected via FireWire (IEEE-1394). Thus, the basic approach is to place a complete Firewire packet inside an Ethernet packet. There are two supported protocols: Ethernet Raw and Ethernet UDP. In both cases, the PC places a "control" word (FwCtrl) just before the Firewire packet. In the first case, the Firewire packet is placed immediately after the Ethernet frame header and control word. In the second case, The Ethernet frame header is followed by the IPv4 header, then the UDP header, and finally the control word and Firewire packet. Firewire response packets (from FPGA to PC) are similar, except that they do not contain a leading control word, but instead include 4 Extra words (8 bytes) following the Firewire packet.
The UDP protocol is recommended for maximum portability.
Terminology:
- PC: control computer
- Bridge: Ethernet to FireWire FPGA bridge board
- FPGA: JHU FPGA1394 board (with companion board such as QLA)
The basic raw Ethernet frame looks like this:
| Dest MAC | Source MAC | Ethertype | Data | Checksum |
|---|---|---|---|---|
| 6 bytes | 6 bytes | 2 bytes | Variable | 4 bytes |
Each FPGA board has a defined MAC address, which is based on the JHU LCSR Company ID (CID) obtained from the IEEE Registration Authority (issued December 17, 2015), which is FA-61-0E. Because it is a CID, and not an Organizationally Unique Identifier (OUI), it can be used to create locally-managed Ethernet MAC addresses (i.e., it is not globally unique). Since a MAC address is 48 bits, the CID forms the first 24 bits and the following convention is used to generate the last 24 bits:
FA-61-0E-13-94-NN
where NN is an 8-bit number determined by the rotary switch on the FPGA1394 board (this requires just 4 bits, so the upper 4 are 0).
For a raw Ethernet frame, the Ethertype field contains the packet length, which cannot be more than 1500 bytes. Values greater than 1536 have specific meanings, such as 0x0800 (2048) to designate an IP packet.
The Data part should be the 16-bit control word, followed by a valid and complete FireWire packet. After a sanity check, the Firewire packet will be sent out to the FireWire bus.
For a quadlet read command initiated by PC to FPGA board 4, the FireWire frame is first generated on the PC according to IEEE1394 specifications; the Ethernet header and checksum are created following that.
The Ethernet frame looks like this:
| Dest MAC | Source MAC | Ethertype | Data | Checksum |
|---|---|---|---|---|
| 6 bytes | 6 bytes | 2 bytes | 18 bytes | 4 bytes |
| FA-61-0E-13-94-04 | PC MAC_ADDR | packetLen | FwCtrl (2), FW frame (16) | CRC32 |
After receiving response for the quadlet read command, the FPGA constructs an Ethernet frame based on the response:
| Dest MAC | Source MAC | Ethertype | Data | Checksum |
|---|---|---|---|---|
| 6 bytes | 6 bytes | 2 bytes | 28 bytes | 4 bytes |
| PC MAC_ADDR | FA-61-0E-13-94-04 | packetLen | FW frame (20), Extra (8) | CRC32 |
The UDP implementation requires a UDP packet, inside an IP packet, which is inside the raw Ethernet frame. In this case, the EtherType is set to 0x0800 for IPv4.
The host PC Ethernet should be configured for "link-local", which uses IP addresses 169.254.xxx.xxx. The FPGA boards do not have hard-coded IP addresses; instead the IP address is assigned by sending a broadcast Ethernet packet that contains a broadcast Firewire quadlet write command, which sets the IP address via FPGA register 11. The firmware implements the ARP protocol, so that the PC can follow the usual process of obtaining the MAC address corresponding to the IP address. Currently, the PC software assigns a default IP address of 169.254.0.100 to the FPGA board, but this can be changed in software. All UDP communications occur via Port 1394.
The Ethernet Raw and Ethernet UDP both follow a similar initialization sequence:
- Broadcast quadlet write to register 11 to set the IP address (UDP only).
- Broadcast quadlet read from register 4 to obtain the hardware version (should be "QLA1"). Note that a flag is set to prevent this broadcast command from being sent over the Firewire bus (i.e., it is only processed by the FPGA that received it via Ethernet). The read response (from the FPGA) also indicates the current Firewire bus generation.
- Broadcast quadlet write to register 1 to initiate a Read of Firewire PHY Register 0. This allows each FPGA board to obtain its Firewire node id.
- Broadcast quadlet read from register 0 to obtain the board number (bits 27-24). This message is also not sent over the Firewire bus, so it returns the board number of the "hub" board (Ethernet/Firewire bridge).
- Scan for up to 16 nodes on the bus, where the following is done for each node (same as Firewire interface):
- Quadlet read from register 4 to obtain the hardware version ("QLA1").
- Quadlet read from register 7 to obtain the firmware version.
- Quadlet read from register 0 to obtain the board number.
All packets from the PC begin with a control word (2 bytes), with the following format:
| MS Byte | LS Byte |
|---|---|
| Control flags | Firewire bus generation from PC |
Currently, the only defined control flag is FW_CTRL_NOFORWARD (bit 0). All other bits should be set to 0.
All packets sent from the FPGA to the PC end with 4 extra words (8 bytes), with the following format:
| Word | Contents (# bits) |
|---|---|
| 0 | Status flags (8), Firewire bus generation from FPGA (8) |
| 1 | Unused (8), Number of packet errors (8) |
| 2 | Time required for FPGA to receive packet (16) |
| 3 | Total time required for FPGA to process packet (16) |
The status flags are as follows:
| 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | SummaryError | InternalError | PacketDropped | FwBusReset |
- SummaryError: 1 -> a packet error occurred (see Ethernet Status)
- InternalError: 1 -> an internal error occurred (see Ethernet Status)
- PacketDropped: 1 -> the packet was dropped due to a Firewire bus reset or a mismatch in the Firewire bus generation number
- FwBusReset: 1 -> the Firewire bus is being reset
The timing information (Words 2 and 3) is in FPGA clock ticks, which has a resolution of 20.345 nsec (49.152 MHz). Note that although the FPGA measures timing for all packets, the PC only receives information for read packets, since write packets do not return a result and therefore there is no extra data. But, the time to receive a write packet should be about the same as the time to receive a read packet. The total time includes the time to receive the packet and to send the response, but the measurement is a little short because it does not include the time to write the value to the transmit queue or to queue the packet for transmission.
The FPGA boards support the ICMP Echo command (used by "ping"). The maximum ping data size is 1472 bytes because we do not fragment packets.
- FireWire Broadcast frame format
| Date | Author | Comment |
|---|---|---|
| 2014-10-11 | Zihan Chen | Outline |
| 2014-10-14 | Long Qian | More details |
| 2016-09-28 | Peter Kazanzides | Added some UDP info |
| 2019-10-25 | Peter Kazanzides | Updated (now use LCSR CID for MAC address) |
| 2020-08-13 | Peter Kazanzides | Added more UDP information |