PNA.mdk

Title : P4 Portable NIC Architecture (PNA)
Title Note: (working draft after version 0.7 release)
Title Footer: &date;
Author : The P4.org Architecture Working Group
Heading depth: 4
Pdf Latex: xelatex
Document Class: [11pt]article
Package: [top=1in, bottom=1.25in, left=1in, right=1in]geometry
Package: fancyhdr

Tex Header:
  \setlength{\headheight}{30pt}
  \renewcommand{\footrulewidth}{0.5pt}

@if html {
body.madoko {
  font-family: utopia-std, serif;
}
title,titlenote,titlefooter,authors,h1,h2,h3,h4,h5 {
  font-family: helvetica, sans-serif;
  font-weight: bold;
}
pre, code {
  language: p4;
  font-family: monospace;
  font-size: 10pt;
}
}

@if tex {
body.madoko {
  font-family: UtopiaStd-Regular;
}
title,titlenote,titlefooter,authors {
  font-family: sans-serif;
  font-weight: bold;
}
pre, code {
  language: p4;
  font-family: LuxiMono;
  font-size: 75%;
}
}

Colorizer: p4
.token.keyword    {
    font-weight: bold;
}

@if html {
p4example {
  replace: "~ Begin P4ExampleBlock&nl;\
                 ````&nl;&source;&nl;````&nl;\
                 ~ End P4ExampleBlock";
  padding:6pt;
  margin-top: 6pt;
  margin-bottom: 6pt;
  border: solid;
  background-color: #ffffdd;
  border-width: 0.5pt;
}
}

@if tex {
p4example {
  replace: "~ Begin P4ExampleBlock&nl;\
                 ````&nl;&source;&nl;````&nl;\
                 ~ End P4ExampleBlock";
  breakable: true;
  padding: 6pt;
  margin-top: 6pt;
  margin-bottom: 6pt;
  border: solid;
  background-color: #ffffdd;
  border-width: 0.5pt;
}
}


@if html {
p4pseudo {
  replace: "~ Begin P4PseudoBlock&nl;\
                 ````&nl;&source;&nl;````&nl;\
                 ~ End P4PseudoBlock";
  padding: 6pt;
  margin-top: 6pt;
  margin-bottom: 6pt;
  border: solid;
  background-color: #e9fce9;
  border-width: 0.5pt;
}
}

@if tex {
p4pseudo {
  replace: "~ Begin P4PseudoBlock&nl;\
                 ````&nl;&source;&nl;````&nl;\
                 ~ End P4PseudoBlock";
  breakable : true;
  padding: 6pt;
  margin-top: 6pt;
  margin-bottom: 6pt;
  background-color: #e9fce9;
  border: solid;
  border-width: 0.5pt;
}
}

@if html {
p4grammar {
  replace: "~ Begin P4GrammarBlock&nl;\
                 ````&nl;&source;&nl;````&nl;\
                 ~ End P4GrammarBlock";
  border: solid;
  margin-top: 6pt;
  margin-bottom: 6pt;
  padding: 6pt;
  background-color: #e6ffff;
  border-width: 0.5pt;
}
}

@if tex {
p4grammar {
  replace: "~ Begin P4GrammarBlock&nl;\
                 ````&nl;&source;&nl;````&nl;\
                 ~ End P4GrammarBlock";
  breakable: true;
  margin-top: 6pt;
  margin-bottom: 6pt;
  padding: 6pt;
  background-color: #e6ffff;
  border: solid;
  border-width: 0.5pt;
}
}

tbd {
    replace: "~ Begin TbdBlock&nl;\
                   TBD: &source;&nl;\
              ~ End TbdBlock&nl;";
    color: red;
}

[TITLE]

~ Begin Abstract
P4 is a domain-specific language for describing how packets are
processed by a network data plane. A P4 program comprises an
architecture, which describes the structure and capabilities of the
pipeline, and a user program, which specifies the functionality of the
programmable blocks within that pipeline. The Portable NIC
Architecture (PNA) is an architecture that describes the structure and
common capabilities of network interface controller (NIC) devices that
process packets going between one or more interfaces and a host
system.
~ End Abstract

[TOC]

# Introduction

Note that this document is still a working draft.  Significant changes
are expected to be made before version 1.0 of this specification is
released.

The Portable NIC Architecture (PNA) is the P4 architecture that defines
the structure and common capabilities for network interface controller
(NIC) devices. PNA comprises two main components:

1. A programmable pipeline that can be used to realize a variety of
different "packet paths" going between the various ports on the device
(e.g., network interfaces or the host system it is attached to), and

2. A library of types (e.g., intrinsic and standard metadata) and
P4~16~ externs (e.g., counters, meters, and registers).

PNA is designed to model the common features of a broad class of NIC
devices. By providing standard APIs and coding guidelines, the hope is
to enable developers to write programs that are portable across
multiple NIC devices that are conformant to the PNA[^PNAPortability].

[^PNAPortability]: Of course, given the tight hardware resource
    constraints on NIC devices, there is no promise that a given P4
    program that compiles on one device will also compile on another
    device. However, it should at least be the case that those P4
    programs that are able to compile on multiple NIC devices should
    process packets as described in this document.

The Portable NIC Architecture (PNA) Model has three programmable P4
blocks and several fixed-function blocks, as shown in Figure
[#fig-nic]. The behavior of the programmable blocks is specified using
P4. The network ports, packet queues, and (optional) inline accelerators
are fixed-function blocks that can be configured by the control
plane, but are not intended to be programmed using P4.

~ Figure {#fig-nic; caption: "Programmable NIC Architecture Block Diagram"; page-align: here;}
![nic]
~
[nic]: figs/flex-routing-arch.png { width: 100%;}

## Packet processing

Packets arriving from a network port or from the hosts first go through a
`MainParser`, which is responsible for extracting all relevant packet headers.
Then the extracted headers and associated metadata are processed by
`MainControl`. The code executed in the `MainControl` transforms headers,
updates stateful elements like counters, meters, and registers, and optionally
associates additional user-defined metadata with the packet. The `MainDeparser`
serializes the headers back into a packet that can be sent onwards.

After the `MainDeparser`, the packet is processed by one or more inline
accelerators. The P4 program executed in the `MainControl` determines whether
and how each inline accelerator processes the packet by executing methods of
a corresponding extern object(s).

Upon completion of processing in the inline accelerators, a packet may either:

- Proceed to the message processing part of the NIC. If the packet had
  originally been received through the network, this is a packet being  received
  by the host or a VM. If the packet had originally arrived to the packet processing
  block from the host, this is enables  on-NIC processing of VM-to-VM or
  host-to-host packets (i.e., on a system with multiple hosts).

- Head towards the network ports. If the packet had originally arrived to the
  packet processing block from the host, this is a packet transmission by the
  host or a VM.  If the packet had originally been received through the network,
  this enables on-NIC processing of port-to-port packets without ever traversing
  the host system.

- Go back into the packet processing block to be processed again (a.k.a.
  recirculation).

The choice of which network port to go to, or whether to loop back, or whether
to  proceed to the hosts (and which one) are all controlled from the P4 code
running in the `MainControl` block, via extern functions defined by this PNA
specification.

The same `MainParser`, `MainControl`, and `MainDeparser` that process packets
from the network are also used to process packets from the host. PNA was
designed this way for two reasons:

  - It is expected that in many cases, the packet processing in both
    directions will have many similarities between them. Writing
    common P4 code for both eliminates code duplication that would
    occur if the code for each direction was written separately.
  - Having a single `MainControl` in the P4 language enables tables
    and externs such as counters and registers to be instantiated
    once, and shared by packets being processed in both
    directions. The hardware of many NICs supports this design,
    without having to instantiate a physically separate table for each
    direction. Especially for large tables used by packet processing
    in both directions, this approach can significantly reduce the
    memory required. It is also critical for some stateful features
    (e.g. those using the table add-on-miss capability defined later
    in this specification) to access the same table in memory when
    processing packets in either direction.

Figure [#fig-nic] shows multiple hosts. Some NICs support PCI Express
connections to multiple host CPU complexes. It is also common for NICs
to have an array of one or more CPU cores inside of the NIC device
itself, and these can be the target for packets received from the
network, and/or the source of packets sent to the network, just as the
other hosts can be. For the purposes of the PNA, such CPU cores are
considered as another host.

## Message processing

The focus in the current version of this specification is on the three
P4-programmable blocks mentioned above. The details of how one can
use P4 to program the message processing portion of a NIC is left as
a future extension of this specification. While there are options for
exactly what packet processing functions can be performed in the four
primary blocks described above, versus the message processing block, the
division is expected to be:

- The primary programmable blocks deal solely with individual network
  packets, which are at most one network maximum transmission unit
  (MTU) in size.
- The message processing block is responsible for converting between
  large messages in host memory and network size packets on the
  network, and for dealing with one or more host operating systems,
  drivers, and/or message descriptor formats in host memory.

For example, in its role of converting between large messages and
network packets in the host-to-network direction, message processing
would implement features like large send offload (LSO), TCP
segmentation offload (TSO), and Remote Direct Memory Access (RDMA)
over Converged Ethernet (RoCE). In the network-to-host direction it
would assist in such features as large receive offload (LRO) and RoCE.

In its role of handling different kinds of operating systems, drivers,
and message descriptor formats, the message processing block may deal
with one or more of the following standards:
- VirtIO
- SR-IOV

Another potential criteria for dividing packet processing
functionality between message processing and the rest of the NIC is
for division of control plane responsibilities. For example, in some
network deployments the NIC message processing block configuration is
tightly coupled with the host operating system, whereas the
`MainControl` is controlled by network-focused control plane software.




## PNA P4~16~ architecture

A programmer targeting the PNA is required to provide P4 definitions
for each of the programmable blocks in the pipeline (see section
[#sec-programmable-blocks]). The programmable block inputs and outputs
are parameterized on the types of user defined headers and metadata.
The top-level PNA program instantiates a package named `main` with the
programmable blocks passed as arguments (see Section TBD for an
example). Note that the `main` package is not to be confused with the
`MainControl`.

This document contains excerpts of several P4~16~ programs that use
the `pna.p4` include file and demonstrate features of PNA. Source code
for the complete programs can be found in the official repository
containing the PNA specification[^PNAExamplePrograms].

[^PNAExamplePrograms]: <https://github.com/p4lang/pna>
    in directory `examples`.  Direct link:
    <https://github.com/p4lang/pna/tree/main/examples>


## Guidelines for Portability

A P4 programmer wishing to maximize the portability of their program should
follow several general guidelines:

- Do not use undefined values in a way that affects the resulting output
  packet(s), or for side effects such as updating `Counter`, `Meter` or
  `Register` instances.

- Use as few resources as possible, e.g. table search key
  bits, array sizes, quantity of metadata associated with packets, etc.


# Naming conventions

In this document we use the following naming conventions:

- Types are named using CamelCase followed by `_t`. For example, `PortId_t`.
- Control types and extern object types are named using CamelCase. For
  example `MainParser`.
- Struct types are named using lower case words separated by `_`
  followed by `_t`. For example `pna_input_metadata_t`.
- Actions, extern methods, extern functions, headers, structs, and
  instances of controls and externs start with lower case and words
  are separated using `_`. For example `send_to_port`.
- Enum members, const definitions, and #define constants are all
  caps, with words separated by `_`. For example `PNA_PORT_CPU`.

Architecture specific metadata (e.g. structs) are prefixed by `pna_`.

# Packet paths {#packet-paths}

Figure [#fig-packet-paths] shows all possible paths for packets that
must be supported by a PNA implementation. An implementation is
allowed to support paths for packets that are not described here.

~Begin TBD
Create another figure with the updated architecture diagram and names for the paths.
~End TBD

~ Figure {#fig-packet-paths; caption: "Packet Paths in PNA"; page-align: here;}
~

Table [#results-of-one-pkt-thru-main] shows what can happen to a
packet as a result of a single time being processed through the four
programmable blocks of the packet processing part of PNA, referred to
here as "main".

~ TableFigure {#results-of-one-pkt-thru-main; \
  caption: Result of packet processed one time by main.}
|-------------|---------------|--------------------|
|             | Processed     | Resulting          |
| Description | next by       | packet(s)          |
+:------------+:--------------+:-------------------+
| packet from   | main, with    | Zero or more mirrored  |
| network port  | direction     | packets, plus at most  |
|---------------|               | |
| packet from   | `NET_TO_HOST` | one of: a net-to-host  |
| net-to-host   |               | recirculated packet,   |
| recirculate   |               | or one to-host packet. |
|---------------|               |                        |
| packet from   |               |                        |
| port loopback |               |                        |
|---------------|---------------|------------------------|
| packet from   | main, with    | Zero or more mirrored  |
| message processing | direction | packets, plus at most |
|---------------|               | |
| packet from   | `HOST_TO_NET` | one of: a host-to-net  |
| host-to-net   |               | recirculated packet,   |
| recirculate   |               | or one to-net packet.  |
|---------------|               |                        |
| packet from   |               |                        |
| host loopback |               |                        |
|---------------|---------------|------------------------|
~

Note that each mirrored packet that results from `mirror_packet`
operations will have its own next place that it will go to be
processed, independent of the original packet, and independent of any
other mirror copies made of the same original packet.

# PNA Data types

## PNA type definitions {#sec-pna-type-definitions}

Each PNA implementation will have specific bit widths in the data
plane for the types shown in the code excerpt of Section
[#sec-pna-type-definitions-code].  These widths are defined in the
target specific `pna.p4` include file.  They are expected to differ
from one PNA implementation to another[^PNAP4TargetSpecific].

[^PNAP4TargetSpecific]: It is expected that `pna.p4` include files for
    different targets will be nearly identical to each
    other.  Besides the possibility of differing bit widths for these
    PNA types, the only expected differences between `pna.p4` files
    for different targets would be annotations on externs, etc. that
    the P4 compiler for that target needs to do its job.

For each of these types, the P4 Runtime API[^P4RuntimeAPI] may use bit
widths independent of the targets. These widths are defined by the P4
Runtime API specification, and they are expected to be at least as
large as the corresponding `InHeader_t` type below, such that they
hold a value for any target. All PNA implementations must use data
plane sizes for these types no wider than the corresponding
`InHeader_t`-defined types.

[^P4RuntimeAPI]: The P4Runtime Specification can be found here:
    <https://p4.org/specs>

### PNA type definition code excerpt {#sec-pna-type-definitions-code}

~ Begin P4Example
[INCLUDE=pna.p4:Type_defns]
~ End P4Example

### PNA port types and values

There are two types defined by PNA for holding different kinds of
ports: `PortId_t` and `InterfaceId_t`.

The type `PortId_t` must be large enough in the data plane to hold one
of these values:

+ a data plane id for one network port
+ a data plane id for one vport

As one example, a PNA target with four Ethernet network ports could
choose to use the values 0 through 3 to identify the network ports,
and the values 4 through 1023 to identify vports.

PNA makes no requirement that the numeric values identifying network
ports must be consecutive, nor for vports.  PNA only requires that for
every possible numeric value `x` with type `PortId_t`, exactly one of
these statements is true:

+ `x` is the data plane id of one network port, but not any vport
+ `x` is the data plane id of one vport, but not any network port
+ `x` is the data plane id of no port, neither a network port nor a vport

## PNA supported metadata types

~ Begin P4Example
[INCLUDE=pna.p4:Metadata_types]
~ End P4Example

## Match kinds
PNA supports the `match_kinds` specified in section 4.3 of the
PSA specification.

## Data plane vs. control plane data representations {#sec-data-plane-vs-control-plane-values}

# Programmable blocks {#sec-programmable-blocks}

The following declarations provide a template for the programmable
blocks in the PNA. The P4 programmer is responsible for implementing
controls that match these interfaces and instantiate them in a package
definition.

It uses the same user-defined metadata type `IM` and header type `IH`
for all ingress parsers and control blocks.  The egress parser and
control blocks can use the same types for those things, or different
types, as the P4 program author wishes.

~ Begin P4Example
[INCLUDE=pna.p4:Programmable_blocks]
~ End P4Example

# Packet Path Details {#packet-path-details}

Refer to section [#packet-paths] for the packet paths provided by PNA.

~Begin TBD
Need to decide where multicast replication can occur, and in what
conditions.
~End TBD

~Begin TBD
Need to decide where packet mirroring occurs, and in what
conditions, and how the mirrored packets differ from the originals.
~End TBD

~Begin TBD
Rewrite this section once the overall architecture is approved
~End TBD

## Initial values of packets processed by main parser

~ Begin Comment
jnfoster: is this a case where pseudocode would be helpful for
defining the initial values of headers/metadata? Or would that be too
precise and therefore constraining?
~ End Comment

### Initial packet contents for packets from ports

Packet is as received from Ethernet port.

User-defined metadata is empty?

### Initial packet contents for packets looped back from host-to-network path

Packet is as came out of host-to-net received from Ethernet port.

There can be user-defined metadata included with these packets.

## Initial values of packets processed in network-to-host direction by main block

### Initial packet contents for normal packets

The packet should be ...

The user-defined metadata should be ...

The standard metadata contents should be specified in detail here.

### Initial packet contents for recirculated packets

Give any differences between this case and previous section.

## Behavior of packets after main block is complete in network-to-host direction

Cases: drop, recirculate, loopback to host-to-net direction, to
message processing.  Describe the conditions in which each occurs.

## Initial values of packets processed in host-to-network direction by main block

### Initial packet contents for normal packets

This is for packets from the message processing block.

### Initial packet contents for recirculated packets

Give any differences between this case and previous section.

### Initial packet contents for packets looped back after network-to-host main processing

## Behavior of packets after main block is complete in host-to-network direction

Cases: drop, recirculate, to queues.  Describe the conditions in which
each occurs.

## Contents of packets sent out to ports

## Functions for directing packets

### Extern function `send_to_port`

~ Begin P4Example
[INCLUDE=pna.p4:send_to_port]
~ End P4Example

The extern function `send_to_port` is used to direct a packet to a
specified network port, or to a vport.  Invoking `send_to_port(x)` is
supported only within the main control.  There are two cases to
consider, detailed below.

+ `x` is a network port id.

Calling `send_to_port(x)` modifies hidden state for this packet, so
that the packet will be transmitted out of the network port with id
`x`.

+ `x` is a vport id.

Calling `send_to_port(x)` modifies hidden state for this packet, so
that the packet will be sent to the vport with id `x` in the host,
without being looped back.

## Packet Mirroring

~ Begin P4Example
[INCLUDE=pna.p4:mirror_packet]
~ End P4Example

The extern function `mirror_packet` is used to cause a mirror copy of
the packet currently being processed to be created.  Invoking
`mirror_packet(x)` is supported only within the main control.

PNA enables multiple mirror copies of a packet to be created during a
single execution of `MainControl`, by calling `mirror_packet` with
different mirror slot id values.  PNA targets should support
`mirror_slot_id` values in the range 0 through 3, at least, but are
allowed to support a larger range.

When `MainControl` begins execution, all mirror slots are initialized
so that they do not create a copy of the packet.

After calling `mirror_packet(slot_id, session_id)`, then when the main
control finishes execution, the target will make a best effort to
create a copy of the packet that will be processed according to the
parameters configured by the control plane for the mirror session
numbered `session_id`, for mirror slot `slot_id`.  Note that this is
best effort -- if the target device is already near its upper limit of
its ability to create mirror copies, then some later mirror copies may
not be made, even though the P4 program requested them.

Each of the mirror slots is independent of each other.  For example,
calling `mirror_packet(1, session_id)` has no effect on mirror slots
0, 2, or 3.

Mirror session id 0 is reserved by the architecture, and must not be
used by a P4 developer.

If multiple calls are made to `mirror_packet()` for the same mirror
slot id in the same execution of the main control, only the last
`session_id` value is used to create a copy of the packet.  That is,
every call to `mirror_packet(slot_id, session_id)` overwrites the
effects of any earlier to `mirror_packet()` with the same `slot_id`.

The effects of `mirror_packet()` calls are independent of calls to
`drop_packet()` and `send_to_port()`.  Regardless of which of those
things is done to the original packet, up to one mirror packet per
mirror slot can be created.

The control plane code can configure the following properties of each
mirror session, independently of other mirror sessions:

+ `packet_contents`

If `PRE_MODIFY`, then the mirrored packet's contents will be the same
as the original packet as it was when the packet began the execution
of the main control that invoked the `mirror_packet()` function.

If `POST_MODIFY`, then the mirrored packet's contents will be the same
as the original packet that is being mirrored, after any modifications
made during the execution of the main control that invoked the
`mirror_packet()` function.

+ `truncate`

`true` to limit the length of the mirrored packet to the
`truncate_length`.  `false` to cause the mirrored packet not to be
truncated, in which case the `truncate_length` property is ignored for
this mirror session.

+ `truncate_length`

In units of bytes.  Targets may limit the choices here, e.g. to a
multiple of 32 bytes, or perhaps even a subset of those choices.

+ `sampling_method`

One of the values: `RANDOM_SAMPLING`, `HASH_SAMPLING`.

If `RANDOM_SAMPLING`, then a mirror copy requested for this mirror
session will only be created with a configured probability given by
the `sample_probability` property.

If `HASH_SAMPLING`, then a target-specific hash function will be
calculated over the packet's header fields resulting in a hash output
value `H`.  A mirror copy will be created if `(H & sample_hash_mask)
== sample_hash_value`.

+ `meter_parameters`

If the conditions specified by the `sampling_method` and other
sampling properties are passed, then a P4 meter dedicated for use by
this mirror session will be updated.  If it returns a `GREEN` result,
then the mirror copy will be created (still with best effort, if the
target device's implementation is still oversubscribed with requests
to create mirror copies).

If the meter update returns any result other than `GREEN`, then no
mirror copy will be created.

+ `destination_port`

A network port id, or a vport id.

If `destination_port` is a network port id, that network port is the
destination of mirrored copy packets created by this session.  If the
`mirror_packet()` call for this session was invoked in the
`NET_TO_HOST` direction, mirror copy packets created will loop back in
the host side of the target, and later come back for processing in the
main block in the `HOST_TO_NET` direction, already destined for the
network port `destination_port`.  That port can be overwritten by
calls to forwarding functions.

If `destination_port` is a vport id, that vport is the destination of
mirrored copy packets created by this session.  If the
`mirror_packet()` call for this session was invoked in the
`HOST_TO_NET` direction, mirror copy packets created will loop back in
the network port side of the NIC, and later come back for processing
in the main block in the `NET_TO_HOST` direction, already destined for
the vport `destination_port`.  That vport can be overwritten by calls
to forwarding functions.

~Begin TBD
When a mirror copied packet comes back to the main control, it will
have some metadata indicating it is mirror copy.  We should define a
way in PNA to recognize such mirror copies, e.g. some new extern
function call returning true if the packet was created by a
`mirror_packet` operation.
~End TBD

## Packet recirculation

# PNA Extern Objects

## Restrictions on where externs may be used {#sec-extern-restrictions}

All instantiations in a P4~16~ program occur at compile time, and can
be arranged in a tree structure we will call the instantiation tree.
The root of the tree `T` represents the top level of the program. Its
child is the node for the package `PNA_NIC` described in Section
[#sec-programmable-blocks], and any externs instantiated at the top
level of the program. The children of the `PNA_NIC` node are the
packages and externs passed as parameters to the `PNA_NIC`
instantiation. See Figure [#fig-instantiation-tree] for a drawing of
the smallest instantiation tree possible for a P4 program written for
PNA.

~ Figure {#fig-instantiation-tree; caption: "Minimal PNA instantiation tree"; page-align: here;}
~

If any of those parsers or controls instantiate other parsers,
controls, and/or externs, the instantiation tree contains child nodes
for them, continuing until the instantiation tree is complete.

For every instance whose node is a descendant of the `Ingress` node in
this tree, call it an `Ingress` instance. Similarly for the other
ingress and egress parsers and controls. All other instances are top
level instances.

A PNA implementation is allowed to reject programs that instantiate
externs, or attempt to call their methods, from anywhere other than
the places mentioned in Table [#table-extern-usage].

~ TableFigure {#table-extern-usage; \
  caption: Summary of controls that can instantiate and invoke externs.}
|-------------------|----------------------------------------------|
| Extern type       | Where it may be instantiated and called from |
+:------------------+:---------------------------------------------+
| `ActionProfile`   | MainControl |
|-------------------|----------------------------------------------|
| `ActionSelector`  | MainControl |
|-------------------|----------------------------------------------|
| `Checksum`        | MainParser, MainControl, MainDeparser |
|-------------------|----------------------------------------------|
| `Counter`         | MainControl |
|-------------------|----------------------------------------------|
| `Digest`          | MainDeparser |
|-------------------|----------------------------------------------|
| `DirectCounter`   | MainControl |
|-------------------|----------------------------------------------|
| `DirectMeter`     | MainControl |
|-------------------|----------------------------------------------|
| `Hash`            | MainControl |
|-------------------|----------------------------------------------|
| `InternetChecksum` | MainParser, MainControl, MainDeparser |
|-------------------|----------------------------------------------|
| `Meter`           | MainControl |
|-------------------|----------------------------------------------|
| `Random`          | MainControl |
|-------------------|----------------------------------------------|
| `Register`        | MainControl |
|-------------------|----------------------------------------------|
~

For example, `Counter` being restricted to "MainControl" means that
every `Counter` instance must be instantiated within the `MainControl`
block, or be a descendant of one of those nodes in the instantiation
tree. If a `Counter` instance is instantiated in Main, for example, then
it cannot be referenced, and thus its methods cannot be called, from any
block except `MainControl` or one of its descendants in the tree.

PNA implementations need not support instantiating these externs at
the top level. PNA implementations are allowed to accept programs that
use these externs in other places, but they need not. Thus P4
programmers wishing to maximize the portability of their programs
should restrict their use of these externs to the places indicated in
the table.

All methods for type `packet_out`, e.g., `emit`, are restricted to be
within deparser control blocks in PNA, because those are the only
places where an instance of type `packet_out` is visible. Similarly
all methods for type `packet_in`, e.g. `extract` and `advance`, are
restricted to be within parsers in PNA programs. P4~16~ restricts all
`verify` method calls to be within parsers for all P4~16~ programs,
regardless of whether they are for the PNA.

See the PSA specification for definitions of all of these externs.
There is work under way as of this writing that may result in these
extern definitions being moved from the PSA specification into a
separate standard library of P4 extern definitions, and if this is
done, both the PSA and PNA specifications will reference that.

## Extern Objects for Inline Accelerators {#sec-inline-accelerators}

A variety of inline accelerators can be present on a PNA target. These
accelerators perform specific functions on a packet. These functions are
typically implemented in hardware. These accelerators perform specific
functions on a packet after the deparser has finished executing.

These hardware functions are represented as extern objects in a P4 program.
An extern object representing a specific accelerator E.g. AES-GCM crypto
accelerator, can be instantiated in a P4 program. The methods defined by
the extern object are used to send and receive information to/from the
inline accelerator. Since the accelerators are present after the deparser,
the information sent to the accelerator takes effect only when packet reaches
the accelerator. Similarly any information received from accelerator is
for the previous function performed on the packet.

This section provides one example definition of a crypto acceleration engine.
Other extern objects can be defined in future based on the functionality
provided by the hardware accelerators.

~Begin P4Example
[INCLUDE=examples/include/crypto-accelerator.p4:Crypto_accelerator_extern_object]
~End P4Example

# PNA Table Properties

Table [#table-table-properties] lists all P4 table properties defined
by PNA that are not included in the base P4~16~ language
specification.

~ TableFigure {#table-table-properties; \
  caption: Summary of PNA table properties.}
|-----------------------|------|---------------------------------------|
| Property name         | Type | See also |
+:----------------------+:-----+:--------------------------------------+
| `add_on_miss`         | `boolean` | Section [#sec-add-on-miss] |
|-----------------------|------|---------------------------------------|
| `pna_direct_counter` | one `DirectCounter` instance name |  |
|-----------------------|------|---------------------------------------|
| `pna_direct_meter` | one `DirectMeter` instance name |  |
|-----------------------|------|---------------------------------------|
| `pna_implementation`  | instance name of one `ActionProfile` |  |
|                       | or `ActionSelector` |  |
|-----------------------|------|---------------------------------------|
| `pna_empty_group_action` | action |  |
|-----------------------|------|---------------------------------------|
| `pna_idle_timeout` | `PNA_IdleTimeout_t` | Section [#sec-idle-timeout] |
|-----------------------|------|---------------------------------------|
~

A PNA implementation need not support both of a `pna_implementation`
and `pna_direct_counter` property on the same table.

Similarly, a PNA implementation need not support both of a
`pna_implementation` and `pna_direct_meter` property on the same
table.

A PNA implementation must implement tables that have both a
`pna_direct_counter` and `pna_direct_meter` property.

A PNA implementation need not support both `pna_implementation` and
`pna_idle_timeout` properties on the same table.

## Tables with add-on-miss capability {#sec-add-on-miss}

PNA defines the `add_on_miss` table property.  If the value of this
property is `true` for a table `t`, the P4 developer is allowed to
define a default action for `t` that calls the `add_entry` extern
function.

When `t.apply()` is invoked, `t`'s lookup key is constructed, and the
entries of the table are searched.  If there is no match, i.e. the
lookup results in a miss, `t`'s default action is executed.  So far,
this is all standard behavior as defined in the P4~16~ language
specification.

If `t`'s default action makes a call to `add_entry`, it causes a new
entry to be added to the table with the same key that was just looked
up and resulted in a miss, and the action name and action parameters
specified by the parameters of the call to the `add_entry` extern
function.  Thus, future packets that invoke `t.apply()` with the same
lookup key will get a match and invoke the specified action (until and
unless this new table entry is removed).  The new table entry will be
matchable when the next packet is processed that invoked `t.apply()`.

Some PNA implementations may allow the control plane software to add,
modify, and delete entries of such a table, but any entries added via
the `add_entry` function do not require the control plane software to
be involved in any way.  Other PNA implementations may choose not to
support control plane modification of the entries of an add-on-miss
table.

It is expected that PNA implementations will be able to sustain
`add_entry` calls at a large fraction of their line rate, but it need
not be at the same packet rate supported for processing packets that
do not call `add_entry`.

~Begin P4Example
[INCLUDE=pna.p4:add_entry_extern_function]
~End P4Example

It is expected that many PNA implementations will restrict
`add_entry()` to be called with the following restrictions:

* Only from within an action
* Only if the action is a default action of a table with property
  `add_on_miss` equal to `true`.
* Only for a table with all key fields having match_kind `exact`.
* Only with an action name that is one of the hit actions of that same
  table.  This action has parameters that are all directionless.
* The type `T` is a struct containing one member for each
  directionless parameter of the hit action to be added.  The member
  names must match the hit action parameter names, and their types
  must be the same as the corresponding hit action parameters.

The new entry will have the same key field values that were
searched for in the table when the miss occurred, which caused the
table's default action to be executed.  The action will be the one
named by the string that is passed as the parameter `action_name`.

If the attempt to add a table entry succeeds, the return value is
`ADD_ENTRY_SUCCESS`, otherwise it will be some other value.  PNA
implementations are free to define additional failure reasons other
than `ADD_ENTRY_NOT_DONE`, but it is perfectly acceptable for a PNA
implementation to only support those two possible return values.


## Table entry idle timeout {#sec-idle-timeout}

PNA defines the table property `pna_idle_timeout` to enable specifying
whether a table should maintain an idle time for each of its entries,
and if so, what the data plane should do when a table entry has not
been matched for a length of time at least its configured idle time.

The value assigned to `pna_idle_timeout` must be a value of type
`PNA_IdleTimeout_t`:

~Begin P4Example
[INCLUDE=pna.p4:enum_PNA_IdleTimeout_t]
~End P4Example

If the property `pna_idle_timeout` is not specified for a table, its
default value is `NO_TIMEOUT`.  Such tables need not maintain an idle
time for any of its table entries, and will not perform any special
action regardless of how long a table entry remains unmatched.

If the property `pna_idle_timeout` is assigned a value of
`NOTIFY_CONTROL`, the behavior is the same as defined in the Portable
Switch Architecture if a table has its property `psa_idle_timeout`
assigned a value of `NOTIFY_CONTROL`.  See the section titled "Table
entry timeout notification" in the PSA specification[^PSA].

[^PSA]: The Portable Switch Architecture specification can be found here:
    <https://p4.org/specs>

If the property `pna_idle_timeout` is assigned a value of
`AUTO_DELETE`, the behavior is similar to the behavior of the value
`NOTIFY_CONTROL`, except that no notification message is generated to
the control plane when an entry's idle time is reached.  Instead, the
data plane deletes the table entry.

PNA implementations may restrict `pna_idle_timeout` to be
`AUTO_DELETE` only for tables that also have `add_on_miss` equal to
`true`.

PNA implementations are expected to be able to perform add-on-miss at
very high rates relative to line rate, and similarly for such
add-on-miss tables, they should be able to perform auto-deletion of
entries in the data plane at a similarly large rate.  If a P4
developer wishes to use the high rate add-on-miss capabilities for a
particular table, it is likely that they do not wish the control plane
to be responsible for keeping up with a high rate of deleting idle
entries, and thus will often use `add_on_miss = true` and
`pna_idle_timeout = PNA_IdleTimeout_t.AUTO_DELETE` together.


# Timestamps


# Atomicity of control plane API operations {#sec-atomicity-of-control-plane-api-operations}

# Appendix: Revision History {#appendix-revision-history; @h1:"A"}

|-----|-----|-----|
| Release | Release Date | Summary of Changes |
+:---:+-----+-----+
| 0.1 | November 5, 2020 | Skeleton specification. |
| 0.5 | May 15, 2021 | Initial draft. |
| 0.7 | December 22, 2022 | Version 0.7 |
+-----+-----+-----+

## Changes made in version 0.7

* Added extern functions `add_entry_if`, `set_entry_expire_time_if`,
  and `update_expire_info`, intended to be more friendly to targets
  with poor support for `if` statements inside of actions.
* Added parameter `expire_time_profile_id` to extern function
  `add_entry`, to specify the initial expire time profile id for a new
  table entry added by the data plane.
* Removed obsolete references to `send_to_vport` in example programs,
  replacing with current `send_to_port`.
* Add `crypto_accelerator` extern for basic encrypt / decrypt of a
  specified portion of a packet, and example program
  `ipsec-acc.p4` demonstrating its use.
* Added `match_kind` `optional`.
* Clarified restrictions on when extern function `add_entry` may be
  called.
* Removed `PreControl`, leaving `MainParser`, `MainControl`, and
  `MainDeparser` as the P4-programmable blocks.
* There is no more "loopback" in PNA.  A packet can be sent to a
  network port or a host port, regardless of where it came from, and
  this does NOT automatically cause the packet leaving the
  MainDeparser to later come back for another pass of processing.  It
  will only do so if recirculation is explicitly requested.
* Defined possible values of type `PNA_HashAlgorithm_t`, to match the
  values defined for PSA.
* Added descriptions of the possible values of `pna_idle_timeout`
  table property, and their behaviors.  What was originally proposed
  as a new table property named `idle_timeout_with_auto_delete` was
  instead defined as a new possible value for `pna_idle_timeout`.
* Removed unused type `PNA_PacketPath_t` from `pna.p4`.
* Modifications to intrinsic metadata fields: Removed `direction`,
  `pass`, `loopedback`.  Added `recirculated`.  For something similar
  to `direction` added extern functions `is_host_port` and
  `is_net_port` that can be called to determine whether a port is a
  host port or network port.  This also motivated a change to the
  parameters of extern function `SelectByDirection`.
* Many minor changes in example P4 programs to keep up with changes to
  the specification and pna.p4 include file.

# Appendix: Open Issues

# Appendix: Rationale for design {#appendix-rationale}

## Why a common pipeline, instead of separate pipelines for each direction?

~Begin TBD
Andy can write this one.  Basic reasons are summarized in
existing slides.
~End TBD

## Is it inefficient to have the `MainParser` redo work? {#appendix-reparsing}

If the only changes made by the decryption fixed function block in the network-to-host
direction were to decrypt parts of the packet that were previously
encrypted, but everything before the first decrypted byte remained
exactly the same, then it seems like it is a waste of effort that the
main parser starts parsing the packet over again from the beginning.

It is true that an IPsec decryption inline extern is unlikely to
change an Ethernet header at the beginning of the packet, but it does
seem likely that it could make the following kinds of changes to parts
of the packet before the first decrypted byte:

- Remove headers: If the received packet was IPsec tunnel mode, it
  might be useful if the inline extern removes the outer IP header,
  since it was added to the packet at the point of IPsec encryption.
  The software sending the packet (before IPsec encryption occurred)
  did not create that header, and the corresponding layer of software
  receiving the decrypted packet does not want to see such
  IPsec-specific headers.
- Modify headers: If the received packet was IPsec transport mode, it
  might be useful if the IP header whose protocol was equal to the
  standard numbers for AH or ESP was changed to be the next header
  after the AH and ESP headers are removed by the inline extern.
  Again, what an IPsec decryption block does might be useful to make
  similar to what the IPsec layer of software does in a software IP
  stack. The layer of software processing the decrypted packet should
  see what the last layer of software sent before it was encrypted.

If any or all of the above are true of the decryption fixed function block's
changes to the packet, then it seems that the only way you could save
the main parser some work is to somehow encode the results of the earlier
parser invocation, and also undo those results for any headers that were modified
in the decryption fixed function block. Then you would also need the main parser to be
able to start from one of multiple possible states in the parser state
machine, and continue from there.

That is all possible to do, but it seems like an awkward thing to
expose to a P4 developer, e.g. should we require them to write a main
parser that has a start state that immediately branches one of 7 ways
based upon some intermediate state that the previous invocation of the parser reached, as
modified by the decryption fixed function block if it modified or removed some of those
headers?

A NIC implementation might do such things, and it seems likely an
implementation might use some of the techniques mentioned in the
previous paragraph, but hidden from the P4 developer. The proposed PNA
design should not prevent this, if an implementer is willing to go to
that effort.

# Appendix: Packet path figures {#appendix-packet-path-figures}

## From network, sent to host

## From network, sent to host, with mirror copy to different host

## From host, to network

## From host, to network, with mirror copy to a different host

## From host, to host

## From port, to port

# Appendix: Packet ordering {#appendix-packet-ordering}