Adding NDN-DPDK Forwarder Support

[justincpresley]

Hey! So I have a use case to support the NDN-DPDK forwarder.
As you may know, ndn-hydra is an distributed repository built using this library. It is soon going to be deployed (and actively used) on a high-performance environment. However, our forwarder is a major bottleneck in initial testing.

I did look into how tightly coupled NFD is with this library and it looks to be extensive. However, if NDN-DPDK becomes supported, I am able to advertise NDN to a broad audience. This also would allow NDN (Hydra) to be actively used to solve problems which would popularize NDN.

Simply said, NDN-DPDK forwarder support would greatly help our application in solving the problem it is trying to solve and as a major user of this library, I would greatly appreciate it if this was included.

[zjkmxy]

Hello. I think the only problem is PIT token. python-ndn does not memorize incoming Interests so it does not apply PIT token when a data is sent. This is hard to fix.

[yoursunny]

For PIT token, it isn't necessary for the library to memorize incoming Interests in a PIT.
Producer handler should be given a reply function, which memorizes the current Interest including its PIT token.

@app.route('/P')
def process_interest(interest: Interest, reply: Callable[[Data|Nack|None], None]):
  # to reply with Data
  reply(data)

  # to reply with Nack
  reply(nack)

  # to not reply i.e. cause a timeout
  reply(None)

• You probably want to create a new decorator instead of adding more flags to @app.route that is already quite complicated.
• The reply function can only be called once; otherwise, it triggers assertion failure.
• For reply(data) and reply(nack), the library checks that the Data/Nack indeed matches the Interest; otherwise, it triggers assertion failure and does not send the packet. This is helpful for catching mistakes like forgetting to set FreshnessPeriod when incoming Interest has MustBeFresh.
• For reply(nack), if it is known that the uplink does not support Nack (e.g. it's recognized as YaNFD), the library may silently drop the packet instead of sending it.
• The reply(None) call isn't required, but it makes the intention explicit and prevents calling the reply function again. The application may opt-in a strict mode where every Interest must be reply()-ed, which helps detecting coroutine memory leaks; in this case, reply(None) would be required.
• The reply function should be called within InterestLifetime; otherwise, the packet is not sent since the downstream node would not accept it, and a warning is logged.

NDNph library has a similar design, but does not have all the features listed above.
https://github.com/yoursunny/NDNph/blob/22e703555503c2d534774064d9bb444ee8011cbf/src/ndnph/app/ping-server.hpp#L36

[zjkmxy]

Thank you for your idea. Will implement this when I have time.

[yoursunny]

Proper NDN-DPDK support has 3 ingredients:

• PIT token
• memif transport
• management protocol

For memif transport, build a Python extension based on libmemif C library.
This should be a separate library that is an optional dependency of python-ndn.
I have a Node.js memif library for reference.

For management protocol, NDN-DPDK natively accepts GraphQL commands, but some deployments use Multiverse that uses a different protocol.
NDN-DPDK does not have a listening socket; both face creation and prefix announcement need to go through an out-of-band management protocol.
See NDNts @ndn/dpdkmgmt for reference, but the GraphQL commands may still change.

[zjkmxy]

This involves multiple changes in NDNApp. I think probably I should refactor the whole class (like NDNAppv2).
Since this takes time and is not of high priority, let me delay this work for now.