Manager to NF Message Passing (#166)

This brings an interface to pass messages from the manager to NFs.  Messages are described by the `onvm_nf_msg` struct by having an ONVM manager defined type and abstract data pointer.  These messages can be created with the manager function `onvm_nf_send_msg`.  Each NF now has a message queue alongside their preexisting Rx and Tx queues.  NFs check for messages in their critical path, and can use the `packet_handler` interface, or the interface where they have direct access to their own rings.

Commit log:

* Implement basic mgr to NF Messaging

We want the manager to be able to send messages to NFs. This commit:
 - Defines a simple message struct (with type, destination, and data)
 - Creates a mempool/ring for messages to be passed to NFs
 - Adds an interface for the manage to enqueue a message
 - nflib maps the ring into its space and can read from it

* Use one msg ring per NF

It is impossible to use one single message queue with the DPDK rings.
One the consumer side, we'd want each NF to "peek" at the head of the
queue and only dequeue if it is the recipient of that message. The DPDK
ring library doesn't support "peek", and that would approach would also
leave the messages vulnerable to denial-of-service by other NFs that
never dequeue their messages. Probably not a decision we want to use...

Instead, create one ring per NF, like we do TX and RX queues. Each NF
can then look up its own ring and use it without fear of inteferance.
Plus, we can simply use `rte_ring_count` to see if there are any
available messages to be processed.

* Read and process messages in nflib

* Fix comment, typo fixes

* Refactor nflib packet processing loop

I did a few things here to make the code simpler:
 - Typedef'd the pkt_handler function signature
 - Extracted the packet and message checking code into their own
   two functions. This means there aren't any confusing semantics
   with their prior use of continue (e.g. even if there are no
   packets to process, then we still want to check for messages)

* Increase the size of the message mempool

This lets us allocate enough msg structs to fill all the NF rings.
This should be an excessive amount...

* Always inline the new packet processing functions

I have them split out for cleanliness, but we can avoid the
performance regression of two function calls each packet
handler loop by doing this.

* Bump msg pool size, add log message on alloc fail

onvm/onvm_mgr/onvm_init.c

@@ -60,6 +60,7 @@ struct port_info *ports = NULL;
 
 struct rte_mempool *pktmbuf_pool;
 struct rte_mempool *nf_info_pool;
+struct rte_mempool *nf_msg_pool;
 struct rte_ring *nf_info_queue;
 uint16_t **services;
 uint16_t *nf_per_service_count;
@@ -73,6 +74,7 @@ struct onvm_service_chain **default_sc_p;
 
 static int init_mbuf_pools(void);
 static int init_client_info_pool(void);
+static int init_nf_msg_pool(void);
 static int init_port(uint8_t port_num);
 static int init_shm_rings(void);
 static int init_info_queue(void);
@@ -128,6 +130,12 @@ init(int argc, char *argv[]) {
                 rte_exit(EXIT_FAILURE, "Cannot create client info mbuf pool: %s\n", rte_strerror(rte_errno));
         }
 
+        /* initialise pool for NF messages */
+        retval = init_nf_msg_pool();
+        if (retval != 0) {
+                rte_exit(EXIT_FAILURE, "Cannot create nf message pool: %s\n", rte_strerror(rte_errno));
+        }
+
 	/* now initialise the ports we will use */
         for (i = 0; i < ports->num_ports; i++) {
                 retval = init_port(ports->id[i]);
@@ -193,6 +201,22 @@ init_mbuf_pools(void) {
         return (pktmbuf_pool == NULL); /* 0  on success */
 }
 
+/**
+ * Set up a mempool to store nf_msg structs
+ */
+static int
+init_nf_msg_pool(void)
+{
+        /* don't pass single-producer/single-consumer flags to mbuf
+         * create as it seems faster to use a cache instead */
+        printf("Creating mbuf pool '%s' ...\n", _NF_MSG_POOL_NAME);
+        nf_msg_pool = rte_mempool_create(_NF_MSG_POOL_NAME, MAX_CLIENTS * CLIENT_MSG_QUEUE_SIZE,
+                        NF_INFO_SIZE, NF_MSG_CACHE_SIZE,
+                        0, NULL, NULL, NULL, NULL, rte_socket_id(), NO_FLAGS);
+
+        return (nf_msg_pool == NULL); /* 0 on success */
+}
+
 /**
  * Set up a mempool to store nf_info structs
  */
@@ -284,7 +308,9 @@ init_shm_rings(void) {
         unsigned socket_id;
         const char * rq_name;
         const char * tq_name;
+        const char * msg_q_name;
         const unsigned ringsize = CLIENT_QUEUE_RINGSIZE;
+        const unsigned msgringsize = CLIENT_MSG_QUEUE_SIZE;
 
         // use calloc since we allocate for all possible clients
         // ensure that all fields are init to 0 to avoid reading garbage
@@ -310,19 +336,26 @@ init_shm_rings(void) {
                 socket_id = rte_socket_id();
                 rq_name = get_rx_queue_name(i);
                 tq_name = get_tx_queue_name(i);
+                msg_q_name = get_msg_queue_name(i);
                 clients[i].instance_id = i;
                 clients[i].rx_q = rte_ring_create(rq_name,
                                 ringsize, socket_id,
                                 RING_F_SC_DEQ);                 /* multi prod, single cons */
                 clients[i].tx_q = rte_ring_create(tq_name,
                                 ringsize, socket_id,
                                 RING_F_SC_DEQ);                 /* multi prod, single cons */
+                clients[i].msg_q = rte_ring_create(msg_q_name,
+                                msgringsize, socket_id,
+                                RING_F_SC_DEQ);                 /* multi prod, single cons */
 
                 if (clients[i].rx_q == NULL)
                         rte_exit(EXIT_FAILURE, "Cannot create rx ring queue for client %u\n", i);
 
                 if (clients[i].tx_q == NULL)
                         rte_exit(EXIT_FAILURE, "Cannot create tx ring queue for client %u\n", i);
+
+                if (clients[i].msg_q == NULL)
+                        rte_exit(EXIT_FAILURE, "Cannot create msg queue for client %u\n", i);
         }
         return 0;
 }

onvm/onvm_mgr/onvm_init.h

@@ -90,9 +90,13 @@
 #define NF_INFO_SIZE sizeof(struct onvm_nf_info)
 #define NF_INFO_CACHE 8
 
+#define NF_MSG_SIZE sizeof(struct onvm_nf_msg)
+#define NF_MSG_CACHE_SIZE 8
+
 #define RTE_MP_RX_DESC_DEFAULT 512
 #define RTE_MP_TX_DESC_DEFAULT 512
 #define CLIENT_QUEUE_RINGSIZE 128
+#define CLIENT_MSG_QUEUE_SIZE 128
 
 #define NO_FLAGS 0
 
@@ -109,6 +113,7 @@
 struct client {
         struct rte_ring *rx_q;
         struct rte_ring *tx_q;
+        struct rte_ring *msg_q;
         struct onvm_nf_info *info;
         uint16_t instance_id;
         /* these stats hold how many packets the client will actually receive,
@@ -164,12 +169,14 @@ struct port_info {
 
 extern struct client *clients;
 
+/* NF to Manager data flow */
 extern struct rte_ring *nf_info_queue;
 
 /* the shared port information: port numbers, rx and tx stats etc. */
 extern struct port_info *ports;
 
 extern struct rte_mempool *pktmbuf_pool;
+extern struct rte_mempool *nf_msg_pool;
 extern uint16_t num_clients;
 extern uint16_t num_services;
 extern uint16_t default_service;

onvm/onvm_mgr/onvm_nf.c

@@ -126,6 +126,24 @@ onvm_nf_check_status(void) {
 }
 
 
+int
+onvm_nf_send_msg(uint16_t dest, uint8_t msg_type, void *msg_data) {
+        int ret;
+        struct onvm_nf_msg *msg;
+
+        ret = rte_mempool_get(nf_msg_pool, (void**)(&msg));
+        if (ret != 0) {
+                RTE_LOG(INFO, APP, "Oh the huge manatee! Unable to allocate msg from pool :(\n");
+                return ret;
+        }
+
+        msg->msg_type = msg_type;
+        msg->msg_data = msg_data;
+
+        return rte_ring_sp_enqueue(clients[dest].msg_q, (void*)msg);
+}
+
+
 inline uint16_t
 onvm_nf_service_to_nf_map(uint16_t service_id, struct rte_mbuf *pkt) {
         uint16_t num_nfs_available = nf_per_service_count[service_id];

onvm/onvm_mgr/onvm_nf.h

@@ -87,6 +87,19 @@ void
 onvm_nf_check_status(void);
 
 
+/*
+ * Interface to send a message to a certain NF.
+ *
+ * Input  : The destination NF instance ID, a constant denoting the message type
+ *          (see onvm_nflib/onvm_msg_common.h), and a pointer to a data argument.
+ *          The data argument should be allocated in the hugepage region (so it can
+ *          be shared), i.e. using rte_malloc
+ * Output : 0 if the message was successfully sent, -1 otherwise
+ */
+int
+onvm_nf_send_msg(uint16_t dest, uint8_t msg_type, void *msg_data);
+
+
 /*
  * Interface giving a NF for a specific server id, depending on the flow.
  *

onvm/onvm_nflib/onvm_common.h

@@ -45,6 +45,8 @@
 #include <rte_mbuf.h>
 #include <stdint.h>
 
+#include "onvm_msg_common.h"
+
 #define ONVM_MAX_CHAIN_LENGTH 4   // the maximum chain length
 #define MAX_CLIENTS 16            // total number of NFs allowed
 #define MAX_SERVICES 16           // total number of unique services allowed
@@ -124,10 +126,12 @@ struct onvm_service_chain {
 #define MZ_SCP_INFO "MProc_scp_info"
 #define MZ_FTP_INFO "MProc_ftp_info"
 
-/* common names for NF states */
 #define _NF_QUEUE_NAME "NF_INFO_QUEUE"
+#define _NF_MSG_QUEUE_NAME "NF_%u_MSG_QUEUE"
 #define _NF_MEMPOOL_NAME "NF_INFO_MEMPOOL"
+#define _NF_MSG_POOL_NAME "NF_MSG_MEMPOOL"
 
+/* common names for NF states */
 #define NF_WAITING_FOR_ID 0     // First step in startup process, doesn't have ID confirmed by manager yet
 #define NF_STARTING 1           // When a NF is in the startup process and already has an id
 #define NF_RUNNING 2            // Running normally
@@ -164,6 +168,20 @@ get_tx_queue_name(unsigned id) {
         return buffer;
 }
 
+/*
+ * Given the name template above, get the mgr -> NF msg queue name
+ */
+static inline const char *
+get_msg_queue_name(unsigned id) {
+        /* buffer for return value. Size calculated by %u being replaced
+         * by maximum 3 digits (plus an extra byte for safety) */
+        static char buffer[sizeof(_NF_MSG_QUEUE_NAME) + 2];
+
+        snprintf(buffer, sizeof(buffer) - 1, _NF_MSG_QUEUE_NAME, id);
+        return buffer;
+
+}
+
 #define RTE_LOGTYPE_APP RTE_LOGTYPE_USER1
 
 #endif  // _COMMON_H_

onvm/onvm_nflib/onvm_msg_common.h

@@ -0,0 +1,55 @@
+/*********************************************************************
+ *                     openNetVM
+ *              https://sdnfv.github.io
+ *
+ *   BSD LICENSE
+ *
+ *   Copyright(c)
+ *            2015-2016 George Washington University
+ *            2015-2016 University of California Riverside
+ *            2010-2014 Intel Corporation
+ *   All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * The name of the author may not be used to endorse or promote
+ *       products derived from this software without specific prior
+ *       written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * onvm_msg_common.h - Shared structures relating to message passing
+                       between the manager and NFs
+ ********************************************************************/
+
+#ifndef _MSG_COMMON_H_
+#define _MSG_COMMON_H_
+
+#include <stdint.h>
+
+#define MSG_NOOP 0
+
+struct onvm_nf_msg {
+        uint8_t msg_type; /* Constant saying what type of message is */
+        void *msg_data; /* These should be rte_malloc'd so they're stored in hugepages */
+};
+
+#endif