This example measures the latency of IPC transmissions between two applications. We compare the latency of iceoryx with message queues and unix domain sockets.
The measurement is carried out with several payload sizes. Round trips are performed for each payload size, using either the default setting or the provided command line parameter for the number of round trips to do. The measured time is just allocating/releasing memory and the time to send the data. The construction and writing of the payload is not part of the measurement.
At the end of the benchmark, the average latency for each payload size is printed.
Create three terminals and run one command in each of them.
The order is first the RouDi daemon, then iceperf-laurel which is the leader in this setup
and then iceperf-hardy for doing the ping pong measurements with iceperf-laurel.
You can set the number of measurement iterations (number of roundtrips) with a command line paramter
of iceperf-laurel (e.g. ./iceperf-laurel 100000)
# If installed and available in PATH environment variable
iox-roudi
# If build from scratch with script in tools
$ICEORYX_ROOT/build/install/prefix/bin/iox-roudi
build/iceoryx_examples/iceperf/iceperf-laurel
build/iceoryx_examples/iceperf/iceperf-hardyIf you would like to test only the C++ API or the C API you can start iceperf-laurel and
iceperf-hardy with the parameter cpp-api or c-api.
build/iceoryx_examples/iceperf/iceperf-laurel 100000 cpp-api
build/iceoryx_examples/iceperf/iceperf-hardy cpp-api
## Expected output
The numbers will differ depending on parameters and the performance of the hardware.
Which technologies are measured depends on the operating system (e.g. no message queue on MacOS).
Here an example output with Ubuntu 18.04 on Intel(R) Xeon(R) CPU E3-1505M v5 @ 2.80GHz.
<!-- @todo Replace this with asciinema recording before v1.0-->
### iceperf-laurel application
****** MESSAGE QUEUE ********
Waiting for: subscription, subscriber [ success ]
Measurement for: 1 kB, 2 kB, 4 kB, 8 kB, 16 kB, 32 kB, 64 kB, 128 kB, 256 kB,
512 kB, 1024 kB, 2048 kB, 4096 kB,
Waiting for: unsubscribe [ finished ]
#### Measurement Result ####
100000 round trips for each payload.
| Payload Size [kB] | Average Latency [µs] |
|------------------:|---------------------:|
| 1 | 3.1 |
| 2 | 3.2 |
| 4 | 3.8 |
| 8 | 5.2 |
| 16 | 7.7 |
| 32 | 13 |
| 64 | 23 |
| 128 | 43 |
| 256 | 81 |
| 512 | 1.6e+02 |
| 1024 | 3e+02 |
| 2048 | 5.9e+02 |
| 4096 | 1.2e+03 |
Finished!
****** UNIX DOMAIN SOCKET ********
Waiting for: subscription, subscriber [ success ]
Measurement for: 1 kB, 2 kB, 4 kB, 8 kB, 16 kB, 32 kB, 64 kB, 128 kB, 256 kB,
512 kB, 1024 kB, 2048 kB, 4096 kB,
Waiting for: unsubscribe [ finished ]
#### Measurement Result ####
100000 round trips for each payload.
| Payload Size [kB] | Average Latency [µs] |
|------------------:|---------------------:|
| 1 | 4.3 |
| 2 | 4.3 |
| 4 | 4.6 |
| 8 | 6 |
| 16 | 8.7 |
| 32 | 14 |
| 64 | 27 |
| 128 | 53 |
| 256 | 1.1e+02 |
| 512 | 2.1e+02 |
| 1024 | 4.2e+02 |
| 2048 | 8.4e+02 |
| 4096 | 1.7e+03 |
Finished!
****** ICEORYX ********
Waiting for: subscription, subscriber [ success ]
Measurement for: 1 kB, 2 kB, 4 kB, 8 kB, 16 kB, 32 kB, 64 kB, 128 kB, 256 kB,
512 kB, 1024 kB, 2048 kB, 4096 kB,
Waiting for: unsubscribe [ finished ]
#### Measurement Result ####
100000 round trips for each payload.
| Payload Size [kB] | Average Latency [µs] |
|------------------:|---------------------:|
| 1 | 0.73 |
| 2 | 0.58 |
| 4 | 0.61 |
| 8 | 0.61 |
| 16 | 0.59 |
| 32 | 0.62 |
| 64 | 0.6 |
| 128 | 0.58 |
| 256 | 0.61 |
| 512 | 0.61 |
| 1024 | 0.58 |
| 2048 | 0.61 |
| 4096 | 0.61 |
Finished!
****** ICEORYX C API ********
Waiting for: subscription, subscriber [ success ]
Measurement for: 1 kB, 2 kB, 4 kB, 8 kB, 16 kB, 32 kB, 64 kB, 128 kB, 256 kB,
512 kB, 1024 kB, 2048 kB, 4096 kB,
Waiting for: unsubscribe [ finished ]
#### Measurement Result ####
100000 round trips for each payload.
| Payload Size [kB] | Average Latency [µs] |
|------------------:|---------------------:|
| 1 | 0.73 |
| 2 | 0.58 |
| 4 | 0.61 |
| 8 | 0.61 |
| 16 | 0.59 |
| 32 | 0.62 |
| 64 | 0.6 |
| 128 | 0.58 |
| 256 | 0.61 |
| 512 | 0.61 |
| 1024 | 0.58 |
| 2048 | 0.61 |
| 4096 | 0.61 |
Finished!
### iceperf-hardy application
****** MESSAGE QUEUE ********
registering with the leader, if no leader this will crash with a message queue error now
****** UNIX DOMAIN SOCKET ********
registering with the leader, if no leader this will crash with a socket error now
****** ICEORYX ********
Waiting for: subscription, subscriber [ success ]
Waiting for: unsubscribe [ finished ]
****** ICEORYX C API ********
Waiting for: subscription, subscriber [ success ]
Waiting for: unsubscribe [ finished ]
## Code walkthrough
Here we roughly describe the setup for performing the measurements. Things like initialization, sending and receiving of data are technology specific and can be found in the respective files (e.g. uds.cpp for
unix domain socket). Our focus here is on the abstraction layer on top which allows us or you to add new IPC technologies to extend and compare them.
### iceperf-laurel application
Besides includes for the different IPC technologies, the topic_data.hpp file is included which contains the PerTopic struct that is used to transfer some information between the applications. Independent of the real payload size, this struct is used as some kind of header in each transferred sample.
```cpp
struct PerfTopic
{
uint32_t payloadSize{0};
uint32_t subPackets{0};
bool run{true};
};With payloadSize as the payload size used for the current measurement. In case it is not possible to transfer the payloadSize with a single data transfer (e.g. OS limit for the payload of a single socket send), the payload is divided into several sub-packets. This is indicated with subPackets. The run flag is used to shutdown iceperf-hardy at the end of the benchmark.
Let's set some constants to prevent magic values. The default number of round trips is set and names for the communication resources that are used.
constexpr int64_t NUMBER_OF_ROUNDTRIPS{10000};
constexpr char APP_NAME[] = "laurel";
constexpr char PUBLISHER[] = "Laurel";
constexpr char SUBSCRIBER[] = "Hardy";The leaderDo() function executes a measurement for the provided IPC technology and number of round trips. For being able to always perform the same steps and avoiding code duplications, we use a base class with technology independent functionality and the technology has to implement the technology dependent part.
void leaderDo(IcePerfBase& ipcTechnology, int64_t numRoundtrips)
{
ipcTechnology.initLeader();
std::vector<double> latencyInMicroSeconds;
const std::vector<uint32_t> payloadSizesInKB{1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096};
std::cout << "Measurement for: ";
for (const auto payloadSizeInKB : payloadSizesInKB)
{
std::cout << payloadSizeInKB << " kB, " << std::flush;
ipcTechnology.prePingPongLeader(payloadSizeInBytes);
auto latency = ipcTechnology.pingPongLeader(numRoundtrips);
latencyInMicroSeconds.push_back(latency);
ipcTechnology.postPingPongLeader();
}
std::cout << std::endl;
ipcTechnology.releaseFollower();
ipcTechnology.shutdown();
std::cout << std::endl;
std::cout << "#### Measurement Result ####" << std::endl;
std::cout << numRoundtrips << " round trips for each payload." << std::endl;
std::cout << std::endl;
std::cout << "| Payload Size [kB] | Average Latency [µs] |" << std::endl;
std::cout << "|------------------:|---------------------:|" << std::endl;
for (size_t i = 0; i < latencyInMicroSeconds.size(); ++i)
{
std::cout << "| " << std::setw(17) << payloadSizesInKB.at(i) << " | " << std::setw(20) << std::setprecision(2)
<< latencyInMicroSeconds.at(i) << " |" << std::endl;
}
std::cout << std::endl;
std::cout << "Finished!" << std::endl;
}Initialization is different for each IPC technology. Here we have to create sockets, message queues or iceoryx publisher and subscriber. With ipcTechnology.initLeader() we are setting up these resources on the leader side. After the definition of the different payload sizes to use, we execute a single round trip measurement for each individual payload size. The leader has to orchestrate the whole process and has a pre and post step for each ping pong round trip measurement. ipcTechnology.prePingPongLeader() sets the payload size for the upcoming measurement. ipcTechnology.pingPongLeader(numRoundtrips) then does the ping pong between leader and follower and returns the time it took to do the provided number of round trips. After the measurements were done for all the different payload sizes, ipcTechnology.releaseFollower() releases the follower that is not aware of things like how many payload sizes are considered. After cleaning up the communication resources with ipcTechnology.shutdown() the results are printed.
In the main() method we create instances for the different IPC technologies we want to compare. Each one is implemented in an own class and implements the pure virtual functions provided with the IcePerfBase class
iceperf-laurel, the leader in this setup, takes the number of round trips to perform for each payload size as command line parameter. We check if one was provided and take this or otherwise use the default one. The higher the number of round trips, the more accurate the measurements will be but you have to consider that an iceperf run can take quite a long time then.
uint64_t numRoundtrips = NUMBER_OF_ROUNDTRIPS;
if (argc > 1)
{
if (!iox::cxx::convert::fromString(argv[1], numRoundtrips))
{
std::cout << "first parameter must be the number of roundtrips" << std::endl;
exit(1);
}
}Now we can create an object for each IPC technology that we want to evaluate and call the leaderDo() function. The naming conventions for the different technologies differ, therefore we do some prefixing if necessary
if (benchmark == Benchmarks::ALL)
{
#ifndef __APPLE__
std::cout << std::endl << "****** MESSAGE QUEUE ********" << std::endl;
MQ mq("/" + std::string(PUBLISHER), "/" + std::string(SUBSCRIBER));
leaderDo(mq, numRoundtrips);
#endif
std::cout << std::endl << "****** UNIX DOMAIN SOCKET ********" << std::endl;
UDS uds("/tmp/" + std::string(PUBLISHER), "/tmp/" + std::string(SUBSCRIBER));
leaderDo(uds, numRoundtrips);
}
std::cout << std::endl << "****** ICEORYX ********" << std::endl;
iox::runtime::PoshRuntime::initRuntime(APP_NAME); // runtime for registering with the RouDi daemon
Iceoryx iceoryx(PUBLISHER, SUBSCRIBER);
leaderDo(iceoryx, numRoundtrips);The main() for iceperf-hardy is similar to iceperf-laurel, only the SUBSCRIBER and PUBLISHER names changed to the other way round. The followerDo() function is much simpler as the follower only reacts and does not the control. Besides ipcTechnology.initFollower() and ipcTechnology.shutdown() all the functionality to do the ping pong for different payload sizes is done in ipcTechnology.pingPongFollower()
void followerDo(IcePerfBase& ipcTechnology)
{
ipcTechnology.initFollower();
ipcTechnology.pingPongFollower();
ipcTechnology.shutdown();
}