Performance benchmarking refactor (ros2#594)

* Refactoring of rosbag2 performance package:
- renamed since now it no longer benchmarks writer only
- generalized byte_producer so that it uses a callback instead of queue, so it can be reused in publisher scheme

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

* Benchmark publishers based on yaml configuration
- can specify multiple groups of publishers (see attached example yaml)
- reuses byte producer

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

* Applying configured QoS settings for publishers.
Also included in yaml example.

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

* linters

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

* Towards common configuration
- separating out common structures
- utility class for common parameter parsing

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

* Barebone launchfile for benchmarks.

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* writer benchmark adapted to yaml file and publisher groups

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

* refactored result writing and bag parameters

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

* linters

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

* Launchfile for benchmarks

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* Change storage config file from non optimized to resilient

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* Max bag size for benchmark launchfile. Launchfile refactor.

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* Copy yaml configs after benchmark is finished.

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* Benchmark results csv file extended

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* added disclaimer about random data and compression

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

* Report gen tool for benchmarks

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* Benchmarks out dir name changed

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* results writer node

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

* documentation

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

* Transport and transportless in launchfile

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* Benchmark launchfile refactor

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* Wait for rosbag listening in benchmark launchfile

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* Uncrustify and some comments for benchmarking tools

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* Added new producers config for benchmarks

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* Missing parameters in transport benchmark

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

added comment in storage_optimized.yaml

* Missing rosbag record parameters in transport benchmark

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* Wait for subscriptions parameter in producers config

Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

* moved utils code from header to source

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

* changed compiler shortcut uint to unsigned int (should fix Windows build)

Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>

Co-authored-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>

rosbag2_performance/rosbag2_performance_writer_benchmarking/CHANGELOG.rst → rosbag2_performance/rosbag2_performance_benchmarking/CHANGELOG.rst

@@ -1,5 +1,5 @@
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Changelog for package rosbag2_performance_writer_benchmarking
+Changelog for package rosbag2_performance_benchmarking
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 0.5.0 (2020-12-02)

rosbag2_performance/rosbag2_performance_writer_benchmarking/CMakeLists.txt → rosbag2_performance/rosbag2_performance_benchmarking/CMakeLists.txt

@@ -1,5 +1,5 @@
 cmake_minimum_required(VERSION 3.5)
-project(rosbag2_performance_writer_benchmarking)
+project(rosbag2_performance_benchmarking)
 
 # Default to C++14
 if(NOT CMAKE_CXX_STANDARD)
@@ -22,23 +22,68 @@ if(BUILD_ROSBAG2_BENCHMARKS)
   find_package(std_msgs REQUIRED)
   find_package(yaml_cpp_vendor REQUIRED)
 
-  add_executable(writer_benchmark src/writer_benchmark.cpp src/main.cpp)
+  add_executable(writer_benchmark
+    src/config_utils.cpp
+    src/result_utils.cpp
+    src/writer_benchmark.cpp)
+
+  add_executable(benchmark_publishers
+    src/benchmark_publishers.cpp
+    src/config_utils.cpp)
+
+  add_executable(results_writer
+    src/config_utils.cpp
+    src/result_utils.cpp
+    src/results_writer.cpp)
+
   ament_target_dependencies(writer_benchmark
     rclcpp
     std_msgs
     rosbag2_compression
     rosbag2_cpp
+    rosbag2_storage
+    yaml_cpp_vendor
+  )
+
+  ament_target_dependencies(benchmark_publishers
+    rclcpp
+    rosbag2_storage
+    std_msgs
     yaml_cpp_vendor
   )
+
+  ament_target_dependencies(results_writer
+    rclcpp
+    rosbag2_storage
+  )
+
   target_include_directories(writer_benchmark
     PUBLIC
     $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
     $<INSTALL_INTERFACE:include>
   )
 
-  install(TARGETS writer_benchmark
+  target_include_directories(benchmark_publishers
+    PUBLIC
+    $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
+    $<INSTALL_INTERFACE:include>
+  )
+
+  target_include_directories(results_writer
+    PUBLIC
+    $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
+    $<INSTALL_INTERFACE:include>
+  )
+
+  install(TARGETS writer_benchmark benchmark_publishers results_writer
     DESTINATION lib/${PROJECT_NAME})
 
+  install(DIRECTORY
+    launch
+    config
+    DESTINATION share/${PROJECT_NAME}
+  )
+
   if(BUILD_TESTING)
     find_package(ament_lint_auto REQUIRED)
     ament_lint_auto_find_test_dependencies()

rosbag2_performance/rosbag2_performance_benchmarking/README.md

@@ -0,0 +1,76 @@
+# Rosbag2 writer benchmarking
+
+The primary package to test performance of the rosbag2.
+
+## How it works
+
+Use `benchmark_launch.py` launchfile to run an entire set of benchmarks.
+
+Launchfile requires two arguments:
+
+- `benchmark` - provides benchmark description (how many repetitions, cache size, database configuration etc.),
+- `producers` - provides producers description (how many publisher/producer instances, frequency, messages size etc.)
+
+Templates for these configuration files are in `config` directory of this package.
+
+To run test benchmark (with `test.yaml` and `mixed_110Mbs.yaml`):
+
+```bash
+ros2 launch rosbag2_performance_benchmarking benchmark_launch.py benchmark:=`ros2 pkg prefix rosbag2_performance_benchmarking`/share/rosbag2_performance_benchmarking/config/benchmarks/test.yaml producers:=`ros2 pkg prefix rosbag2_performance_benchmarking`/share/rosbag2_performance_benchmarking/config/producers/mixed_110Mbs.yaml
+```
+
+The summary of benchmark goes into result file described in benchmark config: `<db_root_folder>/<BENCHMARK_NAME>/summary_result_file` where `BENCHMARK_NAME` is a name generated from config names, transport type and timestamp.
+
+For human friendly output, a postprocess report generation tool can be used. Launch it with benchmark result directory as an `-i` argument (directory with `summary_result_file` file):
+
+```bash
+scripts/report_gen.py -i <BENCHMARK_RESULT_DIR>
+```
+#### Binaries
+
+These are used in the launch file:
+
+*  `benchmark_publishers` - runs publishers based on provided parameters. Used when `no_transport` parameter is set to `False`;
+*  `writer_benchmark` - runs storage-only benchmarking, mimicking subscription queues but using no transport whatsoever. Used when `no_transport` parameter is set to `True`.
+*  `results_writer` - based on provider parameters, write results (percentage of recorded messages) after recording. One of the parameters is the
+storage uri, which is used to read the bag metadata file.
+
+#### Compression
+
+Note that while you can opt to select compression for benchmarking, the generated data is random so it is likely not representative for this specific case. To publish non-random data, you need to modify the ByteProducer.
+
+## Building
+
+To build the package in the rosbag2 build process, make sure to turn `BUILD_ROSBAG2_BENCHMARKS` flag on (e.g. `colcon build --cmake-args -DBUILD_ROSBAG2_BENCHMARKS=1`)
+
+If you already built rosbag2, you can use `packages-select` option to build benchmarks.
+Example: `colcon build --packages-select rosbag2_performance_benchmarking --cmake-args -DBUILD_ROSBAG2_BENCHMARKS=1`.
+
+## General knowledge: I/O benchmarking
+
+#### Background: benchmarking disk writes on your system
+
+It might be useful to first understand what limitation your disk poses to the throughput of data recording.
+Performance of bag write can't be higher over extended period of time (you can only use as much memory).
+
+**Using dd command**
+
+`dd if=/dev/zero of=/tmp/output conv=fdatasync bs=384k count=1k; rm -f /tmp/output`
+
+This method is not great for benchmarking the disk but an easy way to start since it requires no dependencies.
+This will write zeros to the /tmp/output file with block size 384k, 1000 blocks, ends when write finishes.
+Make sure to benchmark the disk which your bags will be written to (check your mount points and change “/tmp/output” to another path if needed).
+Note: this depends on parameters used and whatever else is running on your system but can give you a ballpark figure when ran several times.
+
+**Using fio**
+
+For more sophisticated & accurate benchmarks, see the `fio` command. An example for big data blocks is: `fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=write --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting`.
+
+#### Profiling bags I/O with tools
+
+Tools that can help in I/O profiling: `sudo apt-get install iotop ioping sysstat`
+* `iotop` works similar as `top` command, but shows disk reads, writes, swaps and I/O %. Can be used at higher frequency in batch mode with specified process to deliver data that can be plotted.
+  *  Example use: `sudo iotop -h -d 0.1 -t -b -o -p <PID>` after running the bag.  
+* `ioping` can be used to check latency of requests to device
+* `strace` can help determine syscalls associated with the bottleneck.
+  *  Example use: `strace -c ros2 bag record /image --max-cache-size 10 -o ./tmp`. You will see a report after finishing recording with Ctrl-C.

rosbag2_performance/rosbag2_performance_benchmarking/config/benchmarks/test.yaml

@@ -0,0 +1,16 @@
+rosbag2_performance_benchmarking:
+  benchmark_node:
+    ros__parameters:
+      benchmark:
+        summary_result_file:  "results.csv"
+        db_root_folder:       "rosbag2_performance_test_results"
+        repeat_each:          2     # How many times to run each configurations (to average results)
+        no_transport:         True  # Whether to run storage-only or end-to-end (including transport) benchmark
+        preserve_bags:        False # Whether to leave bag files after experiment (and between runs). Some configurations can take lots of space!
+        parameters:                 # Each combination of parameters in this section will be benchmarked
+          max_cache_size:         [10000000, 100000000]
+          max_bag_size:           [0]
+          compression:            ["", "zstd"]
+          compression_queue_size: [1]
+          compression_threads:    [0]
+          storage_config_file:    ["", "storage_resilient.yaml"]

rosbag2_performance/rosbag2_performance_benchmarking/config/producers/mixed_110Mbs.yaml

@@ -0,0 +1,21 @@
+rosbag2_performance_benchmarking_node:
+  ros__parameters:
+    publishers: # publisher_groups parameter needs to include all the subsequent groups 
+      publisher_groups: [ "10Mbs_many_frequent_small", "100Mbs_large" ]
+      wait_for_subscriptions: True
+      10Mbs_many_frequent_small:
+        publishers_count:   500
+        topic_root:         "benchmarking_small"
+        msg_size_bytes:     100
+        msg_count_each:     2000
+        rate_hz:            200
+      100Mbs_large:
+        publishers_count:   1
+        topic_root:         "benchmarking_large"
+        msg_size_bytes:     10000000
+        msg_count_each:     100
+        rate_hz:            10
+        qos:  # qos settings are ignored for writer only benchmarking
+          qos_depth:          5
+          qos_reliability:    "best_effort" # "reliable"
+          qos_durability:     "volatile" # "transient_local"

rosbag2_performance/rosbag2_performance_benchmarking/config/producers/mixed_110Mbs_low_pubs_count.yaml

@@ -0,0 +1,21 @@
+rosbag2_performance_benchmarking_node:
+  ros__parameters:
+    publishers: # publisher_groups parameter needs to include all the subsequent groups 
+      publisher_groups: [ "10Mbs_many_frequent_small", "100Mbs_large" ]
+      wait_for_subscriptions: True
+      10Mbs_many_frequent_small:
+        publishers_count:   100
+        topic_root:         "benchmarking_small"
+        msg_size_bytes:     500
+        msg_count_each:     2000
+        rate_hz:            200
+      100Mbs_large:
+        publishers_count:   1
+        topic_root:         "benchmarking_large"
+        msg_size_bytes:     10000000
+        msg_count_each:     100
+        rate_hz:            10
+        qos:  # qos settings are ignored for writer only benchmarking
+          qos_depth:          5
+          qos_reliability:    "best_effort" # "reliable"
+          qos_durability:     "volatile" # "transient_local"

rosbag2_performance/rosbag2_performance_benchmarking/config/storage/storage_optimized.yaml

@@ -0,0 +1,3 @@
+# note: this is the default now, using this file does not change how the storage behaves
+write:
+  pragmas: ["journal_mode = MEMORY", "synchronous = OFF"]

rosbag2_performance/rosbag2_performance_benchmarking/include/rosbag2_performance_benchmarking/bag_config.hpp

@@ -0,0 +1,29 @@
+// Copyright 2021, Robotec.ai sp. z o.o.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef ROSBAG2_PERFORMANCE_BENCHMARKING__BAG_CONFIG_HPP_
+#define ROSBAG2_PERFORMANCE_BENCHMARKING__BAG_CONFIG_HPP_
+
+#include <string>
+#include "rosbag2_storage/storage_options.hpp"
+
+struct BagConfig
+{
+  rosbag2_storage::StorageOptions storage_options;
+  std::string compression_format;
+  unsigned int compression_queue_size;
+  unsigned int compression_threads;
+};
+
+#endif  // ROSBAG2_PERFORMANCE_BENCHMARKING__BAG_CONFIG_HPP_

rosbag2_performance/rosbag2_performance_writer_benchmarking/include/rosbag2_performance_writer_benchmarking/byte_producer.hpp → rosbag2_performance/rosbag2_performance_benchmarking/include/rosbag2_performance_benchmarking/byte_producer.hpp

@@ -12,26 +12,18 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-#ifndef ROSBAG2_PERFORMANCE_WRITER_BENCHMARKING__BYTE_PRODUCER_HPP_
-#define ROSBAG2_PERFORMANCE_WRITER_BENCHMARKING__BYTE_PRODUCER_HPP_
+#ifndef ROSBAG2_PERFORMANCE_BENCHMARKING__BYTE_PRODUCER_HPP_
+#define ROSBAG2_PERFORMANCE_BENCHMARKING__BYTE_PRODUCER_HPP_
 
 #include <chrono>
 #include <memory>
 #include <thread>
-#include <vector>
+#include <functional>
 
 #include "rclcpp/utilities.hpp"
-
 #include "std_msgs/msg/byte_multi_array.hpp"
 
-#include "rosbag2_performance_writer_benchmarking/message_queue.hpp"
-
-struct ProducerConfig
-{
-  unsigned int frequency;
-  unsigned int max_count;
-  unsigned int message_size;
-};
+#include "rosbag2_performance_benchmarking/producer_config.hpp"
 
 inline auto generate_random_message(const ProducerConfig & config)
 {
@@ -49,31 +41,46 @@ inline auto generate_random_message(const ProducerConfig & config)
 class ByteProducer
 {
 public:
-  ByteProducer(const ProducerConfig & config, std::shared_ptr<ByteMessageQueue> queue)
+  using producer_initialize_function_t = std::function<void ()>;
+  using producer_callback_function_t = std::function<void (
+        std::shared_ptr<std_msgs::msg::ByteMultiArray>)>;
+
+  using producer_finalize_function_t = std::function<void ()>;
+
+  ByteProducer(
+    const ProducerConfig & config,
+    producer_initialize_function_t producer_initialize,
+    producer_callback_function_t producer_callback,
+    producer_finalize_function_t producer_finalize)
   : configuration_(config),
-    queue_(queue),
+    producer_initialize_(producer_initialize),
+    producer_callback_(producer_callback),
+    producer_finalize_(producer_finalize),
     sleep_time_(configuration_.frequency == 0 ? 1 : 1000 / configuration_.frequency),
     message_(generate_random_message(configuration_))
   {}
 
   void run()
   {
+    producer_initialize_();
     for (auto i = 0u; i < configuration_.max_count; ++i) {
       if (!rclcpp::ok()) {
         break;
       }
-      queue_->push(message_);
+      producer_callback_(message_);
       std::this_thread::sleep_for(std::chrono::milliseconds(sleep_time_));
     }
-    queue_->set_complete();
+    producer_finalize_();
   }
 
 private:
   ProducerConfig configuration_;
-  std::shared_ptr<ByteMessageQueue> queue_;
+  producer_initialize_function_t producer_initialize_;
+  producer_callback_function_t producer_callback_;
+  producer_finalize_function_t producer_finalize_;
   unsigned int sleep_time_;  // in milliseconds
   // for simplification, this pointer will be reused
   std::shared_ptr<std_msgs::msg::ByteMultiArray> message_;
 };
 
-#endif  // ROSBAG2_PERFORMANCE_WRITER_BENCHMARKING__BYTE_PRODUCER_HPP_
+#endif  // ROSBAG2_PERFORMANCE_BENCHMARKING__BYTE_PRODUCER_HPP_

rosbag2_performance/rosbag2_performance_benchmarking/include/rosbag2_performance_benchmarking/config_utils.hpp

@@ -0,0 +1,46 @@
+// Copyright 2020-2021, Robotec.ai sp. z o.o.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef ROSBAG2_PERFORMANCE_BENCHMARKING__CONFIG_UTILS_HPP_
+#define ROSBAG2_PERFORMANCE_BENCHMARKING__CONFIG_UTILS_HPP_
+
+#include <string>
+#include <vector>
+
+#include "rclcpp/node.hpp"
+#include "rosbag2_performance_benchmarking/bag_config.hpp"
+#include "rosbag2_performance_benchmarking/publisher_group_config.hpp"
+
+namespace config_utils
+{
+
+/// Helper function to fill group_config with qos parameters
+void load_qos_configuration(
+  rclcpp::Node & node,
+  PublisherGroupConfig & group_config,
+  const std::string & group_prefix);
+
+/// Acquires the parameter determining whether to wait for subscriber
+bool wait_for_subscriptions_from_node_parameters(rclcpp::Node & node);
+
+/// Acquires publisher parameters from the node
+std::vector<PublisherGroupConfig> publisher_groups_from_node_parameters(
+  rclcpp::Node & node);
+
+/// Acquires bag parameters from the node
+BagConfig bag_config_from_node_parameters(rclcpp::Node & node);
+
+}  // namespace config_utils
+
+#endif  // ROSBAG2_PERFORMANCE_BENCHMARKING__CONFIG_UTILS_HPP_