GSOC 2024 : Zenoh Support & Benchmarking #2844
Comments
|
In addition all of this, i want to say that i have refactored the perception pipeline tutorial in moveit2_tutorials in order to see the effects of rmw_zenoh againts perception pipeline scenario. |
|
Hello all, I want to share the first insights about MoveIt + rmw_zenoh combination. Firstly, when i executed MoveIt demo with rmw_zenoh, even though gripper was loaded and usable in rviz2, manipulator was not loaded and invisible in rviz2 like below image. In addition, planning library was not loaded. As result of this, it also doesn't work planning and executing pipeline.
I started to inspect ros logs once demo was crashed. In ros logs, i saw that problem is based on controller spawner node, especially joint_state_broadcaster. And i also caught some bugs which it is suggested to report it in rmw_zenoh. This errors are below. However i should say that i also observed MoveIt demo + rmw_zenoh is executed without problem. which works so rarely. It generally gives this bellow errors in logs.
I also ran test cases of ros2_control and ros2_controllers using this dockerfile and then the some test cases are crashed. When i ran test cases several times, about 1-2 failures occured after testing. Shortly it is sometimes failed tricycle_controller + joint_trajectory_controller and it is sometimes failed just joint_state_publisher and so on. The common point of this failures was joint_trajectory_controller. Moreover, all failures in joint_trajectory_publisher was due to below segmentation fault. I continued to search this fault and its cause. And at the moment i just can say that segmentation fault occurs at this line of one of joint_trajectory_controller package's test files Furthermore, i run only failed test case of joint_trajectory_controller with gdb debug and result is below. Finally i want to add that, i also ran MoveIt test cases with both rmw_zenoh_cpp and rmw_fastrtps_cpp middlewares using this dockerfile. It is surprising that the execution with rmw_zenoh gives less failures in test cases than the execution with rmw_fastrtps. Here is results. the execution of moveit test cases with rmw_zenoh the execution of moveit test cases with rmw_fastrtps That's all at the moment in testing side. I need to work more on this errors to give more information about that subject. |
|
For benchmarking, i just have created this repository in my account and i will develop benchmark code on this repository until the end of GSOC project. Here is our repository. I give a link for those who wanted to track the zenoh benchmarking progress. https://github.com/CihatAltiparmak/moveit_middleware_benchmark I need to refactor some part of repository at the moment, however i will. We want that our benchmarks is based on rmw_zenoh. Whereas we can switch to dds-based middleware + zenoh_dds_bridge combination rather than rmw_zenoh due to the fact that rmw_zenoh is the experimental middleware and still has some bugs. This doesn't mean that our benchmark is not based on rmw_zenoh. We will continue to track the development progress of rmw_zenoh closely. Maybe we can fail in this progress in side of benchmark of rmw_zenoh but i will try to implement this benchmark codes in a way it can be able to runnable even after GSOC project. |
sjahr
added
persistent
|
Hello all, We want to share some updates related to this project. Integration of zenoh to MoveItWhen we ran MoveIt with rmw_zenoh middleware, we observed some stuck in the action clients and this stuck occurs due to rmw_wait having a data race. This seems related to rmw_zenoh, not MoveIt. Additionaly, the reason of this stuck seems MultiThreadedExecutor. rmw_zenoh seems to work without problem with SingleThreadedExecutor but has problems with MultiThreadedExecutor. For those seeking more information about this problem, here is some additional content. It should also be mentioned that it seems navigation2 works well with rmw_zenoh especially in this webinar. when I searched whether navigation2 uses MultiThreadedExecutor, i saw that navigation2 doesn't use MultiThreadedExecutor but uses SingleThreadedExecutor while MoveIt uses MultiThreadedExecutor in some part of code. Furthermore, this stucks helped us to discover some bad implementations inside MoveIt. I'm still working on a better implementation. We also discussed trying rmw_zenoh in MoveIt, modified in a way to use another private node spinning technique instead of MultiThreadedExecutor, both to avoid breaking the original MoveIt flow and to enable the use of rmw_zenoh without problem. By the way there are two upstream in rmw_zenoh to fix this bug. I tried it but yet to run with MoveIt. You can find these upstreams here. BenchmarkingWe are also working on benchmarking the effects of middleware in MoveIt. We implemented the benchmark code for perception pipeline in this development branch. We will add more scenario to this repository. |
|
Hello all, In today's meeting, one of the topics we talked about was rmw_zenoh integration to MoveIt. I was able to successfully run MoveIt using rmw zenoh with some modifications. You can visit this draft pull request to see what's changed. However i must say that this is experimental pull request and it can be observed some unexpected effects. In order to run MoveIt using rmw_zenoh, you can use my fork at the moment. I am still waiting for better rmw_wait implementation and following rmw_zenoh progress. I also created an issue related to TrajectoryExecutionManager. For benchmarking, unfortunately we have some problems. I don't want to share middleware benchmarking results for perception pipeline scenario at the moment and we are investigating it. CPU time is much less than Real Time in our benchmarking results. That's problem. On the other side, we also discussed about benchmarking memory usage and I suggested this repository. |
|
Hello everyone, I want to share some updates about benchmarking side. The middleware benchmarking tool is still being developed rollingly by us. I conducted some benchmarks for two scenarios, one of them is a basic service-client benchmark and other one is perception pipeline (we couldn't conduct this benchmark using camera topic but it can be conducted using bag file.). You can take a look at some documentation here to get what corresponding scenario aims. Seems like rmw_zenoh (ros2/rmw_zenoh@c12ff3e) is as fast as rmw_fastrtps with some configurations in tcp. I strongly suggest you to visit this thread in Zenoh's discord community. This is the initial benchmark results:
I couldn't understand why rmw_zenoh is dramatically slower than rmw_fastrtps. I will investigate this problem. But i should also say that rmw_zenoh defaultly uses TCP whereas rmw_cyclonedds uses UDP. I will conduct this benchmarks by configuring rmw_zenoh to UDP and let you to know it in second post. I have also worked on visualization side and CI side to conduct benchmarks in github CI. This website doesn't present the real benchmarking results. It's just for dummy visualization. When we merge other PR's in moveit_middleware_benchmark repository, we will be able to start to get some visualizations. Here is the example website. https://darkusalphahydranoid.github.io/moveit_middleware_benchmark_experimental By the way you can try our benchmark tool by pulling its docker image like below. Terminal 1 docker pull ghcr.io/cihataltiparmak/moveit_middleware_benchmark:latest
# run docker image
# inside docker
cd /ws_moveit
source /opt/ros/rolling/setup.bash
source install/setup.bash
export RMW_IMPLEMENTATION=rmw_zenoh_cpp
ros2 launch moveit_middleware_benchmark scenario_basic_service_client_benchmark.launch.pyTerminal 2 Don't be hesitate to ask me about anything related to this GSOC project (benchmarking, bugs in rmw_zenoh and so on). I'm ready to answer you. This is my mail : cihataltiparmak1@gmail.com |
Hello everyone,
I will work on integrating rmw_zenoh middleware to MoveIt and benchmarking the effects of Zenoh at MoveIt in scope of the Google Summer of Code Project with my mentors @sjahr and @henningkayser . We will share this project's updates here.
Our goals are below.
The primary goals of these project are above. However, please don't hesitate to share your ideas. I am open to new ideas.
The text was updated successfully, but these errors were encountered: