-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add tutorial on integration testing #4881
base: rolling
Are you sure you want to change the base?
Conversation
Signed-off-by: Arne Baeyens <mail@arnebaeyens.com>
This pull request has been mentioned on ROS Discourse. There might be relevant details there: https://discourse.ros.org/t/towards-a-rosdoc-integation-testing-tutorial/40701/2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First of all, thank you for the tutorial. This is documentation we really need.
I've left quite a number of comments inline. Please take a look when you get a chance.
.. TestingIntegration: | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this is a new tutorial that has nothing linking to it, we don't need this.
.. TestingIntegration: |
Before starting this tutorial, | ||
it is recommended to have completed the following tutorials on launching nodes: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Before starting this tutorial, | |
it is recommended to have completed the following tutorials on launching nodes: | |
Before starting this tutorial, it is recommended to have completed the following tutorials on launching nodes: |
Where unit tests focus on validating a very specific piece of functionality, | ||
integration tests go all the way in the other direction. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where unit tests focus on validating a very specific piece of functionality, | |
integration tests go all the way in the other direction. | |
Where unit tests focus on validating a very specific piece of functionality, integration tests focus on validating the interaction between pieces of code. |
In ROS 2 this means launching a system of one or several nodes, | ||
for example the Gazebo simulator and the Nav2 navigation stack. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In ROS 2 this means launching a system of one or several nodes, | |
for example the Gazebo simulator and the Nav2 navigation stack. | |
In ROS 2 this is often accomplished by launching a system of one or several nodes, for example the `Gazebo simulator <https://gazebosim.org/home>`__ and the `Nav2 navigation <https://github.com/ros-planning/navigation2.git>`__ stack. |
A key aspect of ROS 2 integration testing is that nodes part of different tests | ||
shouldn't communicate with each other, even when run in parallel, | ||
which will be achieved here using a specific test runner | ||
that picks unique ROS domain id's. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A key aspect of ROS 2 integration testing is that nodes part of different tests | |
shouldn't communicate with each other, even when run in parallel, | |
which will be achieved here using a specific test runner | |
that picks unique ROS domain id's. | |
A key aspect of ROS 2 integration testing is that nodes that are part of different tests | |
shouldn't communicate with each other, even when run in parallel. | |
This will be achieved here using a specific test runner that picks unique :doc:`ROS domain IDs <../../../Concepts/Intermediate/About-Domain-ID>`. |
* An undecorated class inheriting from ``unittest.TestCase``: | ||
houses the active tests, including set up and teardown. | ||
One has access to the ROS logging through ``proc_output``. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
* An undecorated class inheriting from ``unittest.TestCase``: | |
houses the active tests, including set up and teardown. | |
One has access to the ROS logging through ``proc_output``. | |
* An undecorated class inheriting from ``unittest.TestCase``: This houses the active tests, including set up and teardown, and gives access to ROS logging through ``proc_output``. |
* A second class inheriting from ``unittest.TestCase``, | ||
decorated with ``@launch_testing.post_shutdown_test()``. | ||
As the name implies, these tests run after all nodes have shutdown. | ||
A common assert here is to check the exit codes, | ||
to ensure all nodes exited cleanly. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
* A second class inheriting from ``unittest.TestCase``, | |
decorated with ``@launch_testing.post_shutdown_test()``. | |
As the name implies, these tests run after all nodes have shutdown. | |
A common assert here is to check the exit codes, | |
to ensure all nodes exited cleanly. | |
* A second class inheriting from ``unittest.TestCase`` decorated with ``@launch_testing.post_shutdown_test()``: These are tests that run after all nodes have shutdown; it is common to assert that the nodes exited cleanly. |
The launch test is subsequently registered in the ``CMakeLists.txt`` | ||
using the custom cmake macro ``add_ros_isolated_launch_test`` | ||
that ensures that each launch test runs with a unique ``ROS_DOMAIN_ID``, | ||
avoiding undesired cross communication. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The launch test is subsequently registered in the ``CMakeLists.txt`` | |
using the custom cmake macro ``add_ros_isolated_launch_test`` | |
that ensures that each launch test runs with a unique ``ROS_DOMAIN_ID``, | |
avoiding undesired cross communication. | |
The launch test is subsequently registered in the ``CMakeLists.txt`` using the custom cmake macro ``add_ros_isolated_launch_test`` which ensures that each launch test runs with a unique ``ROS_DOMAIN_ID``, | |
avoiding undesired cross communication. |
To finish, tools such as Xunit Viewer ease visualizing the test results | ||
in a more colorful way than the colcon utilities. | ||
We hope this tutorial gave the reader a good grasp | ||
of how to conduct integration tests in ROS 2, | ||
delineating tests, and analyzing their results. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To finish, tools such as Xunit Viewer ease visualizing the test results | |
in a more colorful way than the colcon utilities. | |
We hope this tutorial gave the reader a good grasp | |
of how to conduct integration tests in ROS 2, | |
delineating tests, and analyzing their results. |
* Extend the two basic active tests with one that checks | ||
whether the turtle is responsive to twist commands, | ||
and another that verifies whether the spawn service | ||
sets the turtle to the intended pose | ||
(see the `Turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`). | ||
* Instead of Turtlesim, launch the | ||
:doc:`Gazebo simulator <../../Advanced/Simulators/Gazebo/Gazebo>` | ||
and simulate *your* robot in there, automating tests | ||
that would otherwise depend on manually operating your physical robot. | ||
* Go through the | ||
`launch_testing documentation <https://docs.ros.org/en/{DISTRO}/p/launch_testing/index.html>`_ | ||
and explore the many possibilities for integration testing it offers. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is basically "homework", and while I don't necessarily disagree with it, this isn't what we do in our other tutorials. I would suggest just removing this for now.
This PR adds a tutorial on integration testing with
launch_testing
and complements the two existing tutorials on unit testing. This topic was first discussed in #4827 and thereafter on ROS discourse. This is my first PR for the ROS 2 project, all feedback welcome!Resolves #4827 and closes ros2/launch#812.