Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tutorial on integration testing #4881

Open
wants to merge 1 commit into
base: rolling
Choose a base branch
from

Conversation

abaeyens
Copy link

@abaeyens abaeyens commented Dec 1, 2024

This PR adds a tutorial on integration testing with launch_testing and complements the two existing tutorials on unit testing. This topic was first discussed in #4827 and thereafter on ROS discourse. This is my first PR for the ROS 2 project, all feedback welcome!

Resolves #4827 and closes ros2/launch#812.

Signed-off-by: Arne Baeyens <mail@arnebaeyens.com>
@ros-discourse
Copy link

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/towards-a-rosdoc-integation-testing-tutorial/40701/2

Copy link
Contributor

@clalancette clalancette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, thank you for the tutorial. This is documentation we really need.

I've left quite a number of comments inline. Please take a look when you get a chance.

Comment on lines +1 to +2
.. TestingIntegration:

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is a new tutorial that has nothing linking to it, we don't need this.

Suggested change
.. TestingIntegration:

Comment on lines +19 to +20
Before starting this tutorial,
it is recommended to have completed the following tutorials on launching nodes:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Before starting this tutorial,
it is recommended to have completed the following tutorials on launching nodes:
Before starting this tutorial, it is recommended to have completed the following tutorials on launching nodes:

Comment on lines +28 to +29
Where unit tests focus on validating a very specific piece of functionality,
integration tests go all the way in the other direction.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Where unit tests focus on validating a very specific piece of functionality,
integration tests go all the way in the other direction.
Where unit tests focus on validating a very specific piece of functionality, integration tests focus on validating the interaction between pieces of code.

Comment on lines +30 to +31
In ROS 2 this means launching a system of one or several nodes,
for example the Gazebo simulator and the Nav2 navigation stack.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In ROS 2 this means launching a system of one or several nodes,
for example the Gazebo simulator and the Nav2 navigation stack.
In ROS 2 this is often accomplished by launching a system of one or several nodes, for example the `Gazebo simulator <https://gazebosim.org/home>`__ and the `Nav2 navigation <https://github.com/ros-planning/navigation2.git>`__ stack.

Comment on lines +34 to +37
A key aspect of ROS 2 integration testing is that nodes part of different tests
shouldn't communicate with each other, even when run in parallel,
which will be achieved here using a specific test runner
that picks unique ROS domain id's.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A key aspect of ROS 2 integration testing is that nodes part of different tests
shouldn't communicate with each other, even when run in parallel,
which will be achieved here using a specific test runner
that picks unique ROS domain id's.
A key aspect of ROS 2 integration testing is that nodes that are part of different tests
shouldn't communicate with each other, even when run in parallel.
This will be achieved here using a specific test runner that picks unique :doc:`ROS domain IDs <../../../Concepts/Intermediate/About-Domain-ID>`.

Comment on lines +388 to +390
* An undecorated class inheriting from ``unittest.TestCase``:
houses the active tests, including set up and teardown.
One has access to the ROS logging through ``proc_output``.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* An undecorated class inheriting from ``unittest.TestCase``:
houses the active tests, including set up and teardown.
One has access to the ROS logging through ``proc_output``.
* An undecorated class inheriting from ``unittest.TestCase``: This houses the active tests, including set up and teardown, and gives access to ROS logging through ``proc_output``.

Comment on lines +391 to +395
* A second class inheriting from ``unittest.TestCase``,
decorated with ``@launch_testing.post_shutdown_test()``.
As the name implies, these tests run after all nodes have shutdown.
A common assert here is to check the exit codes,
to ensure all nodes exited cleanly.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* A second class inheriting from ``unittest.TestCase``,
decorated with ``@launch_testing.post_shutdown_test()``.
As the name implies, these tests run after all nodes have shutdown.
A common assert here is to check the exit codes,
to ensure all nodes exited cleanly.
* A second class inheriting from ``unittest.TestCase`` decorated with ``@launch_testing.post_shutdown_test()``: These are tests that run after all nodes have shutdown; it is common to assert that the nodes exited cleanly.

Comment on lines +397 to +400
The launch test is subsequently registered in the ``CMakeLists.txt``
using the custom cmake macro ``add_ros_isolated_launch_test``
that ensures that each launch test runs with a unique ``ROS_DOMAIN_ID``,
avoiding undesired cross communication.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The launch test is subsequently registered in the ``CMakeLists.txt``
using the custom cmake macro ``add_ros_isolated_launch_test``
that ensures that each launch test runs with a unique ``ROS_DOMAIN_ID``,
avoiding undesired cross communication.
The launch test is subsequently registered in the ``CMakeLists.txt`` using the custom cmake macro ``add_ros_isolated_launch_test`` which ensures that each launch test runs with a unique ``ROS_DOMAIN_ID``,
avoiding undesired cross communication.

Comment on lines +402 to +406
To finish, tools such as Xunit Viewer ease visualizing the test results
in a more colorful way than the colcon utilities.
We hope this tutorial gave the reader a good grasp
of how to conduct integration tests in ROS 2,
delineating tests, and analyzing their results.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To finish, tools such as Xunit Viewer ease visualizing the test results
in a more colorful way than the colcon utilities.
We hope this tutorial gave the reader a good grasp
of how to conduct integration tests in ROS 2,
delineating tests, and analyzing their results.

Comment on lines +411 to +422
* Extend the two basic active tests with one that checks
whether the turtle is responsive to twist commands,
and another that verifies whether the spawn service
sets the turtle to the intended pose
(see the `Turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`).
* Instead of Turtlesim, launch the
:doc:`Gazebo simulator <../../Advanced/Simulators/Gazebo/Gazebo>`
and simulate *your* robot in there, automating tests
that would otherwise depend on manually operating your physical robot.
* Go through the
`launch_testing documentation <https://docs.ros.org/en/{DISTRO}/p/launch_testing/index.html>`_
and explore the many possibilities for integration testing it offers.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is basically "homework", and while I don't necessarily disagree with it, this isn't what we do in our other tutorials. I would suggest just removing this for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants