Integration Test Framework (#224)

* Quick POC code

* Fix flake8 errors

* Move integration flow

* Autogenerate input types

* Dynamic input typing works

* Dynamic publishing works

* Fix launch configs

* Expected outputs working

* Improve import of generated code

* Arrays as inputs/outputs work

* Inline documentation

* Move integration tests to a new package

* ros launch works

* Fix linting

* Better typing organization

* Change variables named input to input_dict

* Add diagram

* Add global_launch and integration_test folders to workspace

* Move template testplan to README

* Fix gitignore

* Change generated code import

* Make script executable

* Simplify getting config absolute path

* Use f strings

* Echo launch file generation info

* Remove launch file generation

* Remove launch file generation pt 2

* No more need for autogenerated file

* Move main function to top

* Improved exception handling

---------

Co-authored-by: Patrick Creighton <pcreighton429@gmail.com>

.devcontainer/Dockerfile

@@ -1,4 +1,4 @@
-FROM ghcr.io/ubcsailbot/sailbot_workspace/dev:plantuml2
+FROM ghcr.io/ubcsailbot/sailbot_workspace/dev:integration_tests
 
 # Copy configuration files (e.g., .vimrc) from config/ to the container's home directory
 ARG USERNAME=ros

.devcontainer/base-dev/base-dev.Dockerfile

@@ -204,7 +204,9 @@ ENV DEBIAN_FRONTEND=
 # install dev python3 dependencies
 RUN pip3 install \
     # to be able to run juypter notebooks
-    ipykernel
+    ipykernel \
+    # for integration_tests package
+    types-PyYAML
 
 # install other helpful apt packages
 ENV DEBIAN_FRONTEND=noninteractive

.gitignore

@@ -5,6 +5,7 @@
 
 # vcstool packages
 /src/*
+!/src/integration_tests
 !/src/global_launch
 !/src/polaris.repos
 

sailbot.code-workspace

@@ -18,6 +18,12 @@
         {
             "path": "src/docs"
         },
+        {
+            "path": "src/global_launch"
+        },
+        {
+            "path": "src/integration_tests"
+        },
         {
             "path": "src/local_pathfinding"
         },
@@ -491,6 +497,7 @@
                     "controller",
                     "custom_interfaces",
                     "global_launch",
+                    "integration_tests",
                     "local_pathfinding",
                     "network_systems",
                 ]

src/integration_tests/README.md

@@ -0,0 +1,50 @@
+# Integration Tests
+
+The Integration Tests package defines Polaris' software verification suite. With it, we can specify testplans that
+encompass the entire software project to verify the integration of our various software projects.
+
+## Defining Tests
+
+Tests are defined using `.yaml` files found under the `testplan/` directory.
+Each testplan should implement the following template:
+
+```yaml
+timeout_sec: # Integer (Optional): Number of seconds allowed for the test to run (default 3 seconds)
+
+required_packages: # List of packages and configuration files
+  - name: # String: one of (boat_simulator, controller, local_pathfinding, network_systems, virtual_iridium, website)
+    configs: # List[String] (Optional): config files relative to the package's config/ folder
+
+inputs: # List of inputs to drive the test
+  - type: # String: one of (ROS, HTTP<TODO>)
+    name: # String: Name of ROS topic or HTTP<TODO> target
+    data: # Varies: Define the type and value of the input
+      dtype: # One of (HTTP type<TODO>, custom_interfaces type, std ROS message type)
+      field_name: # dtype: field_val (both field_name and field_val depend on the dtype)
+                  # if dtype is a builtin type (ex. int32), then set "field_name: <field_val>" as "val: <int val>"
+
+expected_outputs: # List of expected outputs
+  - type: # String: one of (ROS, HTTP<TODO>)
+    name: # String: Name of ROS topic or HTTP<TODO> target
+    data: # Varies: Define the type and value of the output
+      dtype: # One of (HTTP type<TODO>, custom_interfaces type, std ROS message type)
+      DONT_CARE: # Boolean (Optional): Specify True if output data does not matter. Default is False.
+      field_name: # dtype (Optional): field_val (both field_name and field_val depend on the dtype)
+                  # Required if DONT_CARE is False
+```
+
+See [example.yaml](testplans/example.yaml) for an example implementation.
+
+## Invoking with ros2 run
+
+To directly run a testplan, we can use a `ros2 run` command as follows:
+
+```shell
+ros2 run integration_tests run --ros-args -p testplan:=<path/to/testplan.yaml>
+```
+
+For example:
+
+```shell
+ros2 run integration_tests run --ros-args -p testplan:=testplans/example.yaml
+```

src/integration_tests/diagrams/src/structure.puml

@@ -0,0 +1,74 @@
+@startuml Structure
+title Integration Test Structure
+
+!include %getenv("PLANTUML_TEMPLATE_PATH")
+
+box "Integration Test Node"
+    participant "Main" as main
+    participant "Test Sequence" as seq
+    participant "Driver" as drv
+    participant "Monitor" as mon
+end box
+
+autonumber
+
+-> main ++ : Run Integration Test
+note left of main
+For example:
+<code>
+ros2 launch
+integration_tests
+main_launch.py
+config:=config/example.yaml
+endnote
+
+==Initialization==
+
+--> main : Read ROS parameters (testplan filepath)
+main -> seq --++ : Get testplan
+--> seq : Read testplan file
+seq -> seq : Parse fields
+seq -> seq : Save inputs, expected outputs, timeout
+seq -> seq : Launch packages and save process IDs
+seq -> main --++ : return
+seq --> main : Get inputs
+main --> drv : Set inputs
+seq --> main : Get expected outputs
+main --> mon : Set expected outputs
+
+==Run==
+
+main -> drv  --++ : Send inputs
+drv -> drv : Drive inputs
+drv -> main --++ : return
+main -> main : Wait for timeout
+deactivate main
+
+opt If monitored output is updated
+    mon <-- ++ : Read output
+    mon -> mon : Save output value(s)
+    note left of mon
+        This "if" step is asynchronous
+    endnote
+    /'autonumber and #transparent trick to buffer deactivation'/
+    autonumber stop
+    mon -[#transparent]-> mon
+    autonumber resume
+    deactivate mon
+end
+
+==Evaluation==
+
+main -> main ++ : Timeout triggered
+main -> seq --++ : Notify test sequence
+seq -> seq : Kill package processes
+seq -> main --++ : return
+main -> mon --++ : Verify outputs
+loop for each monitored output
+    mon -> mon : Compare expected with received outputs
+end
+mon -> main --++ : return warnings and failures
+main -> main : Output results
+<- main -- : Exit
+
+@enduml