Update documentation

octoprobe · Jan 21, 2025 · 3e8d313 · 3e8d313
1 parent 53db2a3
commit 3e8d313
Show file tree

Hide file tree

Showing 32 changed files with 369 additions and 58 deletions.
diff --git a/README_BOM.md b/README_BOM.md
diff --git a/README_installation.md b/README_installation.md
diff --git a/README_manual_tests.md b/README_manual_tests.md
@@ -1,5 +1,7 @@
 # Manual tests
 
+These manual tests allow to verify the functionality of `testbed_micropython`.
+
 ## Group: Flash
 
 ### repl blocks

diff --git a/README_multiprocessing.md b/README_multiprocessing.md
diff --git a/docs/design/10_firmwarebartender.rst b/docs/design/10_firmwarebartender.rst
@@ -0,0 +1,6 @@
+Design: FirmwareBartender
+=============================
+
+TODO: Describe workflow
+
+TODO: Describe multiprocessing
diff --git a/docs/design/20_testbartender.rst b/docs/design/20_testbartender.rst
@@ -0,0 +1,8 @@
+Design: TestBartender
+=============================
+
+TODO: Archtictural decision: Describe difference between pytest (testcollection is done at the beginning) and testbed_micropython where testcollection is done on the fly.
+
+TODO: Describe classes: Test Specification vs Test Run.
+
+TODO: Describe pytest for TestBartender.
diff --git a/docs/design/30_multiprocessing.rst b/docs/design/30_multiprocessing.rst
@@ -0,0 +1,23 @@
+Design: Multiprocessing
+=============================
+
+
+.. mermaid::
+
+   sequenceDiagram
+      participant tr as Testrunner
+      participant b as TestBartender/BuildBartender
+      participant tx as TargetCtx
+      participant f as TestFunc/BuildFunc
+      tr->>b: testrun = testrun_next()
+      b->>tx: run(AsyncResult(testrun))
+      tx->>+f: f(unique_id)
+      f->>-tx: Event(unique_id)
+      tx->>b: done(unique_id)
+
+
+TODO: Describe how '--no-multiprocessing' is implemented.
+
+TODO: Describe communication between main and subprocesses.
+
+TODO: Describe how FirmwareBartender and TestBartender cooperate.
diff --git a/docs/design/index.rst b/docs/design/index.rst
@@ -0,0 +1,8 @@
+Design
+==========================
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   **
diff --git a/docs/developer_guide/10_debugging.rst b/docs/developer_guide/10_debugging.rst
@@ -0,0 +1,21 @@
+Debugging
+=========
+
+* --multiprocessing
+* TestBartender
+
+
+Firmware flashing is flaky
+--------------------------------
+
+This is the fastest way to flash the firmware
+
+.. code:: 
+
+    mptest.cli test --flash-force --only-test=RUN-TESTS_EXTMOD_HARDWARE --only-board=LOLIN_C3_MINI
+
+If only the questioned tentacle is connected, this is even faster:
+
+.. code:: 
+
+    mptest.cli flash
diff --git a/docs/developer_guide/20_enhancement.rst b/docs/developer_guide/20_enhancement.rst
@@ -0,0 +1,2 @@
+Implement Enhancements
+======================
diff --git a/docs/developer_guide/index.rst b/docs/developer_guide/index.rst
@@ -0,0 +1,8 @@
+Installation and Debugging
+==========================
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   **
diff --git a/docs/images/tentacle_DAQ_connector_side.jpg b/docs/images/tentacle_DAQ_connector_side.jpg
diff --git a/docs/images/tentacle_DAQ_connector_top.jpg b/docs/images/tentacle_DAQ_connector_top.jpg
diff --git a/docs/images/tentacle_stack_top_a.jpg b/docs/images/tentacle_stack_top_a.jpg
diff --git a/docs/images/tentacle_stack_top_b.jpg b/docs/images/tentacle_stack_top_b.jpg
diff --git a/docs/images/tentacle_stack_top_c.jpg b/docs/images/tentacle_stack_top_c.jpg
diff --git a/docs/images/tentacle_two_top.jpg b/docs/images/tentacle_two_top.jpg
diff --git a/docs/index.rst b/docs/index.rst
@@ -4,9 +4,30 @@
 Octoprobe: testbed_micropyton
 =============================
 
+.. image:: images/tentacle_stack_top_a.jpg
+    :align: center
+    :height: 400px
+
 This testbed is used to release test the micropython firmware.
 
-There is no sphinx documentation yet.
+.. toctree::
+    :glob:
+    :maxdepth: 2
+
+    /user_guide/index
+
+.. toctree::
+    :glob:
+    :maxdepth: 2
+
+    /developer_guide/index
+
+.. toctree::
+    :glob:
+    :maxdepth: 2
+
+    /design/index
+
 
 
 Indices and tables

diff --git a/docs/tentacles.md b/docs/tentacles.md
diff --git a/docs/user_guide/10_features.rst b/docs/user_guide/10_features.rst
@@ -0,0 +1,78 @@
+Features
+========
+
+Features to support testing
+-----------------------------------------
+
+* Allows to test **by hand**
+
+  Just connect one or more tentacles and run the tests from the micropython repo from the command line against the board on the tentacle.
+
+  For example `run-perfbench.py --pyboard --device=/dev/ttyUSB2`
+
+* Allows to test **ad hoc**
+
+  Just connect one tentacle to your computer and run tests on that tentacle.
+
+  For example `mptest test`.
+
+* Allows to test **regressions**
+
+  Set up a github runner and connect many tentacles to do automated regessions tests.
+
+* Dynamic number of tentacles
+
+  As soon as tentacles are connected via USB, they will be used for the tests.
+
+  Every tentacle is identified by its serial number. A inventory lookup will provide the required information about that tentacle.
+
+* Tab completion
+
+  `mptest test --only-board / --only-test` provide tab completion.
+
+* Timing report
+
+  To save time, everything should run in parallel: Building and testing.
+  `task_report.md/.txt/.html` summarizes where the time was spent.
+
+* Labels `mptest labels`
+
+  The labels in front of every tentacle may be created automatically.
+
+* Current state `mptest list`
+
+  This command shows the connected tentacles and the implemented tests.
+
+
+Technical features
+-----------------------------------------
+
+* Verify flashed firmware
+
+  After flashing a firmware, the strings `sys.version / sys.implementation` are compared against the build result.
+  This allows to detect if the wrong firmware was flashed to the wrong tentacle.
+  *Nothing is more tedious than testing against the wrong target*!
+
+* One tentacle covering multipe `boards`
+
+  For example the `RPI_PICO2_W` supports `FUT_WLAN` and is therefore a superset of `RPI_PICO2`.
+  So it would be beneficial to have ONE tentacle to test both.
+  I decided not to support this as it would increase overall complexity. The workaround is to solder TWO tentacles which is not very expensive/time consuming and speeds up the tests!
+
+* Journalctl
+
+  Some usb errors might pop up in `dmesg` or `Journalctl`. For example limited USB bandwidth.
+  `testbed_micropython` starts `journalctl` in the background and terminates loudly when a critical message is detected.
+
+* multiprocessing
+
+  The python multiprocessing package is used to allow parallelism. The timeout feature of that package that package is used to recover from blocking which might be caused by a blocking test or mpremote.
+
+* DAQ connector
+
+  Every tentacle provides a 12 pin connector were a DAQ may be connected to spy on the signals.
+
+* Bring boards in to boot mode (programming mode): `mptest debugbootmode`
+
+  Figuring out how to put a board into boot mode is tedious work.
+  `mptest debugbootmode` allows to run just that code.
diff --git a/docs/user_guide/20_user_guide.rst b/docs/user_guide/20_user_guide.rst
@@ -0,0 +1,126 @@
+User Guide
+==========
+
+Indended audience: A MicroPython firmware or test developer who wants to use `testbed_micropython`.
+
+Summary
+-----------------------
+
+`testbed_micropython` is a wrapper around these tests: https://github.com/micropython/micropython/tree/master/tests/xx.py.
+
+The testbed resposibility is:
+
+* query all connected tentacles
+* build the firmware required by the connected tentacles and board-variants
+* flash the tentacle if required. Some tentacles have to be flashed multiple times to be able to test board-variants.
+* run all tests against these tentacles
+* collect all build results / test results
+
+Installation
+------------------------
+
+Follow these instructions: https://www.octoprobe.org/testbed_showcase/installation/10_ubuntu.html
+
+Please replace every occurence of `testenv_showcase` with `testenv_micropython`.
+
+Usage scenarios
+-----------------------
+
+Usage principles
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A firmware developer typically has the micropython repo checked out and works within that folder.
+
+These 3 companion programs ease the work:
+
+* `mpremote`: Communicate with a board.
+* `mpbuild`: Build a firmware using docker containers.
+* `mptest`: Test the firmware. `mptest` makes intense use of `mpremote` and `mpbuild`.
+
+Automated Regression Test - full coverage
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* Goal:
+
+  * As much coverage as possible
+  * Run fully unattended
+
+.. code::
+
+    mptest test --micropython-tests=<git/branch>  --firmware-build=<git/branch> --flash-force --git-clean
+
+* `mptest` will:
+
+  * clone/checkout the required branches
+  * build the firmware
+  * run the tests
+  * collect the results.
+
+Automated Regression Test - limited coverage
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* Goal:
+
+  * As above
+  * The testrun should be faster (as a developer might be waiting for the results)
+  * The testresult should be limited (as a developer just changed code for one board)
+
+.. code::
+
+    mptest test .. --only-board=RPI_PICO2 --only-test=RUN-TESTS_NET_HOSTED
+
+Working on the firmware: Build and test
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* Goal:
+
+  * The developer is working on the C-code and wants to have test feedback fast.
+
+* Workflow:
+
+  * The developer has the repo checked out and is working on the code.
+  * The current directory is always the micropython git repo.
+  * Use `mptest` to compile and test the firmware.
+
+.. code::
+
+    # The current directory is the micropython git repo
+    mptest test
+
+This now will build/test the firmware against all connected tentacles.
+
+As the git repo is 'dirty' the firmware will ALWAYS be flashed.
+
+.. code::
+
+    # As above, but the tests are taken from another branch
+    mptest test --micropython-tests=<git/branch>
+
+
+Working on a test: Flash once, test many times
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* Goal:
+
+  * The developer is improving the tests and wants to have test feedback fast.
+  * As build/flash is slow, this step should only be done once.
+
+* Workflow:
+
+  * The developer has the repo checked out and is working on the test code.
+  * The current directory is always the micropthon git repo.
+  * Preparation: Build the firmware and flash the tentacles - this is slow!
+
+.. code::
+
+    # Preparation: Compile firmware and flash the tentacles
+    mptest flash --firmware-build=<git/branch>
+
+This will take a few minutes.
+
+.. code::
+
+    # Run the tests
+    mptest test --flash-skip
+
+This is fast - the tests immediately!