MATLAB testing GitHub Action

[jared321]

Check to see if MATLAB tests are being automatically run in GitHub actions. If not, then implement this including (hopfully) code coverage.

[jared321]

Unfortunately, I believe that this task is intimately tied to the MATLAB path issue #127. I will try to fix up paths as managed by POUNDERS and its tests to get this working.

Requirements

• Some test cases such as Testpounders just call individual scripts such as test_one_m_pounders. The paths shall be setup automatically within each of these test/example executables so that a user can run MATLAB from the POUNDERS test folder and execute correctly the function by itself with no need for unnecessary additions to the MATLAB path. At the moment, the only necessary cases should be related to BenDFO which is a repo separate from IBDFO and not included in it as a submodule (as MINQ is).
• If a test/example function sets the path, before terminating it shall reset the path to its state upon entry so that it does not alter the path at the scope of the calling code. At the moment I am not taking this to the logical end where a program should satisfy this requirement if it aborts or terminates early by assert.
• Users should be able to execute the full test suite from different folders such as pounders, pounders/m, and pounders/m/tests without needing to adjust paths. This implies that paths in the code and tests should not be relative to the current working directory. This requirement is inline with the fact that runtests will run tests only in the current working directory but can be asked to look for tests in subfolders.
  • Because of this requirement, we should be able to run the tests from the root of the repository, which would run tests on all MATLAB subpackages with tests. As a result, there would be a single coverage report for all MATLAB.
• This implementation shall be such that users can run code coverage associated with the full test suite of each MATLAB subpackage locally and generate an HTML format coverage report. In addition, they should be able to run the full IBCDFO MATLAB test suite and generate a related HTML format report.
• The MATLAB testing GitHub action shall generate a full POUNDERS test suite HTML format code coverage report and make it available as an artifact. In addition, it shall create a cobertura XML-format coverage report, which can be used with code coverage services such as CodeCov. The action shall publish the report with the code coverage service.
• (optional) If the Python coverage report can be written in cobertura XML format, then the action shall publish the Python and MATLAB reports with the code coverage simultaneously so that the full code base is grouped as a whole in the service.

[jared321]

I have updated manifold sampling as well so that in theory it could be tested with coverage in the action as well. However, it has an external dependency to the allcomb function in the MathWorks file exchange. To download this for working locally, I had to log in with a MathWorks username/password.

I would like to avoid having a CI action that logs into FileExchange with an account since that is more complex and I prefer to avoid potential security issues. A few other possibilities to explore:

• MATLAB has a feature called Projects, which appear to be analogous to virtual environments. In particular, you can install setup/teardown scripts that can setup/teardown the environment. See if these can be setup to automatically acquire and download dependencies in FileExchange without an account.
• MATLAB has a feature called Toolboxes which community developers can setup. They appear to be analogous to Python packages. See if these can be setup to automatically acquire and download dependencies in FileExchange without an account.
• MATLAB has several GitHub actions that I am using for CI. These already install a version of MATLAB without me inputing any user or license information. It would make sense if these actions could access FileExchange dependencies directly since MathWorks would be able to see what projects are using the files. I need to see if this exists or contact their CI teams to see if it is in the works.

Note that use of Projects/Toolboxes should be investigated simultaneously with the MATLAB path management Issue #127.

[jared321]

Merged into develop. Closed.