Proposal for framework usage and configuration

[miguelafsilva5]

Hello,

As we have discussed in previous meetings, me and @Diogo21Costa were to propose the organization of our framework across all bao components.
We assume that our repo is to be added across all bao elements, similarly to the ci repo.
Taking this into consideration our proposed directory organization is the following: (- is a directory and * is a file)

- baremetal (example Module under test)
 - bao-tests (our repo)
    - templates
        * config_template.yaml
    - framework
        * **config.yaml**    (personal configuration file for toolchains and paths. should be added to gitignore)
    - build <---- add to gitignore
        * img.bin (output from compile and tool)
 - src (defaul on all repos)
    - tests (this directory can be configured in the config.yaml, but this is our default suggestion)
        - src
            * ABCD.c (files with test implementation)
            * XPTO.c
        - example
            * **example.yaml** (test battery descriptor)
            * config.tool (envisioned file to configure/use the tool to generate the remaining bao stack binaries)

Lastly, the we have two suggestions for the highlighted .yaml files (config.yaml and example.yaml).
The config.yaml is to filled by each developer with their own directoy path. From early discussions, we established that all directories must absolute, and thus, such a file is needed. This is what we propose:

bao_test_cfg:
  bao_test_dir: /path/to/bao-tests/dir  (to be removed later)
  tests_dir: /home/usr/bao/baremetal/src/tests

platform_cfg:
  platforms:
    qemu-aarch64-virt:
      toolchain: /path/to/toolchain
    qemu-riscv64-virt:
      toolchain: /path/to/toolchain
     zcu102:
      toolchain: /path/to/toolchain
    

The example.yaml file should be placed inside a folder on the path defined in tests_dir. The folder represents a group of tests (battery). These tests can be configured in a single yaml file or in multiple files. The idea is to allow the developer to group their tests, and use multiple yaml files to differentiate tests running in different platforms, for instance.
(Note that when running our framework a battery name should be provided, e.g., make run example)

tests_cfg:
  log_level: int_log_level

  platforms:
    qemu-aarch64-virt
    zcu102

  suites:
    ABCD:
        - ABCD_test1_name
        - ABCD_test2_name
    XPTO:
        - XPTO_test1_name
    SUITE1: all

If you have any suggestions or see any problems, feel free to discuss.

Thank you,
Miguel Silva

@bao-project/core

[josecm]

Thanks for the staring the discussion @miguelafsilva5. There are some points I want to discuss:

• config.yaml (personal configuration file for toolchains and paths. should be added to gitignore)

I'm struggling what to understand why we need to configure the toolchains using the yaml files. Can't we just re-use the Makefile and tool infrastructure that will build the the module-under-test? Those build components must already know what toolchain to use outside the testing framework.

From early discussions, we established that all directories must absolute, and thus

Can you refresh my mind on why they must be absolute. And why the are absolutely needed. I'd say we could have relative path defaults (from which we could eventually calculate the absolute ones), making this file optional (only needed if the developer wants to configure these paths to custom ones.

• src (defaul on all repos)
  • tests (this directory can be configured in the config.yaml, but this is our default suggestion)

I also don't like having the tests directory inside the src directory. I would suggest they can be side by side. To be clearer which folder would have the actual tests we would have the bao-tests submodule named test-framework. And maybe have it inside the actual tests folder:

- baremetal
    - src
    - tests
       - test-framework (bao-test submodule)
      -  

Now as for the remaining files in the tests directory, I'm still not sure how they are actually used. First, I'm confused on the difference between test batteries and test suites. Aren't they both "groups of tests? Why do we need two levels of grouping? What is the semantics of each level here. Nevertheless, FWIU, the yaml battery files are used to trigger either a build or a run of one or multiple suites of tests. Based on the suite definitions, the framework will gather the needed source files for each suite from tests/src and build them. Is this correct?

Would it make sense to define each tests, then test suites, then batteries separately? That is, separate folder and/or files?

Is each test supposed to be compiled in a single binary? If so, e could move part of the logic to Makefiles directly: having a separate folder for each directory containing the sources and the Makefile on how to build that test. Or are suites grouped in a single binary? If so, we could have a Makefile per-suite that would gather the makefiles of each test.

I'm thinking we first need to define these terms (test, suite, battery) clearly before coming up with the file layout and build logic.

• config.tool (envisioned file to configure/use the tool to generate the remaining bao stack binaries)

Do you mean the partition configuration in the device-tree format (as used to configure the tool "normally")? I would say this is an issue I hadn't forseen because the partition binary paths are supposed to be embedded in the device tree.

[miguelafsilva5]

Hello @josecm.
I was working on the PR draft so we could have a better discussion but I ended up with some doubts so I decided replying before posting the draft. We are also wainting for some PR to be pushed to the main branch, as some yaml-related functions are not working as intended.
I'll try to address all your points but they will proabably be out of order.

Now as for the remaining files in the tests directory, I'm still not sure how they are actually used. First, I'm confused on the difference between test batteries and test suites. Aren't they both "groups of tests? Why do we need two levels of grouping? What is the semantics of each level here. Nevertheless, FWIU, the yaml battery files are used to trigger either a build or a run of one or multiple suites of tests. Based on the suite definitions, the framework will gather the needed source files for each suite from tests/src and build them. Is this correct?

Would it make sense to define each tests, then test suites, then batteries separately? That is, separate folder and/or files?

I think the first to be clarified is what we envision as batteries, suites and tests. Test or a unit test is self explanatory. Basically a C function/macro created by the developer that performs a set of checks resulting in a success or fail.

BAO_TEST(SUITE_NAME, TEST_NAME){ //code EXPECTED_EQUAL(x,y); //code EXPECTED_EQUAL(z,y); }

A suite is a conceptual set of tests grouped by what the unit/peripheral/function/feature they test. For instance, the developer could have a suite of tests for Memory tests and another for validating a Timer configuration. This grouping allows for a easier use of the framework when we to not run every single test. Additonally, the developer can opt to run a specific suite of tests without having to specific each test indiviually.
The suites are defined by the user when they declare a test.

Let me preface the next explanation with saying that battery is not a term that the test framework team agree upon, or even discussed. I was lacking a better term to talk about it. But we agree on the concept it represents. The idea behind the "battery" of tests, is all the tests that will run on a specific platform for a specific module under test. Let's say we have 2 different platforms, qemu-aarch64-virt and zcu-102. We imagine that there are tests that are incompatible between the two platforms, e.g., different peripheral or memory addresses. Having a yaml to describe the test battery allows us and developer to easily indicate which tests and suites should run in the different platforms. It also allows the user to specify a set of test to be executed in multiple platform (later stages of ci, when the tests will be executed in a virtual platform and a live board).

I also don't like having the tests directory inside the src directory. I would suggest they can be side by side. To be clearer which folder would have the actual tests we would have the bao-tests submodule named test-framework. And maybe have it inside the actual tests folder:

I think the bao-tests is a relic from the past and we never updated the name. @ESCristiano will take care of that shortly. Your suggestion seems fine to me. I think it is an easy fix as we currently have those directories defined in a yaml file.

I'm struggling what to understand why we need to configure the toolchains using the yaml files. Can't we just re-use the Makefile and tool infrastructure that will build the the module-under-test? Those build components must already know what toolchain to use outside the testing framework.

I think @Diogo21Costa should be the one to answer this. But from my perspective and from what we have worked until now, since we didn't have an established a definited directory tree, this yaml file help us configuring the framework tool without much hassle of going through multiple makefiles. (Particularly for me, that I originally had the framework repo outside of baremetal/bao directories).

Can you refresh my mind on why they must be absolute. And why the are absolutely needed. I'd say we could have relative path defaults (from which we could eventually calculate the absolute ones), making this file optional (only needed if the developer wants to configure these paths to custom ones.

From what I remember, in the early stages of development of the framework, we were testing on the baremetal example. The makefile there groups the sources and includes based on a prefix calculated by the makefile. However, there are some sources from our framework that need to be included. Since the prefix was calculated from the relative path, we add to include our sources by providing the directory with something like this /path/to/baremetal/src/../../../bao-tests. There was also an additional problem but I'm not remembering what it was.

Is each test supposed to be compiled in a single binary? If so, e could move part of the logic to Makefiles directly: having a separate folder for each directory containing the sources and the Makefile on how to build that test. Or are suites grouped in a single binary? If so, we could have a Makefile per-suite that would gather the makefiles of each test.

Each test doesn't produce a single binary. A "battery" generates a binary. At least for now. The idea is to have a yaml defining which tests and suites will be executed, and then our framework will make sure those tests are included and the module under test is compiled to the right platform. Until now, we use the Makefile you provide (e.g. baremetal or bao-hypervisor). There are no additional makefiles from our side.

• config.tool (envisioned file to configure/use the tool to generate the remaining bao stack binaries)

Do you mean the partition configuration in the device-tree format (as used to configure the tool "normally")? I would say this is an issue I hadn't forseen because the partition binary paths are supposed to be embedded in the device tree.

This was something me and Diogo discussed as a possibility. We don't have that much knowledge on how the tool will work and its inputs and outputs. But we imagined a test case where we were testing communication from guest to guest or guest to service. For those tests (and all others, probably) we imagine that we need to provide the tool with the correct inputs to generate all the other binaries. But correct me if I'm wrong.

And finally. To address @danielRep point. Our goal was the same when we proposed this sort of structure. The idea is to have a default organization already defined in the yaml files. Most yaml configurations will only serve has configurability points for someone who doesn't want to follow the suggested/default organization. For the majority of developers, it shouldn't be necessary to configure anything.

I'm working on the PR draft. Hopefully it will be ready by the next meeting.

[josecm]

The distinction between test/suite/battery is now clear.

I think @Diogo21Costa should be the one to answer this. But from my perspective and from what we have worked until now, since we didn't have an established a definited directory tree, this yaml file help us configuring the framework tool without much hassle of going through multiple makefiles. (Particularly for me, that I originally had the framework repo outside of baremetal/bao directories).

I'm still confused. What I meant is that the toolchains to be used should be decided by the make based on the target platform. The only reason I need for defining the toolchains in a config file is if a user wants to override the default toolchains defined by the makefile.

From what I remember, in the early stages of development of the framework, we were testing on the baremetal example. The makefile there groups the sources and includes based on a prefix calculated by the makefile. However, there are some sources from our framework that need to be included. Since the prefix was calculated from the relative path, we add to include our sources by providing the directory with something like this /path/to/baremetal/src/../../../bao-tests. There was also an additional problem but I'm not remembering what it was.

I think we can adapt the makefiles themselves to support your use case. I recently changed the structure of the baremetal makefile so that it is re-usable in other projects that make use of the baremetal runtime. Take a look at how FreeRTOS uses it.

The idea is to have a default organization already defined in the yaml files. Most yaml configurations will only serve has configurability points for someone who doesn't want to follow the suggested/default organization

Are the defaults also in some yaml file? I think the default relative paths could just be hardcoded in the framework's code.

[ESCristiano]

I also don't like having the tests directory inside the src directory. I would suggest they can be side by side. To be clearer which folder would have the actual tests we would have the bao-tests submodule named test-framework. And maybe have it inside the actual tests folder:

I think the bao-tests is a relic from the past and we never updated the name. @ESCristiano will take care of that shortly. Your suggestion seems fine to me. I think it is an easy fix as we currently have those directories defined in a yaml file.

We haven't previously discussed changing the name from bao-tests to test-framework, but I do agree with that change.
Regarding the structure, I was thinking of something similar to what @josecm suggested.

MUT_Dir/

• src
• bao-tests (or the new name, test-framework)
• tests/
  • testX/
  • textY/

[danielRep]

The idea is to have a default organization already defined in the yaml files. Most yaml configurations will only serve has configurability points for someone who doesn't want to follow the suggested/default organization

Are the defaults also in some yaml file? I think the default relative paths could just be hardcoded in the framework's code.

Yeah, I also agree that those relative paths should be hardcoded and the developer has also the flexibility to use their own organization, or it will seem as configurable too if you understand what i mean.

[josecm]

One idea: maybe we could have a draft PR with this structure so we could look at it more clearly as we discuss.

[danielRep]

Thanks for opening the discussion @miguelafsilva5 .

From early discussions, we established that all directories must absolute, and thus

Can you refresh my mind on why they must be absolute. And why the are absolutely needed. I'd say we could have relative path defaults (from which we could eventually calculate the absolute ones), making this file optional (only needed if the developer wants to configure these paths to custom ones.

• src (defaul on all repos)

  • tests (this directory can be configured in the config.yaml, but this is our default suggestion)

I also don't like having the tests directory inside the src directory. I would suggest they can be side by side. To be clearer which folder would have the actual tests we would have the bao-tests submodule named test-framework. And maybe have it inside the actual tests folder:

I agree with the overall points mentioned by @josecm. My main point will be that we should try to avoid the dispersion of files and the increase on the number of configuration files. IMHO, we can force, e.g., directory paths, if the only reason to allow them to be in different places is to satisfy the "developers preference". Sorry, if I misunderstood any of the points; nevertheless, with a draft PR probably I can have a better understanding of the overall structure.