diff --git a/.github/workflows/antora-build.yml b/.github/workflows/antora-build.yml new file mode 100644 index 0000000..5b55482 --- /dev/null +++ b/.github/workflows/antora-build.yml @@ -0,0 +1,40 @@ +name: Publish to GitHub Pages with Lunr Search Extension +on: + push: + branches: [main] + # Allows you to run this workflow manually from the Actions tab + workflow_dispatch: +concurrency: + group: pages + cancel-in-progress: true +# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages +permissions: + contents: read + pages: write + id-token: write +jobs: + build: + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - name: Checkout repository + uses: actions/checkout@v4 + - name: Install Node.js + uses: actions/setup-node@v3 + with: + node-version: '18' + - name: Install Antora and the Antora Lunr Extension + run: npm i antora @antora/lunr-extension + - name: Generate Site + run: npx antora --fetch antora-playbook.yml + - name: Setup Pages + uses: actions/configure-pages@v3 + - name: Upload artifact + uses: actions/upload-pages-artifact@v2 + with: + path: build/site + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@v2 \ No newline at end of file diff --git a/antora-playbook.yml b/antora-playbook.yml new file mode 100644 index 0000000..5210cf8 --- /dev/null +++ b/antora-playbook.yml @@ -0,0 +1,16 @@ +site: + title: OpenMSL + url: https://persival-gmbh.github.io/asciidoc-test + start_page: OpenMSL::index.adoc +content: + sources: + - url: https://github.com/Persival-GmbH/asciidoc-test.git + branches: HEAD + edit_url: false + start_path: doc +ui: + bundle: + url: https://gitlab.com/antora/antora-ui-default/-/jobs/artifacts/HEAD/raw/build/ui-bundle.zip?job=bundle-stable + #url: https://code.asam.net/common/antora-ui/-/jobs/artifacts/main/raw/build/ui-bundle.zip?job=bundle-stable + snapshot: false + supplemental_files: ./supplemental_ui diff --git a/antora.yml b/antora.yml new file mode 100644 index 0000000..cfbc45f --- /dev/null +++ b/antora.yml @@ -0,0 +1,5 @@ +# Component name (without spaces!). All content repositories with the same component name are merged in Antora. +name: OpenMSL +version: + # Update version number accordingly. This is the version for the main branch + main: '' \ No newline at end of file diff --git a/doc/antora.yml b/doc/antora.yml new file mode 100644 index 0000000..fda2234 --- /dev/null +++ b/doc/antora.yml @@ -0,0 +1,9 @@ +# Component name (without spaces!). All content repositories with the same component name are merged in Antora. +name: OpenMSL +version: + # Update version number accordingly. This is the version for the main branch + main: '' +nav: +- modules/ROOT/nav.adoc +- modules/organization/nav.adoc +- modules/test-architecture/nav.adoc diff --git a/doc/modules/ROOT/images/envited.png b/doc/modules/ROOT/images/envited.png new file mode 100644 index 0000000..6d3bd31 Binary files /dev/null and b/doc/modules/ROOT/images/envited.png differ diff --git a/doc/modules/ROOT/nav.adoc b/doc/modules/ROOT/nav.adoc new file mode 100644 index 0000000..20d25d1 --- /dev/null +++ b/doc/modules/ROOT/nav.adoc @@ -0,0 +1,3 @@ +* xref:index.adoc[] +** xref:sub-libraries.adoc[] +** xref:related_work.adoc[] \ No newline at end of file diff --git a/doc/modules/ROOT/pages/index.adoc b/doc/modules/ROOT/pages/index.adoc new file mode 100644 index 0000000..b0c2756 --- /dev/null +++ b/doc/modules/ROOT/pages/index.adoc @@ -0,0 +1,17 @@ += Introduction + +image::envited.png[envited,800] + +ENVITED stands for Environment for Virtual Test Drive encompassing all components for virtual test and validation of Advanced Driver Assistent Systems (ADAS) including but not limited to standardized data sets, e.g. digital maps, scenario data, simulation models and their respective test and validation methods. +OpenMSL aims to connect and demonstrate the seamless interaction of projects in the domain using elaborate test pipelines, also to assure the compliance of all simulation entities with relevant standards. +It also connects to projects outside of OpenMSL, you can find a list on the xref:related_work.adoc[Related Work] page. + +All repositories are organized in xref:sub-libraries.adoc[Sub-Libraries] and each sub-library (SL) represents the best practices in the automotive industry on how to use, create and apply standard compliant simulation data and models regarding a specific topic or application area guided by expert maintainers of the ENVITED community. + +As part of these best practices, OpenMSL provides unified test architectures for simulation components. +For OSMP compliant simulation models, an xref:test-architecture:index.adoc[OSMP Test Architecture] is defined. +The architecture is applied in the template repositories of the corresponding sub-libraries. + +Learn more about the ENVITED research cluster of the Automotive Solution Center for Simulation e.V, our governance rules, contribution guidelines and our code of conduct on the xref:organization:index.adoc[Organization] page. + +We are looking forward to welcome you as member of our community! \ No newline at end of file diff --git a/doc/modules/ROOT/pages/related_work.adoc b/doc/modules/ROOT/pages/related_work.adoc new file mode 100644 index 0000000..ad1920b --- /dev/null +++ b/doc/modules/ROOT/pages/related_work.adoc @@ -0,0 +1,45 @@ +# Related Work + +## Standards + +https://www.asam.net/standards/detail/opendrive/[ASAM OpenDrive]: +Road networks are specified in the OpenDrive format. +It is used to test simulation models in a co-simulation directly in the GitHub CI pipeline. + +https://www.asam.net/standards/detail/openscenario/[ASAM OpenScenario]: +How objects move in a scene is described in the OpenScenario format. +It is used to test simulation models in a co-simulation directly in the GitHub CI pipeline. + +https://github.com/OpenSimulationInterface/open-simulation-interface[ASAM OpenSimulationInterface]: +All simulation models on OpenMSL use OSI as the standard interface for model input and model output. + +https://fmi-standard.org/[Modelica FMI]: +All models on OpenMSL are packaged as an FMU, compliant with the FMI standard. +This enables co-simulation of multiple models from different suppliers. + +https://github.com/PMSFIT/SSPTraceability[SSP Traceability]: +To specify a system for co-simulation, the SSP standard is used. +This XML standard defines, which FMUs belong to a certain system and how they are connected in a so-called System Structure Definition (SSD). +Also, parameters can be assigned to the different FMUs. + +## Other Projects + +https://github.com/esmini/esmini[esmini]: +For co-simulation of OpenMSL models directly in the GitHub CI pipeline, esmini is used to play OpenScenario files as input for the models. + +https://github.com/CATIA-Systems/FMPy[FMPy]: +To check, if an FMU build by the GitHub pipeline is conforming to the FMI standard, FMPy validate is used. + +https://github.com/modelica-tools/FMUComplianceChecker[FMU Compliance Checker]: +Alongside the aforementioned FMPy, the official FMU Compliance Checker by Modelica is also used to ensure that all FMUs comply with the FMI standard. + +https://github.com/eclipse/openmcx[OpenMCx]: +Simulations with multiple models packaged as FMUs need a co-simulation master. +To test models in the GitHub CI pipeline, OpenMCx is used to connect multiple FMUs according to a System Structure Definition (SSD). + +https://github.com/OpenSimulationInterface/osi-sensor-model-packaging[OSI Sensor Model Packaging]: +OSMP is used as a template to package all models using OSI and FMI. + +## Webinars + +NOTE: Put webinars here? \ No newline at end of file diff --git a/doc/modules/ROOT/pages/sub-libraries.adoc b/doc/modules/ROOT/pages/sub-libraries.adoc new file mode 100644 index 0000000..7fa169d --- /dev/null +++ b/doc/modules/ROOT/pages/sub-libraries.adoc @@ -0,0 +1,93 @@ += Sub-Libraries + +## SL1 - Perception Sensor Models + +This sub-library is a collection of https://github.com/OpenSimulationInterface/open-simulation-interface[OSI] compliant sensor models according to the https://github.com/OpenSimulationInterface/osi-sensor-model-packaging[OSMP] specification including a template repository +demonstrating the https://setlevel.de/assets/forschungsergebnisse/Credible-Simulation-Process-v1.0.pdf[Credible Simulation Process] by running full scale https://ssp-standard.org/[SSP] based co-simulations in the CI pipeline. + +Initiated: 2022-07-25 + +### Maintainer + +- https://github.com/LukasElster[Lukas Elster] (FZD TU Darmstadt) +- https://github.com/ClemensLinnhoff[Clemens Linnhoff] (Persival GmbH) +- https://github.com/FM-juergenW[Jürgen Wille] (FrontMod GmbH) + +### Repositories + +- https://github.com/openMSL/sl-1-0-sensor-model-repository-template[sl-1-0-sensor-model-repository-template] +- https://github.com/openMSL/sl-1-1-reflection-based-radar-object-model[sl-1-1-reflection-based-radar-object-model] +- https://github.com/openMSL/sl-1-2-reflection-based-lidar-object-model[sl-1-2-reflection-based-lidar-object-model] +- https://github.com/openMSL/sl-1-3-object-based-generic-perception-object-model[sl-1-3-object-based-generic-perception-object-model] +- https://github.com/openMSL/sl-1-4-object-based-camera-object-model[sl-1-4-object-based-camera-object-model] + +## SL2 - Traffic Participant Models + +This sub-library is a set of OSI compliant traffic participant models, which include pedestrian models, SSP based ALKS systems, automated road users and others to demonstrate closed loop simulations in combination with other sub-libraries utilizing open-source simulators such as https://github.com/esmini/esmini[esmini]. + +Initiated: Call for participation. Get engaged mailto:hello@envited.market[hello@envited.market] + +#### Maintainer + +- TBD +- TBD +- TBD + +#### Repositories + +- https://github.com/openMSL/sl-2-0-traffic-participant-model-repository-template[sl-2-0-traffic-participant-model-repository-template] + +## SL3 - Scenario Data + +This sub-library contains example scenario data following the https://www.asam.net/standards/detail/openscenario/[ASAM OpenSCENARIO] standard to provide interpretations for legislative documents such as the UN Regulation No. 157 in order to discuss them in the community. +In addition, the best practices to establish quality gates for scenario databases to clearly show the quality of scenario data are shown. + +Initiated: Call for participation. Get engaged mailto:hello@envited.market[hello@envited.market] + +### Maintainer + +- TBD (BMW AG) +- TBD +- TBD + +### Repositories + +- https://github.com/asam-oss/OSC-ALKS-scenarios[sl-3-1-osc-alks-scenarios] + +## SL4 - Static Environment Data + +The German research project https://www.gaia-x4plcaad.info/[GaiaX 4 PLC-AAD] develops quality metrics and tools to evaluate the successful integration of https://www.asam.net/standards/detail/opendrive[ASAM OpenDRIVE] maps +with e.g. https://www.khronos.org/gltf/[glTF] 3D models and their respective material data extentions. + +Initiated: Call for participation. Get engaged mailto:hello@envited.market[hello@envited.market] + +### Maintainer + +- TBD (BMW AG) +- TBD +- TBD + +### Repositories + +- In discussion + +## SL5 - Tooling + +This sub-library contains various tools to import, export, analyze and visualize co-simulation data. + +Initiated: Call for participation. Get engaged mailto:hello@envited.market[hello@envited.market] + +### Maintainer + +- TBD (Persival GmbH) +- TBD +- TBD + +### Repositories + +- https://github.com/openMSL/sl-5-1-srmd-validator[sl-5-1-srmd-validator] +- https://github.com/openMSL/sl-5-2-osi-field-checker[sl-5-2-osi-field-checker] +- https://github.com/openMSL/sl-5-3-osmp-network-proxy[sl-5-3-osmp-network-proxy] +- https://github.com/openMSL/sl-5-4-standalone-osi-trace-file-player[sl-5-4-standalone-osi-trace-file-player] +- https://github.com/openMSL/sl-5-5-osi-trace-file-player[sl-5-5-osi-trace-file-player] +- https://github.com/openMSL/sl-5-6-osi-trace-file-writer[sl-5-6-osi-trace-file-writer] \ No newline at end of file diff --git a/doc/modules/organization/nav.adoc b/doc/modules/organization/nav.adoc new file mode 100644 index 0000000..5b97690 --- /dev/null +++ b/doc/modules/organization/nav.adoc @@ -0,0 +1,2 @@ +* xref:index.adoc[] +** xref:governance_rules.adoc[] \ No newline at end of file diff --git a/doc/modules/organization/pages/governance_rules.adoc b/doc/modules/organization/pages/governance_rules.adoc new file mode 100644 index 0000000..057485f --- /dev/null +++ b/doc/modules/organization/pages/governance_rules.adoc @@ -0,0 +1,119 @@ += Governance Rules + +## §1 Purpose of ENVITED OpenMSL + +The Open-Source Model & Simulation Library is anchored in the Data Pool of the ENVITED research cluster and is operated by the Automotive Solution Center for Simulation e.V. - ASCS. +It is thus a further, independent instance in the data pool in addition to the data marketplace. +The purpose of OpenMSL is to bundle open-source material in the area of simulation-driven product lifecycle processes for highly automated vehicles, to make it openly accessible, and to sustainably enable active and cooperative further development by stakeholders. +The GitHub-based OpenMSL supports knowledge transfer and in addition the research and development of methods and processes in the mentioned area. +This includes simulation models as well as data and software applications for the preparation, processing or evaluation of models as well as documents for knowledge transfer. +The content of OpenMSL thus favors the processing of research questions and standardization approaches in the ENVITED research cluster and beyond. + +## §2 Bodies of ENVITED OpenMSL + +The governing bodies of OpenMSL are the ENVITED Technical Steering Committee (TSC), the Sub Library Maintainers (SLM), and the ASCS office. + +### 2.1 ENVITED Technical Steering Committee (TSC) + +The ENVITED Technical Steering Committee consists of 3-5 persons from among the ENVITED members, who are democratically elected by the annual ENVITED Cluster Meeting. +The TSC passes resolutions by simple majority, which are to be documented by the ASCS office. +The tasks of the TSC include the definition of the OpenMSL Governance Rules and the approval of data clusters. +In addition, the TSC has the ability to pass resolutions to remove open-source material from OpenMSL or the public that has not been actively used or developed over an extended period of time. +TSC meetings corresponding to OpenMSL are to be documented publicly with details of the persons participating, the agenda and the contents of the discussion. + +### 2.2 Sub Library Maintainers (SLMs) + +Sub Library Maintainers (SLMs) monitor the activities in the corresponding sub library and define the specific governance rules. +At least three SLMs from different ENVITED members are to be designated for each sub library. +The SLMs are democratically determined by the ENVITED members during the initialization phase and subsequently annually by the ENVITED Cluster Assembly. +If a SLM resigns during his or her term of office, the SLMs of the affected sub library can appoint a successor for the remaining term of office. +The minimum number of SLMs in the sub library must always be observed. +SLMs pass resolutions by simple majority. If no clear decision can be made, e.g., due to a tie vote, the next higher decision-making bodies are called in (SLMs → ENVITED TSC → ENVITED Assembly → ASCS Board → ASCS General Meeting). +An ENVITED member may provide more than one SLM in a sub library, but the voting right remains the same. +SLM meetings are to be documented publicly with details of the persons participating, the agenda and the contents of the discussion. + +### 2.3 Office of the ASCS + +The ASCS office is responsible for the organization, operation and documentation of the OpenMSL. +It serves as the central contact point for inquiries regarding the OpenMSL. + +## §3 Sub Libraries + +Sub Libraries bundle topic-specific content of the OpenMSL. +The ENVITED TSC decides on the number and characteristics of the sub libraries in accordance with the topic orientation and resources in the ENVITED research cluster. +New sub libraries can be requested via an application to the ASCS office and approved via the TSC.The following requirements apply for this purpose. + +### 3.1 Requirements for contributing open-source material to the OpenMSL + +Contribution of open-source material to the OpenMSL may be made under the following conditions: + +1. data contributors are clearly identifiable: + - Members in Automotive Solution Center for Simulation e.V. + - Companies / organizations stating their DUNS number + - Publicly funded projects (under confirmation by the consortium leader) +2. the open-source material to be contributed must be assignable to a sub library or be placed in a new sub library, +3. the open-source material to be contributed must be licensed under a common open-source license as listed on https://opensource.org/[opensource.org] and documented in a LICENSE file, +4. the open-source material to be contributed must be presented to the appropriate SLMs and comply with the rules of the sub library, +5. the ENVITED OpenMSL disclaimer must be agreed to and documented in a DISCLAIMER file. + +All intellectual property remains with the original contributors and is subject to the original license terms described in the LICENSE file. + +> **DISCLAIMER for GitHub libraries** +> +> This open-source material has been classified by the Sub Library Maintainers - SLMs of the sub library `` on `` as directly supporting the activities in the ENVITED research cluster. +> For this reason, the open source material is hosted by the Automotive Solution Center for Simulation e.V. (ASCS) in this Open Source Model & Simulation Library (OpenMSL). +> This open-source material was created by `` and is monitored by the SLMs of the ``. +> +> The open-source material is not directly maintained or managed by ASCS. +> The owners or maintainers of this open-source material can be found in the GitHub sub library of the same name. All intellectual property remains with the original contributors and is subject to the original license terms listed in the LICENSE file. +> +> The ASCS reserves the right to remove open-source material and associated repositories from its ENVITED OpenMSL at any time. +> ASCS, as well as the appropriate SLMs, regularly monitor the activity of open-source material to ensure that it is still relevant to the ENVITED Research Cluster. +> Open-source material that no longer meets this will be removed from the platform by decision of the ENVITED TSC. +> +> This open-source material will be made available under open-source licenses and as is. +> This is done for the protection of the creators without any warranty and under exclusion of liability for damages resulting from any use. +> +> Creators of additional open-source material that may be relevant to the ENVITED OpenMSL are welcome to contact the ASCS office at [hello@envited.market](mailto:hello@envited.market) + +### 3.2 Repository naming scheme + +Each repository within OpenMSL is assigned a unique identifier. +This identifier combines the ID of the sub-library with a consecutive repository ID within the sub-library conforming to the schema: + +`sl--` + +This identifier string is the prefix of every repository within OpenMSL. +The second part is an individual name for the material contained in the repository. +The name shall give concise information about the repository's contents. +The general repository naming scheme therefore reads: + +`sl---` + +## §4 Implementation phases of the OpenMSL + +Implementation phases define different stages in which data sub libraries or parts of them can be located. + +### 4.1 Initialization Phase + +In the initialization phase, the new sub library and its model(s) or data are first proposed or presented. +All ENVITED members can participate in this phase after invitation by the ASCS and express their interest as SLM. +The ENVITED TSC decides on the initiation of the new sub library. +Before filling the repository, the SLMs define governance rules on how and under which structure and quality metrics the data are to be entered. + +### 4.2 Active Phase + +In the active phase, after approval by the SLMs, the open-source data is entered into the repository. +This is usually done by the data providers, the SLMs, or the ASCS office. +The active phase allows public access to the data sets as well as the comment function regarding applicability, functionality and necessary updates. +Change proposals (pull requests) can be decided or accepted by the SLMs. + +### 4.3 Project Phase + +If more extensive measures are required to update or expand open-source data in a sub library, the project phase provides a formal definition of an implementation project. +Here, a concrete project plan with motivation, objectives, and implementation time frame must be defined with the involvement of participating or supporting ENVITED members. +The initiation is done by the SLMs. +All ENVITED members are informed about the initiation of a project phase by the ASCS office and given the opportunity to participate. +In addition to resources / funds of the members, third-party funds as well as funds from the ENVITED Research Cluster can be used in the project phase. +The latter also applies to the awarding of contracts to external service providers, whereby the awarding guidelines of the ASCS must be observed here. +The final decision on the allocation of funds is made by the ASCS Executive Board after recommendation by the ENVITED TSC. diff --git a/doc/modules/organization/pages/index.adoc b/doc/modules/organization/pages/index.adoc new file mode 100644 index 0000000..2e23d4b --- /dev/null +++ b/doc/modules/organization/pages/index.adoc @@ -0,0 +1,43 @@ += Organization + +This GitHub organization is maintained by the https://www.asc-s.de/[Automotive Solution Center for Simulation e.V.] - a non-profit industry club registered in Germany. +The main focus of its activities is the concentration of expertise from automotive and supply industry, software and hardware manufacturers, engineering service providers and research institutes. +The asc(s brings together key leaders from the industry with the best and most renowned scientists in the field of virtual vehicle development. Its members benefit from a worldwide unique research +and development landscape with special focus on high-performance computing, driving simulation and digitalization. + +The https://envited.market/[ENVITED Research Cluster] is an initiative to create the Environment for Virtual Test Drive in a collaborative way, defining a methodology for a standard data based virtual proof of validation. +Virtual validation is becoming an essential part of development processes for highly automated driving (HAD) functions. Standards for model and system interchange are vital for cross-company and cross-domain virtual +integration and simulation of HAD functions. + + +## ASC(S e.V. Governance Rules + +The collaboration of contributors in this GitHub organization is organized under the umbrella of the Automotive Solution Center for Simulation e.V. +In this community the general xref:governance_rules.adoc[Governance Rules] needed to fulfill industry compliance regulations are set. + +## Code of Conduct + +As a community we need to define how we want to interact with each other respectfully and make sure that we create an inclusive environment that encourages contributions. +For this reason OpenMSL has a clear https://github.com/openMSL/.github/blob/main/CODE_OF_CONDUCT.md[Code of Conduct] that applies to everyone. Read https://opensource.guide/code-of-conduct/[here] why we need a code of conduct. +A modified version of https://www.contributor-covenant.org/[this] document is currently in use. + +## Contributing Guidelines + +Thanks for your interest in contributing! There are many ways to contribute to this project. +Get started https://github.com/openMSL/.github/blob/main/CONTRIBUTING.md[here]. + +## PR and Issue Templates + +This repository also defines the standard issue and pull request templates valid for the overall organization. + +## Licensing + +The open-source material to be contributed must be licensed under a common open-source license as listed on https://opensource.org/[opensource.org] and documented in a LICENSE file. + +The license tells you what rights you have as a developer, provided by the copyright holder. +It is important that the contributor fully understands the licensing rights and agrees to them. Sometimes the copyright holder isn’t the contributor, such as when the contributor is doing work on behalf of a company. + +## Public organization profile + +The profile information is described in https://github.com/openMSL/.github/blob/main/profile/README.md[.github/profile/README.md] and displayed on the organization main page. +Read the corresponding GitHub https://docs.github.com/en/organizations/collaborating-with-groups-in-organizations/customizing-your-organizations-profile[documentation] for more information on organization profiles. diff --git a/doc/modules/test-architecture/images/cl2_test_esmini.svg b/doc/modules/test-architecture/images/cl2_test_esmini.svg new file mode 100644 index 0000000..c2c0722 --- /dev/null +++ b/doc/modules/test-architecture/images/cl2_test_esmini.svg @@ -0,0 +1,176 @@ + +OpenMCx Co-SimulationScenario DatabaseModel FMUesmini FMUProcessing FMUOpenScenario FileOSI DataOSI Data diff --git a/doc/modules/test-architecture/images/cl2_test_tracefile.svg b/doc/modules/test-architecture/images/cl2_test_tracefile.svg new file mode 100644 index 0000000..4a9c110 --- /dev/null +++ b/doc/modules/test-architecture/images/cl2_test_tracefile.svg @@ -0,0 +1,189 @@ + +OpenMCx Co-SimulationTrace File DatabaseModel FMUTrace File PlayerFMUProcessing FMUInput Trace FilesOSI DataOSI Data diff --git a/doc/modules/test-architecture/images/cl3_test.svg b/doc/modules/test-architecture/images/cl3_test.svg new file mode 100644 index 0000000..ea332c4 --- /dev/null +++ b/doc/modules/test-architecture/images/cl3_test.svg @@ -0,0 +1,454 @@ + +Post-ProcessingOpenMCx Co-SimulationTrace File DatabasePython AnalysisTrace File WriterFMUModel FMUFMUTrace File PlayerInput Trace Files Expectation Trace Files OSI Trace FileOSI DataOSI Data diff --git a/doc/modules/test-architecture/images/credibility_assessment_level.png b/doc/modules/test-architecture/images/credibility_assessment_level.png new file mode 100644 index 0000000..2a4d29d Binary files /dev/null and b/doc/modules/test-architecture/images/credibility_assessment_level.png differ diff --git a/doc/modules/test-architecture/images/system_structure.png b/doc/modules/test-architecture/images/system_structure.png new file mode 100644 index 0000000..491346c Binary files /dev/null and b/doc/modules/test-architecture/images/system_structure.png differ diff --git a/doc/modules/test-architecture/nav.adoc b/doc/modules/test-architecture/nav.adoc new file mode 100644 index 0000000..ca582c6 --- /dev/null +++ b/doc/modules/test-architecture/nav.adoc @@ -0,0 +1,5 @@ +* xref:index.adoc[] +** xref:cl0.adoc[] +** xref:cl1.adoc[] +** xref:cl2.adoc[] +** xref:cl3.adoc[] \ No newline at end of file diff --git a/doc/modules/test-architecture/pages/cl0.adoc b/doc/modules/test-architecture/pages/cl0.adoc new file mode 100644 index 0000000..d1eb7e0 --- /dev/null +++ b/doc/modules/test-architecture/pages/cl0.adoc @@ -0,0 +1,9 @@ += CL 0: License Check + +For credibility assessment level 0 the software licenses are checked. +A GitHub Action checks every file containing code for an SPDX license identifier. +The identified licenses are verified against a list of approved open source licenses. +Approved licenses are: Apache-2.0, BSD-2-Clause, BSD-3-Clause, MIT, MPL-2.0. +Please contact the maintainers if you have other requirements regarding open source licenses. + +An implementation of the GitHub action for this credibility assessment level can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/.github/workflows/cl0.yml[sensor model template repository]. \ No newline at end of file diff --git a/doc/modules/test-architecture/pages/cl1.adoc b/doc/modules/test-architecture/pages/cl1.adoc new file mode 100644 index 0000000..0e8909a --- /dev/null +++ b/doc/modules/test-architecture/pages/cl1.adoc @@ -0,0 +1,62 @@ += CL 1: Code Verification + +Credibility assessment level 1 contains multiple different tests for code quality. +In the first test, linters are used to ensure general quality of code and documentation. +In the example linked below, linters for C++ code, markdown files, and xml files such as SRMD and modelDescription are implemented. +Further linters have to be added, if additional programming languages are used. +In the second test, the model is build using cmake. +The third test category runs unit tests defined for the individual model. + +An implementation of the GitHub action for this credibility assessment level can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/.github/workflows/cl1.yml[sensor model template repository]. + +## CL 1.1: Linter + +These tests consist of linters checking the formatting against style guides for code, markdown and xml. + +### C++ Linter + +C++ code is checked with clang according to the clang-format and clang-tidy files in the repository. +It is mandatory for every code repository to provide these config files. +As a best practice, https://github.com/eklitzke/clang-format-all[clang-format-all] can be used to automatically format all C++ files in the repository. +Be aware, that formatting might break functionality, e.g. by a different order of includes. +Be sure to check after formatting. + +An implementation of the corresponding GitHub action can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/.github/workflows/cpp-linter.yml[sensor model template repository]. + +### Markdown Linter + +Markdown files are checked by a markdown linter according to https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md[these] basic rules. + +An implementation of the corresponding GitHub action can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/.github/workflows/markdown.yml[sensor model template repository]. +It also contains an example https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/.github/workflows/markdownlint.json[markdownlint.json] specifying variations of the applied rules. + +### SRMD Validator + +Every model repository SHALL include an https://pmsfit.github.io/SSPTraceability/master/#_srmd[SRMD file] conforming to the https://pmsfit.github.io/SSPTraceability/master/[SSP tracability] process. +This file is checked against the https://github.com/PMSFIT/SSPTraceability[SRMD XML schema]. + +## CL 1.2: Build + +These tests check if the model can be build and the built FMU is compliant with the https://github.com/modelica/fmi-standard[FMI standard]. + +### Cmake + +This test checks if cmake runs without errors and the model can be built with all necessary dependencies, such as OSI with Protobuf. + +An implementation of the corresponding GitHub action can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/.github/workflows/build.yml[sensor model template repository]. + +### FMU Compliance Checker + +The official https://github.com/modelica-tools/FMUComplianceChecker[FMU Compliance Checker] by Modelica is used to check if the built FMU is compliant with the https://github.com/modelica/fmi-standard[FMI standard]. +This includes checking the binaries as well as included xml files such as the modelDescription. + +Since the FMU Compliance Checker does not seem to be further maintained, https://github.com/CATIA-Systems/FMPy[FMPy] is used as a second validation tool. + +## CL 1.3 Unit Tests + +Models SHALL include individual unit tests. +The tests SHALL be located in a folder _test/unit_ which MUST include the applied unit test library (e.g. https://github.com/google/googletest[GoogleTest]). +Unit tests have to be custom to the implementation of the individual model. +Therefore, no standard unit tests are part of this test pipeline. + +However, some examples can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/unit[sensor model template repository]. \ No newline at end of file diff --git a/doc/modules/test-architecture/pages/cl2.adoc b/doc/modules/test-architecture/pages/cl2.adoc new file mode 100644 index 0000000..352baac --- /dev/null +++ b/doc/modules/test-architecture/pages/cl2.adoc @@ -0,0 +1,86 @@ +# CL 2: Qualitative Verification + +For credibility assessment level 2, the model is build and applied in a co-simulation. +https://github.com/eclipse/openmcx[OpenMCx] is used as a co-simulation platform. +There are two possibilities for generating the input for the model. + +1. The model input is provided by an OSI trace file. + This trace file is read by an https://gitlab.com/persival-open-source/open-simulation-interface/osi-trace-file-player[OSI trace file player FMU] which is connected to the model FMU. + The connection interface is OSI with data according to the https://opensimulationinterface.github.io/osi-antora-generator/asamosi/latest/sensor-model/spec/model_types.html[model type]. + In case of a sensor model, the connection interface is osi3::SensorView. + For other models it can be osi3::SensorData, osi3::TrafficCommand etc. + The co-simulation setup of this approach is depicted in the first image below. +2. The model input is provided by a scenario engine. + The OpenScenario player https://github.com/esmini/esmini[esmini] is used to play an OpenScenario file within an OSMP FMU and output osi3::SensorView data for the model. + The co-simulation setup of this approach is depicted in the second image below. + In a future adaptation, the esmini FMU will be enabled to receive osi3::TrafficCommand data to test further model types. + +[.float-group] +-- +[.left] +.Tracefile test setup for CL 2 +[#img-cl2_test_tracefile,link=_images/cl2_test_tracefile.svg] +image::cl2_test_tracefile.svg[Tracefile test setup for CL 2,350] + +[.right] +.Esmini test setup for CL 2 +[#img-cl2_test_esmini,link=_images/cl2_test_esmini.svg] +image::cl2_test_esmini.svg[Esmini test setup for CL 2,350] +-- + +The output of the model can be used in three different ways: + +1. It can be disregarded, just to test if the model runs at all. This is referred to as a smoke test. An example can be found https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/integration/001_smoke_test_tracefile[here]. +2. It can be connected to a processing FMU evaluating the model output. This way, the model output is tested "on-the-fly" step by step during the co-simulation. This can for example be an interface test, e.g. https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/integration/003_output_osi_fields[here]. +3. A https://github.com/Persival-GmbH/osi-sensordata-trace-file-writer[trace file writer] can be used as a processing FMU to generate a trace file from the model output during the co-simulation. +This trace file will be supplied as a GitHub Action Artifact, so it can be downloaded for offline analysis. +But the trace file can also be directly analyzed in the pipeline by a python script. +This functionality is part of CL3. + +All tests in CL 2 are considered integration tests, since they all require a co-simulation with at least one other FMU. +Therefore, every test is located in an individual folder in _test/integration_. +The individual test folder SHALL follow the naming scheme "xxx_short_description", where xxx is a three digit consecutive number. +The folder SHALL contain a https://ssp-standard.org/publications/SSP10RC1/SystemStructureAndParameterization10RC1.pdf[system structure definition file (.ssd)]. +In this file, the utilized model input (trace file player or esmini) as well as the processing FMU (evaluation or trace file writer) are specified. +The test folder SHALL additionally contain a README.md file, which describes the test system, scenario and pass/fail criterion. +A https://github.com/openMSL/.github/blob/main/doc/integration_test_readme_template.md[template] is provided with the corresponding sections for this readme file. +Other simulation artefacts such as the trace file or scenario to be played as well as auxiliary files for the evaluation FMU and a python analysis script are also to be placed in that test folder. +Example implementations in the _test/integration_ folder can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/integration/[sensor model template repository]. + +An implementation of the GitHub action for this credibility assessment level can also be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/.github/workflows/cl2.yml[sensor model template repository]. + +The three different test methods are described in more detail in the following sections. + +## CL 2.1 Smoke Tests + +Typically in software testing, smoke test are done by just running the system under test stand-alone. +But since OSMP models always require a model input, smoke test are also located in the integration test folder. +For the smoke tests, the model is employed in a co-simulation and fed with inputs. +The model is just connected either to a trace file player or a scenario engine to generate the model input. +The output of the model is not used. +The model SHALL go through the tests without any run time errors. + +An example implementation of this test can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/integration/001_smoke_test[sensor model template repository]. + +## CL 2.2 Interface Tests + +In this test category, in addition to a trace file player or scenario engine, the model output is connected to an evaluation FMU. +With this setup, the model output is tested against various criteria. + +### OSI Field Check + +In this test, the model output is connected to the https://github.com/openMSL/sl-1-5-sensor-model-testing/tree/main/src/osi-field-checker[OSI Field Checker] FMU provided in this repository. +It checks, if the OSI fields required for the model under test are filled in the model output. +The required output fields are specified in a txt file, where every line is one OSI field. +If a field is missing in the output, it will be shown as an annotation in the GitHub action. +More detailed information about the OSI Field Checker can be found in the respective https://github.com/openMSL/sl-1-5-sensor-model-testing/tree/main/src/osi-field-checker#readme[Readme]. + +An example implementation of this test can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/integration/002_output_osi_fields[sensor model template repository]. + +### Value Range Check + +(not yet implemented) + +Furthermore, the value ranges of the output parameters are tested. +E.g. if a lidar sensor has intensity outputs in the interval [0, 100], the sensor model SHALL NOT output any values outside of this range. +These tests on value ranges might also include the timing of the model. +Additional test can be performed on the SensorViewConfigRequest during the initialization of the model FMU, if implemented. diff --git a/doc/modules/test-architecture/pages/cl3.adoc b/doc/modules/test-architecture/pages/cl3.adoc new file mode 100644 index 0000000..6f77910 --- /dev/null +++ b/doc/modules/test-architecture/pages/cl3.adoc @@ -0,0 +1,45 @@ +# CL 3: Quantitative Verification + +For credibility assessment level 3, the model is again employed in a co-simulation with a trace file player or scenario engine. +During the simulation, a trace file is generated containing the model output of every simulation time step. +This is done by connecting a https://github.com/Persival-GmbH/osi-sensordata-trace-file-writer[trace file writer] to the model output in the SSP specification of the test. +The trace file writer is pre-built in every test environment. +The FMU is located at _/tmp/tracefile_writer_fmu/osi-sensordata-trace-file-writer.fmu_. +After the co-simulation is finished, the GitHub action scans the test folder for python files. +If it finds one, the python script is called with the path to the just generated trace file as an argument. +The file name of the python file does not matter. +However, if there is more than one python file in the directory, the analysis will not be performed since it is unclear, which file to use. +If supplementary files are necessary for the analysis, they are to be placed in a subfolder. + +The purpose is to test the proper implementation of the model code in terms of its quantifiable requirements. +E.g. the sensor is rotated around all 3 axis, and it is evaluated, if the coordinate transformations in the model are correct. +This can be achieved in two ways: + +1. The simulation trace file is analyzed by a custom standalone python script. +2. The simulation trace file is compared to an expectation trace file with a python script. + +The general setup of this test level is depicted in the following image and both approaches are described in more detail in the following sections. + +.Test setup for credibility assessment level 3 +[#img-cl3_test,link=_images/cl3_test.svg] +image::cl3_test.svg[Test setup for credibility assessment level 3,600] + +## CL 3.1 Standalone Trace File Analysis + +The OSI trace file generated by the co-simulation is read in by a python script after the simulation has finished. +The python script can perform a variety of tests on the trace file, depending on the specific scenario and use-case. +For example, the scenario places an object at a certain distance inside the field of view of a sensor. +The python script can analyze the sensor model output and check, if the object was detected at the expected distance. +Another example would be to place an object outside of the field of view and check, that it was not detected. +Furthermore, distribution functions for detections or expected number of detections on an object can be checked. + +An example implementation of this test can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/integration/004_tracefile_analysis[sensor model template repository]. + +## CL 3.2 Trace File Comparison + +(not yet implemented) + +This test is similar to 3.1. +But instead of analyzing the simulation trace file with certain criteria, the python script compares the model output to an expectation trace files. +The expectation trace file can either contain other simulated data or even measurement data from a real system to compare the model under test to. +This comparison is done with appropriate metrics. +Required thresholds for these metrics determine the pass/fail criterion of these tests. \ No newline at end of file diff --git a/doc/modules/test-architecture/pages/index.adoc b/doc/modules/test-architecture/pages/index.adoc new file mode 100644 index 0000000..c1a26b2 --- /dev/null +++ b/doc/modules/test-architecture/pages/index.adoc @@ -0,0 +1,20 @@ +:fn-disclaimer: footnote:Ahmann2022[M. Ahmann, V. T. Le, F. Eichenseer, F. Steimann, and M. Benedikt, “Towards Continuous Simulation Credibility Assessment,” Proceedings of Asian Modelica Conference 2022, Tokyo, Japan, Nov. 2022.] + += OSMP Test Architecture + +The OSMP Test Architecture is designed to test any simulation model complying to the https://github.com/OpenSimulationInterface/osi-sensor-model-packaging[OSMP] specification. +A variety of model types from environmental effect models over sensor models to traffic participant models can be tested with this architecture. +The architecture is divided into 3 main test levels based on the credibility assessment level definition by Ahmann et al.{fn-disclaimer}. +Because the test architecture is specifically designed for OpenMSL open source models, a license check is needed. +The license check is preceding the 3 main credibility assessment levels as level 0. +This yields the following credibility assessment level line-up: + +- CL 0: xref:cl0.adoc[License Check] +- CL 1: xref:cl1.adoc[Code Verification] +- CL 2: xref:cl2.adoc[Qualitative Verification] +- CL 3: xref:cl3.adoc[Quantitative Verification] + +Every credibility assessment level is tested with a separate GitHub action and is further described in the following sections. + +.Discrete credibility assessment level +image::credibility_assessment_level.png[Discrete credibility assessment level,800] diff --git a/doc/modules/test-architecture/pages/osmp_test_architecture.md b/doc/modules/test-architecture/pages/osmp_test_architecture.md new file mode 100644 index 0000000..180dc5c --- /dev/null +++ b/doc/modules/test-architecture/pages/osmp_test_architecture.md @@ -0,0 +1,8 @@ + +## CL X: Validation + +(not yet implemented)
+A future credibility assessment level might constitute the validation of the model against real world data. +In pre-defined validation scenarios, the model output is compared to measured sensor data with suitable metrics. +A similar test setup as in CL 3 might be applied. + diff --git a/supplemental_ui/partials/footer-content.hbs b/supplemental_ui/partials/footer-content.hbs new file mode 100644 index 0000000..c0854b3 --- /dev/null +++ b/supplemental_ui/partials/footer-content.hbs @@ -0,0 +1,4 @@ +
+

OpenMSL documentation is licensed under the terms of the MPL-2.0 license.

+
+ diff --git a/supplemental_ui/partials/header-content.hbs b/supplemental_ui/partials/header-content.hbs new file mode 100644 index 0000000..64dd188 --- /dev/null +++ b/supplemental_ui/partials/header-content.hbs @@ -0,0 +1,19 @@ +
+ +