Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

depends.py should set unique configuration names for the outcome file #7290

Closed
gilles-peskine-arm opened this issue Mar 15, 2023 · 0 comments · Fixed by #8807
Closed

depends.py should set unique configuration names for the outcome file #7290

gilles-peskine-arm opened this issue Mar 15, 2023 · 0 comments · Fixed by #8807
Labels
bug component-test Test framework and CI scripts size-s Estimated task size: small (~2d)

Comments

@gilles-peskine-arm
Copy link
Contributor

We don't get good reporting through the outcome file for testing done via depends.py. The problem is that depends.py doesn't set the different configuration field, so all entries in the outcome file end up with just the all.sh component name in the configuration field. Outcome analysis assumes that there is a single outcome line for a given combination of {platform, configuration, test suite, test case name}.

This is a regression when moving from the separate perl scripts to the combined depends.py. It happened because we didn't yet have outcome files when we started work on depends.py, and we missed this addition when we finalized it.

The goal of this issue is for depends.py to give each configuration a unique name and set the environment variable MBEDTLS_TEST_CONFIGURATION, similar to what test-ref-configs.pl does and what the old scripts (curves.pl, etc.) did. The names going forward don't have to be exactly the same as the old names, they just need to be non-ambiguous and sensible.

@gilles-peskine-arm gilles-peskine-arm added bug size-s Estimated task size: small (~2d) component-test Test framework and CI scripts labels Mar 15, 2023
gilles-peskine-arm added a commit to gilles-peskine-arm/mbedtls that referenced this issue Feb 12, 2024
Set unique configuration names in the outcome file. This was lost in the
rewrite from depends-*.pl to depends.py.

Fix Mbed-TLS#7290

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
gilles-peskine-arm added a commit to gilles-peskine-arm/mbedtls that referenced this issue Feb 12, 2024
Set unique configuration names in the outcome file. This was lost in the
rewrite from depends-*.pl to depends.py.

Fix Mbed-TLS#7290

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
gilles-peskine-arm added a commit to gilles-peskine-arm/mbedtls that referenced this issue Feb 12, 2024
Set unique configuration names in the outcome file. This was lost in the
rewrite from depends-*.pl to depends.py.

Fix Mbed-TLS#7290

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
gilles-peskine-arm added a commit to gilles-peskine-arm/mbedtls that referenced this issue Feb 21, 2024
Set unique configuration names in the outcome file. This was lost in the
rewrite from depends-*.pl to depends.py.

Fix Mbed-TLS#7290

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
@github-project-automation github-project-automation bot moved this to Done in Friction Feb 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug component-test Test framework and CI scripts size-s Estimated task size: small (~2d)
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant