Skip to content

tests: drivers: console: Skip semihosting console device testing #25195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 11, 2020
Merged

tests: drivers: console: Skip semihosting console device testing #25195

merged 1 commit into from
May 11, 2020

Conversation

stephanosio
Copy link
Member

@stephanosio stephanosio commented May 11, 2020 •
edited
Loading

The semihosting console test (drivers.console.semihost) is currently
only supported on the ARM Cortex-M QEMU targets.

While running this test on real hardware targets is possible, there is
no standardised way for sanitycheck to validate its output, so we
filter by CONFIG_QEMU_TARGET to skip running it on non-QEMU targets
(including physical hardware targets with --device-testing).

Signed-off-by: Stephanos Ioannidis root@stephanos.io

Fixes #25191

@stephanosio stephanosio requested a review from nashif as a code owner May 11, 2020 10:00
@stephanosio stephanosio requested a review from erwango May 11, 2020 10:00
@stephanosio stephanosio added area: Console area: Drivers area: Sanitycheck Sanitycheck has been renamed to Twister area: Tests Issues related to a particular existing or missing test labels May 11, 2020
@stephanosio stephanosio added this to the v2.3.0 milestone May 11, 2020
@stephanosio stephanosio added the bug The issue is a bug, or the PR is fixing a bug label May 11, 2020
nashif
nashif requested changes May 11, 2020
View reviewed changes
Copy link
Member

@nashif nashif left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that build_only: true does not prevent the test from running on
the QEMU targets.

build_only is for all, so it will never run on qemu. Maybe:
filter: CONFIG_CPU_CORTEX_M and CONFIG_QEMU_TARGET

@stephanosio
Copy link
Member Author

stephanosio commented May 11, 2020 •
edited
Loading

build_only is for all, so it will never run on qemu. Maybe:
filter: CONFIG_CPU_CORTEX_M and CONFIG_QEMU_TARGET

@nashif So sanitycheck is broken, I suppose? Never mind, I was on another branch with different filter.

stephanos@DOMINUS:~/Dev/zephyr-public/zephyr$ scripts/sanitycheck -v -N -T tests/drivers/console
Renaming output directory to /home/stephanos/Dev/zephyr-public/zephyr/sanity-out.33
INFO    - JOBS: 32
INFO    - Selecting default platforms per test case
INFO    - Building initial testcase list...
INFO    - 20 test configurations selected, 476 configurations discarded due to filters.
INFO    - Adding tasks to the queue...
INFO    -  1/20 qemu_cortex_a53           tests/drivers/console/drivers.console.semihost     SKIPPED (filter)
INFO    -  2/20 native_posix              tests/drivers/console/drivers.console.uart         SKIPPED (filter)
INFO    -  3/20 qemu_xtensa               tests/drivers/console/drivers.console.uart         SKIPPED (filter)
INFO    -  4/20 qemu_cortex_r5            tests/drivers/console/drivers.console.semihost     SKIPPED (filter)
INFO    -  5/20 frdm_k64f                 tests/drivers/console/drivers.console.semihost     PASSED (build)
INFO    -  6/20 nsim_em                   tests/drivers/console/drivers.console.uart         PASSED (build)
INFO    -  7/20 frdm_k64f                 tests/drivers/console/drivers.console.uart         PASSED (build)
INFO    -  8/20 qemu_cortex_r5            tests/drivers/console/drivers.console.uart         PASSED (qemu 5.534s)
INFO    -  9/20 qemu_x86_64               tests/drivers/console/drivers.console.uart         PASSED (qemu 2.260s)
INFO    - 10/20 qemu_x86_coverage         tests/drivers/console/drivers.console.uart         PASSED (qemu 2.143s)
INFO    - 11/20 qemu_x86                  tests/drivers/console/drivers.console.uart         PASSED (qemu 2.114s)
INFO    - 12/20 qemu_x86_nommu            tests/drivers/console/drivers.console.uart         PASSED (qemu 2.110s)
INFO    - 13/20 qemu_nios2                tests/drivers/console/drivers.console.uart         PASSED (qemu 2.005s)
INFO    - 14/20 qemu_cortex_a53           tests/drivers/console/drivers.console.uart         PASSED (qemu 2.031s)
INFO    - 15/20 mps2_an385                tests/drivers/console/drivers.console.uart         PASSED (qemu 2.012s)
INFO    - 16/20 mps2_an385                tests/drivers/console/drivers.console.semihost     PASSED (qemu 2.012s)
INFO    - 17/20 qemu_riscv32              tests/drivers/console/drivers.console.uart         PASSED (qemu 2.006s)
INFO    - 18/20 qemu_cortex_m0            tests/drivers/console/drivers.console.uart         PASSED (qemu 2.013s)
INFO    - 19/20 qemu_riscv64              tests/drivers/console/drivers.console.uart         PASSED (qemu 2.007s)
INFO    - 20/20 qemu_cortex_m0            tests/drivers/console/drivers.console.semihost     PASSED (qemu 2.014s)

INFO    - 16 of 16 tests passed (100.00%), 0 failed, 4 skipped with 0 warnings in 17.19 seconds
INFO    - In total 2 test cases were executed on 15 out of total 249 platforms (6.02%)

I did filter: CONFIG_CPU_CORTEX_M and CONFIG_QEMU_TARGET initially and that didn't work either.

@nashif
Copy link
Member

nashif commented May 11, 2020

filter works fine for me:

INFO    - Adding tasks to the queue...
INFO    - 1/5 frdm_k64f                 tests/drivers/console/drivers.console.semihost     SKIPPED (filter)
INFO    - 2/5 qemu_cortex_a53           tests/drivers/console/drivers.console.semihost     SKIPPED (filter)
INFO    - 3/5 qemu_cortex_r5            tests/drivers/console/drivers.console.semihost     SKIPPED (filter)
INFO    - 4/5 mps2_an385                tests/drivers/console/drivers.console.semihost     PASSED (qemu 2.015s)
INFO    - 5/5 qemu_cortex_m0            tests/drivers/console/drivers.console.semihost     PASSED (qemu 2.013s)

@stephanosio
tests: drivers: console: Skip semihosting console device testing
e834234 
The semihosting console test (`drivers.console.semihost`) is currently
only supported on the ARM Cortex-M QEMU targets.

While running this test on real hardware targets is possible, there is
no standardised way for sanitycheck to validate its output, so we
filter by `CONFIG_QEMU_TARGET` to skip running it on non-QEMU targets
(including physical hardware targets with `--device-testing`).

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
@stephanosio stephanosio force-pushed the tests_drivers_console_no_deivce_testing branch from 2579325 to e834234 Compare May 11, 2020 12:01
@stephanosio stephanosio requested a review from nashif May 11, 2020 12:01
@carlescufi carlescufi merged commit 4b37e6c into zephyrproject-rtos:master May 11, 2020
@stephanosio stephanosio deleted the tests_drivers_console_no_deivce_testing branch May 11, 2020 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nashif nashif nashif approved these changes

@erwango erwango erwango approved these changes

Assignees
No one assigned
Labels
area: Console area: Drivers area: Sanitycheck Sanitycheck has been renamed to Twister area: Tests Issues related to a particular existing or missing test bug The issue is a bug, or the PR is fixing a bug
Projects
None yet
Milestone
v2.3.0
Development

Successfully merging this pull request may close these issues.

tests/drivers/console: drivers.console.semihost can't work
4 participants
@stephanosio @nashif @erwango @carlescufi