Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sanitycheck generated incorrect acrn.xml on acrn platform #23309

Closed
chen-png opened this issue Mar 6, 2020 · 5 comments · Fixed by #23513
Closed

Sanitycheck generated incorrect acrn.xml on acrn platform #23309

chen-png opened this issue Mar 6, 2020 · 5 comments · Fixed by #23513
Assignees
Labels
area: Sanitycheck Sanitycheck has been renamed to Twister bug The issue is a bug, or the PR is fixing a bug priority: low Low impact/importance bug

Comments

@chen-png
Copy link
Collaborator

chen-png commented Mar 6, 2020

Describe the bug
Running sanitycheck on acrn platform, the generated two files acrn.xml and sanitycheck.xml are not same in sanity-out directory, the test results in these two files are different.

the result in sanitycheck.xml should be right, it's the same with sanitycheck.log.

To Reproduce
Steps to reproduce the behavior:

  1. sanitycheck -p acrn --device-testing --device-serial /dev/ttyUSB1 -T tests
    --west-flash="acrn_up2.sh"
  2. See error

Screenshots or console output

  1. test results in acrn.xml
    testsuites><testsuite errors="49" failures="0" name="Sanitycheck" skipped="0" tests="51" time="1458.823267"

  2. test results in sanitycheck.xml
    testsuites><testsuite errors="0" failures="7" name="Sanitycheck" skip="20" tests="125" time="1458.823267"

Environment (please complete the following information):

  • OS: fedora28
  • Toolchain: zephyr-sdk-0.11.1
  • Commit ID: 46330f1
@chen-png chen-png added the bug The issue is a bug, or the PR is fixing a bug label Mar 6, 2020
@nashif
Copy link
Member

nashif commented Mar 9, 2020

@chen-png this issue is very difficult to reproduce, I do not have the same setup with acrn in place, Do you get this issue with boards other than acrn?

@nashif nashif added priority: low Low impact/importance bug area: Sanitycheck Sanitycheck has been renamed to Twister labels Mar 9, 2020
@chen-png
Copy link
Collaborator Author

Do you get this issue with boards other than acrn?

no, other boards are good.

@chen-png
Copy link
Collaborator Author

for acrn, I found that in monitor_serial function, when invoking ser.readline() to get the serial output, the returned line value couldn't be correctly matched in test harness handle function, so sanitycheck couldn't get the pass/fail result foo each test unit, causing the generated acrn.xml error.

@chen-png
Copy link
Collaborator Author

the normal output should like this, the serial should read line by line.(i add a print to serial)

DEBUG - DEVICE: ===========================================
line: ================================================
DEBUG - DEVICE: starting test - test_fifo_cancel_wait
line: starting test - test_fifo_cancel_wait
DEBUG - DEVICE: PASS - test_fifo_cancel_wait
line: PASS - test_fifo_cancel_wait
DEBUG - DEVICE: ======================================
line: ==============================================
DEBUG - DEVICE: starting test - test_fifo_is_empty_thread
line: starting test - test_fifo_is_empty_thread
DEBUG - DEVICE: PASS - test_fifo_is_empty_thread
line: PASS - test_fifo_is_empty_thread

but for acrn output, it likes this, it didn't get the output line by line.
DEBUG - DEVICE:
line:
starting test - test_fifo_cancel_wait
starting test - test_fifo_cancel_wait
PASS - test_fifo_cancel_wait
PASS - test_fifo_cancel_wait

starting test - test_fifo_is_empty_thread
starting test - test_fifo_is_empty_thread
PASS - test_fifo_is_empty_thread
PASS - test_fifo_is_empty_thread

@chen-png
Copy link
Collaborator Author

for acrn, each serial line has a '\r' char in front of the output, likes this:
b'\r PASS - test_fifo_is_empty_thread \r\n'

after the decode('utf-8', 'ignore') and rstrip(), the result likes this:
\r PASS - test_fifo_is_empty_thread

but this can't be match successfully in result_re.match in harness.py, this \r will influence the match result.
also in the generated sanitycheck.log, in each serial output, there is a ^M char.

so we should delete this \r char in serial output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Sanitycheck Sanitycheck has been renamed to Twister bug The issue is a bug, or the PR is fixing a bug priority: low Low impact/importance bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants