Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Fix spectest failure output #548

Merged
merged 2 commits into from
Sep 24, 2020
Merged

test: Fix spectest failure output #548

merged 2 commits into from
Sep 24, 2020

Conversation

gumb0
Copy link
Collaborator

@gumb0 gumb0 commented Sep 23, 2020

Before:

Running tests from /home/andrei/dev/wasm-spec/test/core/json/func.json
........................................................................s.FF......ssssssssss..............................ssssssssssss
Line 350: assert_return FAILED Incorrect returned value. Expected: 0 (0x0) Actual: 2a (0x2a)
Line 351: assert_return FAILED Incorrect returned value. Expected: 0 (0x0) Actual: 2a (0x2a)
134 tests ran from func.json.
  PASSED 109, FAILED 2, SKIPPED 23.

After:

Running tests from /home/andrei/dev/wasm-spec/test/core/json/func.json
........................................................................s.FF......ssssssssss..............................ssssssssssss
Line 350: assert_return FAILED Incorrect returned value. Expected: 0 (0x0) Actual: 42 (0x2a)
Line 351: assert_return FAILED Incorrect returned value. Expected: 0 (0x0) Actual: 42 (0x2a)
134 tests ran from func.json.
  PASSED 109, FAILED 2, SKIPPED 23.

@codecov
Copy link

codecov bot commented Sep 23, 2020

Codecov Report

Merging #548 into master will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #548   +/-   ##
=======================================
  Coverage   98.74%   98.74%           
=======================================
  Files          58       58           
  Lines        8765     8765           
=======================================
  Hits         8655     8655           
  Misses        110      110           

@gumb0 gumb0 requested review from chfast and axic September 23, 2020 16:01
Copy link
Collaborator

@chfast chfast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a regression test? In CTest you can check output.

@axic
Copy link
Member

axic commented Sep 23, 2020

And make sure the expected is non-zero so we can see the hex/dec there too.

@gumb0 gumb0 requested a review from chfast September 24, 2020 10:17
set_tests_properties(
fizzy/smoketests/spectests/failures-output
PROPERTIES
PASS_REGULAR_EXPRESSION "FAILED Incorrect returned value\\. Expected: 2 \\(0x2\\) Actual: 10 \\(0xa\\)"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
PASS_REGULAR_EXPRESSION "FAILED Incorrect returned value\\. Expected: 2 \\(0x2\\) Actual: 10 \\(0xa\\)"
PASS_REGULAR_EXPRESSION "FAILED Incorrect returned value\\. Expected: 12 \\(0xc\\) Actual: 10 \\(0xa\\)"

So that both expected and actual are >9.

@gumb0 gumb0 requested a review from axic September 24, 2020 12:13
@gumb0 gumb0 merged commit 4dbe155 into master Sep 24, 2020
@gumb0 gumb0 deleted the spectest-failure-output branch September 24, 2020 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants