Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: aggregate-timings-from-log subcmd. #149

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

ErichDonGubler
Copy link
Collaborator

No description provided.

@ErichDonGubler ErichDonGubler self-assigned this Aug 9, 2024
@ErichDonGubler ErichDonGubler added enhancement New feature or request A-webgpu-cts labels Aug 9, 2024
@ErichDonGubler ErichDonGubler force-pushed the aggregate-timings-from-log branch from f2949c5 to 74337a9 Compare August 10, 2024 03:03
@ErichDonGubler ErichDonGubler force-pushed the aggregate-timings-from-log branch from 74337a9 to f2ea983 Compare December 20, 2024 15:58
This isn't actually a valid subtest outcome. 😅 Sorry for the confusion!
TODO: Move from current incorrect flow to:

```
not_running_test -|test_start|-> running_test
	* TEST-START | <test_path>

running_test -|test_end_with_ok_or_crash|-> not_running_test
	* TEST-OK | <test_path> | took <duration>ms
	* TEST-CRASH | <test_path> | took <duration>ms
	* TEST-KNOWN-INTERMITTENT-CRASH | <test_path> | took <duration>ms

running_test -|test_end_with_unexpected_crash|-> ending_test
	* TEST-UNEXPECTED-CRASH | <test_path> | took <duration>ms
	* TEST-UNEXPECTED-CRASH | <test_path> | expected <outcome>
	  TEST-INFO expected <outcome> | took <duration>ms

ending_test -|subtest_unexpected|-> not_running_test
	* TEST-UNEXPECTED-PASS | <test_path> | <subtest> | expected <outcome>
	  TEST-INFO | expected <outcome>
	* TEST-UNEXPECTED-FAIL | <test_path> | <subtest> - <msg>

running_test -|subtest_pass_or_fail|-> running_test
	# N.B., a trailing space!
	* TEST-PASS | <test_path> | <subtest> 
	* TEST-FAIL | <test_path> | <subtest> - <msg>

# Timeouts

running_test -|subtest_timeout|-> ending_test_with_timeout

ending_test_with_timeout -|subtest_notrun|-> ending_test_with_timeout
	* TEST-UNEXPECTED-NOTRUN | <test_path | <subtest> - expected PASS
	* TEST-UNEXPECTED-NOTRUN | <test_path | <subtest> - expected FAIL
	  TEST-INFO | expected FAIL
	* TEST-KNOWN-INTERMITTENT-NOTRUN | <test_path> | <subtest>
	* TEST-UNEXPECTED-NOTRUN | <test_path> | <subtest> [msg]

ending_test_with_timeout -|test_end_with_expected_timeout|-> not_running_test
	* TEST-TIMEOUT | <test_path> | took <duration>ms

ending_test_with_timeout -|test_end_with_unexpected_timeout|-> not_running_test
	* TEST-UNEXPECTED-TIMEOUT | <test_path> | took <duration>ms
	  TEST-INFO took <duration>ms
```

TEST-UNEXPECTED
@ErichDonGubler ErichDonGubler force-pushed the aggregate-timings-from-log branch from f2ea983 to 26470b8 Compare January 28, 2025 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-webgpu-cts enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant