Skip Flaky Java Windows Tests #1746
Conversation
…nto flakytestremoval
|
Found most of them!
|
|
In python, I know we can implement a common decorator like |
Oh yeah I love pytest where this kind of thing is easy Even JUnit has a nice feature for it with |
Codecov Report
@@ Coverage Diff @@
## master #1746 +/- ##
=======================================
Coverage 45.38% 45.38%
=======================================
Files 64 64
Lines 2591 2591
Branches 60 60
=======================================
Hits 1176 1176
Misses 1415 1415 Help us with your feedback. Take ten seconds to tell us how you rate us. |
| @@ -1133,7 +1134,11 @@ public void testModelWithInvalidCustomPythonDependency() | |||
| @Test( | |||
| alwaysRun = true, | |||
| dependsOnMethods = {"testModelWithInvalidCustomPythonDependency"}) | |||
| public void testLoadingMemoryError() throws InterruptedException { | |||
| public void testLoadingMemoryError() throws Exception { | |||
There was a problem hiding this comment.
throws allows multiple exception types to be reported. Is there a reason why you changed the type to "Exception" instead of "InterruptedException, SkipException"?
There was a problem hiding this comment.
Thanks for the tip, yeah fixed that just now
|
@rohithkrn @mreso we're all green
|
Description
We have about ~14/192 tests that break constantly on Windows CI. There are probably more flaky ones we can fix in the future but at least with these fixed we can expect the Windows CI to turn green. The way to properly fix this would also involve mocking TS more so we don't actually have to run an inference.
We haven't had the bandwidth to fix the root cause for a few months so for now it may be best to skip and revisit as a better engineering task.
Not sure if this is the best way to skip tests using
testnginJavaso open to feedbackThe CI report will clearly state which tests passed, failed or were skipped
Type of change
Please delete options that are not relevant.
Feature/Issue validation/testing
Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.
After change
Validate Windows CI below
Test B
There's tons of examples of broken windows tests - just browse https://github.com/pytorch/serve/actions/workflows/ci_cpu.yml
Some tests that fail frequently are