SauceLabs scripts seem brittle

[benlesh]

Every time we merge new code something different (and completely arbitrary) is red in the SauceLabs out put.

I like that we're testing in multiple browsers, but is there a way to get those to rerun? I think right now @kwonoj is the only one that can rerun the tests.

Longer term, is there a less brittle solution?

[kwonoj]

To re execute test, simply retrigger travis would makes it work. I can do it as I have permission to travis to retrigger, anyone else have permission in this repo can do.

Only caveat is it's per-test, not per-browser, once it's triggered, it's against all browsers. I haven't looked into way to trigger certain browser only (maybe in saucelab dashboard allows those, but for saucelabs I don't have permission to access for this repo)

Mostly test failures are occurring due to some unknown timeout, or emulator based failure (mobile browers). I'm looking way to enhance those behaviors as well in meantime.

[kwonoj]

Couple of test cases are seems bit fragile on browser environment, mostly seems related with kind of timing issues occurs on async tests. Will look into those as well to see if test case can be more bulletproof.

https://saucelabs.com/tests/e2b25337f85c4f14b346a65d2a040a47
https://saucelabs.com/tests/06b02acae8d74918ab2cf3fc89492d2a

[kwonoj]

No updates yet, just planned progress - I'm trying to utilize mochify by introducing changes to accept local custom ui instead of handmade testing configurations. Will post PR once it's available.

[kwonoj]

#1775 might give chances to remove local custom interfaces which is major roadblock to adapt test runner.

[kwonoj]

I got access to https://www.browserstack.com/ and evaluating it to see if it's connector works reliably than sauce labs.

[kwonoj]

working changes on browserstack leaves at https://github.com/kwonoj/RxJS/tree/test-browserstack, but hesitate to create PR at this moment since #2261 pretty much blocks lot of browser testing stability. Not sure how to proceed between integrate browserstack first, or try to fix issue first. will think around this weekend.

[kwonoj]

I'm going to close this for now, as our test ergonomics is quite complex to achieve this goal reliably. there are few prior tasks need to be done and it'll be dealt separately / gradually.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.