Intern creates one BrowserStack session for all tests

[sap1ens]

Not sure if it's a bug, suggestion or discussion, but I want to know the official opinion. Thanks!

When I specify multiple tests using functionalSuites config field and run tests using BrowserStack tunnel, only one session is created in BrowserStack (everything is treated as a single test). As a result we have a few issues:

• It's practically impossible to use BrowserStack for debugging for a large amount of tests. There is no navigation, you have to scroll over a huge log.
• Tests are not fully isolated. For example, localStorage is shared between all tests.

The question: how to force Intern framework to create a new session for every single test? It seems like it's impossible at the moment after looking at the codebase.

May be it's ok to have it as a default behaviour, but we should have an option to run all tests in separate sessions.

PS: I would assume that this behaviour is applied to other tunnels as well

[jason0x43]

Intern manages tests in a hierarchical fashion. At the lowest level are tests; these are functions that test some specific feature or function. Tests are grouped into suites. A top-level suite is created for each environment being tested. When Intern runs, it starts the top level suite for each environment in that environment, and that suite runs all of its tests and child suites.

It would be incredibly inefficient to start a new browser session for each test. We could potentially start a new session for each suite, but again it would be very inefficient if the suites were small (which is generally better for organizing tests).

Regarding isolation, that's what the beforeEach, afterEach, setup, and teardown methods are for. If local storage should be cleared before or after a test, a suite's beforeEach or afterEach method should call localStorage.clear().

There are a couple of options for debugging tests. For both unit and functional tests, Intern's reporters will tell you which test is failing. For unit tests, you can use the remote browser's debugging tools to narrow down the problem further once you know which test is having problems. For functional tests, you can use Node's debugging mode, or something like node-inspector, to step through the functional test code.

[sap1ens]

Hey @jason0x43

Thanks for your reply. Everything you mentioned is well-known, but it doesn't solve the original issue, BrowserStack (or any other tunnel) is practically useless after suite is finished. Of course it's possible to use various debugging tools, but when one single test fails I want to quickly see the execution log of that test, screenshots and video. Right now Intern doesn't allow to do that (performance reasons, fine), but I'm wondering if it's possible to introduce another option/flag/mode to use a session per test.

I would love to contribute that myself, but it seems like the current codebase doesn't have a straightforward way to implement that. Some tips are welcome :)

[jason0x43]

Hmmm...well, I really can't see a session per test being viable in most cases; it would really, really, really slow down the testing process, like, by orders of magnitude. You would lose any gains in ease of reading test results because of increased test time.

There are a couple things you can do in the short term (until there's a better method for handling large test results). For screenshots, you can write a simple reporter that will take screenshots on test failures using the takeScreenshot method on the remote object (not videos, though). For examining failing tests in more detail, you can use the grep option to re-run just the failing test(s).

[sap1ens]

It's not a problem to take a screenshot, BrowserStack provides all necessary information. The problem is following: I have to scroll through, let's say, 20 minutes of text log and screenshots to find the failed test in BrowserStack log :-/

grep helps a lot, but sometimes you can't reproduce the failure and you have to see the log from the original test run.

Also, additional time is not always a problem. For example, we run UI tests daily, not per commit, so we can afford to wait longer and have true isolation and test separation instead.

[csnover]

Have you tried using getLogsFor?

[sap1ens]

@csnover not really, but I'll try!

[sap1ens]

Btw, BrowserStack guys helped with a sample project: https://github.com/UmangSardesai/intern-browserstack/tree/master/Parallel%20Functional%20Testing/tests

It works for a very specific configuration and has some limitations, but it's a step forward. Any feedback about the "intern-parallel.js" file?

Thanks!

[mehmetg]

I am trying to do the same using the bdd interface and like you said it is not a viable solution for anything other than that specific setup. It feels like the unit test approach is getting in the way of browser testing.

[csnover]

Automatic test sharding/pooling across multiple remote instances is a feature that has been requested before, I don’t know what to do about the other part. I opened a separate ticket at #546 for the sharding. I think getLogsFor and clearLocalStorage is really the right solution for the other part of this ticket. Is there something else to be addressed here?

[sap1ens]

Agreed, #546 should solve the original issue. Thanks!

[yepitschunked]

@jason0x43 What happens if you use httpOnly cookies? There's no way to delete them in Selenium as far as I can tell. Only solution is to fire up a new session, which is impossible without some hacking.

[csnover]

@wonnage Cookies with httpOnly flag are supposed to be accessible to Selenium and deleted when calling clearCookies, though I understand support from the various drivers may be weak here since it was not mandatory in the JsonWireProtocol due to early implementation limitations (I am not sure if this is an issue with any modern implementation). A separate enhancement patch to allow sessions to be restarted programmatically would be accepted, though the what was said about this being slow is still valid, so it probably isn’t something anyone would actually want to do automatically for all tests, rather, only when you absolutely need it.