Test cases don't actually run when using :js-env :chrome (or :firefox)

[jffry]

When running the tests in Chrome or Firefox, the browsers open briefly, but no actual tests get executed. The same tests work just fine in PhantomJS.

Here's a small repo demonstrating the issue: https://github.com/SparkFund/cljs-test-problem, and here's the output I see when I attempt to run the tests in Chrome (latest Chrome, latest MacOS, problem can be reproduced by colleagues):

$ node --version
v6.10.0

$ npm install --global karma karma-chrome-launcher karma-firefox-launcher karma-cljs-test
...

$ boot test-in-chrome
Adding: ([adzerk/boot-cljs "1.7.228-2"] [doo "0.1.7"]) to :dependencies
Compiling ClojureScript...
• cljs_test/generated_test_suite.js
Running cljs tests...

;; ======================================================================
;; Testing with Chrome:

[doo] Created karma conf file: /var/folders/7m/8846n6kn08v2cw2m09hdfs540000gn/T/karma_conf4885380502268738690.js
[doo] Started karma server
25 05 2017 15:10:21.314:WARN [watcher]: All files matched by "/Users/jeff/.boot/cache/tmp/Users/jeff/spark/cljs-test-problem/86v/oz0m8x/cljs_test/generated_test_suite.out/**/*.js" were excluded or matched by prior matchers.
25 05 2017 15:10:21.317:WARN [watcher]: All files matched by "/Users/jeff/.boot/cache/tmp/Users/jeff/spark/cljs-test-problem/86v/oz0m8x/cljs_test/generated_test_suite.out/*.js" were excluded or matched by prior matchers.
[doo] Started karma run
Close Karma run
Shutdown Karma Server

I attempted to inspect either the generated Karma config file or the contents of generated_test_suite.out but it appears those temp files get cleaned up post-haste and are no longer around to look at.

As far as I know I'm using the most recent versions of all my deps. Please let me know if there's more I can do to help troubleshoot this, thanks!

[crisptrutski]

Thanks for the repo, there are probably a few things at play here, and the chief root cause is that the karma server is out of process and therefore async unlike phantom.

1. A silly "backwards boolean" bug in the latest release caused the update-fs? parameter (which is force-set to true for karma runs) to have the opposite behaviour (hides filesystem mutations from subsequent and async tasks). Worked around this by rebuilding the library locally.

2. The process is exiting too early, wiping out the fileset before karma has a chance to run.

Tried a quick workaround for this: boot test-in-chrome wait, and with (1) above this resulted in tests actually running:

[doo] Started karma run
WARN [watcher]: All files matched by "..../cljs-test-problem/1030/rzams8/cljs_test/generated_test_suite.out/**/*.js" were excluded or matched by prior matchers.
WARN [watcher]: All files matched by "..../cljs-test-problem/1030/rzams8/cljs_test/generated_test_suite.out/*.js" were excluded or matched by prior matchers.
WARN [watcher]: All files matched by "..../cljs-test-problem/1030/rzams8/cljs_test/generated_test_suite.out/**/*.js" were excluded or matched by prior matchers.
WARN [watcher]: All files matched by "..../cljs-test-problem/1030/rzams8/cljs_test/generated_test_suite.out/*.js" were excluded or matched by prior matchers.
WARN [web-server]: 404: /generated_test_suite.out/cljs_deps.js
LOG: 'Testing sparkfund.ui-test'
Chrome 58.0.3029 (Mac OS X 10.10.5) sparkfund.ui-test plus-one FAILED
	Fail (plus-one) (at http:438:14)
	Expected (= 5 (ui/plus-one 1))
	Actual: (not (= 5 2))

For CI would need to tediously find a sufficient -t parameter for the wait task.

3. Minor issue but just noting above - should fix the redundant file paths in the generated config.

The most straightforward solution could be for the boot process to indeed terminate, but to have written out the test files via target for the karma server to find. But then returning a CI friendly exit code is out of the question.

So a point release fixing (1) and blocking on karma server calls is likely the way forward.

Unfortunately I haven't had much time free to touch this library - you can see the neglected issues and PRs. Any help would be greatly appreciated! Otherwise, knowing now that there are actively affected users, I'll make a push in any case 🙂

[jffry]

Thanks for the detailed reply - on a weekend, no less! For now at least, it works just fine to run the tests in Phantom, but I'd love to help get this fixed so we can run our tests in the actual browsers that are representative of what our users have.

Do you have a patch already for fixing :update-fs? or would a PR for that be welcome? I'm willing to take a crack at the three problems you outlined, and being able to poke around at the generated Karma configs and files will be a useful first step. So I think I'll try to get that working and send in a PR for it if you're amenable.

Thanks for your help and please don't feel pressured to make time to work urgently on these things!

[crisptrutski]

Have published 0.3.1 with update-fs? fixed up

Suggest fiddling with wait to get single runs working, and watch based workflows should be OK

Definitely amenable to PRs 😍

[crisptrutski]

Closing this ticket as it's gone cold - please reconnect if you're still having issues!

[binarykitchen]

Afraid this is still happening with doo v0.1.8 using chrome headless

[alexandergunnarson]

Using 0.3.4 and appending wait to the tasks (e.g. (comp (test-cljs) (wait))) works for me on both Mac OS X 10.14.1 and the latest Docker image of Alpine Linux. However @crisptrutski's observation that For CI would need to tediously find a sufficient -t parameter for the wait task unfortunately still holds so not a workaround I'd like to deal with permanently. Still, appreciate your hard work!

[alexandergunnarson]

(Thanks for reopening this @crisptrutski!)

[alexandergunnarson]

Also even adding wait is only sometimes a workaround. With the exact same Docker image (openjdk:8-alpine with headless Chrome and Boot installed — yes Alpine Linux is a bit finicky when it comes to setting up Chrome, and I had to pass the --no-sandbox flag in the custom test runner config, but I don't think that's the issue) and no file/environment changes between runs, it's kind of a random toss-up as to whether running cljs-test with :chrome-headless will actually run the tests. When it does fail I get the following stacktrace:

;; ======================================================================
;; Testing with Chrome-headless:

[doo] Started karma server
[doo] Started karma run
                                java.lang.Thread.run              Thread.java:  748
  java.util.concurrent.ThreadPoolExecutor$Worker.run  ThreadPoolExecutor.java:  624
   java.util.concurrent.ThreadPoolExecutor.runWorker  ThreadPoolExecutor.java: 1149
                 java.util.concurrent.FutureTask.run          FutureTask.java:  266
                                                 ...
                 clojure.core/binding-conveyor-fn/fn                 core.clj: 2030
                                   boot.core/boot/fn                 core.clj: 1031
                                 boot.core/run-tasks                 core.clj: 1021
                      boot.task.built-in/fn/fn/fn/fn             built_in.clj:  567
                      boot.task.built-in/fn/fn/fn/fn             built_in.clj:  394
                      boot.task.built-in/fn/fn/fn/fn             built_in.clj:  627
boot.user$eval1816$fn__1817$fn__1824$fn__1825.invoke                         :  769
  crisptrutski.boot-cljs-test/-prep-cljs-tests/fn/fn       boot_cljs_test.clj:  154
                  adzerk.boot-cljs/eval4605/fn/fn/fn            boot_cljs.clj:  169
                  adzerk.boot-cljs/eval4662/fn/fn/fn            boot_cljs.clj:  250
   crisptrutski.boot-cljs-test/-run-cljs-tests/fn/fn       boot_cljs_test.clj:  227
              crisptrutski.boot-cljs-test/run-tests!       boot_cljs_test.clj:  216
           crisptrutski.boot-cljs-test/run-tests!/fn       boot_cljs_test.clj:  187
clojure.lang.ExceptionInfo: Test failures signalled by Karma
    boot.util/omit-stacktrace?: true
clojure.lang.ExceptionInfo: Test failures signalled by Karma
    line: 876
Close Karma run
Shutdown Karma Server

And when it succeeds:

;; ======================================================================
;; Testing with Chrome-headless:

[doo] Started karma server
[doo] Started karma run
LOG: 'Testing <namespace>'
LOG: 'Testing <namespace>'
HeadlessChrome 71.0.3578 (Linux 0.0.0): Executed 2 of 2 SUCCESS (0.036 secs / 0.002 secs)
TOTAL: 2 SUCCESS
Shutdown Karma Server
Close Karma run

I added (wait :time (long (* 40 1000))) after this (was originally like 1-10 seconds but just wanted to give enough breathing room). When it does fail it exits before the wait time has finished.

[20goto10]

(Years later...) You should probably not use any of this anymore, but if you can't switch to newer tools for some reason, it's worth noting that results differ depending on the compiler optimization parameter. I found that nothing ran with chrome-headless when the compiler :optimizations flag is set to :none (which is the default) but if you pass it as :advanced then the tests DO run. Based on the OP's example, you can get output like this:

(deftask test-in-chrome
  "Prerequisites: npm install --global karma karma-chrome-launcher karma-cljs-test"
  []
    (test-cljs :js-env :chrome-headless
               :optimizations :advanced))

I replaced :none with the :whitespace optimization instead, which supposedly just compresses spaces, newlines, and removes comments, and is thus the closest you can get to :none, and that worked as well. I have no idea why :none doesn't work; I was able to copy out the generated test suite files for both :none and :advanced, update the karma conf to my new paths, and re-run them manually using "karma run" without problems. They were only skipped when run directly through boot-cljs-test. (All I get when that example runs is a bunch of parse errors, but when it doesn't run I don't get any output from Karma at all.)

Also note I did not need to add a wait when running it this way on the above example, but I did need it on my own project (a much bigger, slower suite of tests), so I recommend including it anyway... Your mileage may vary.

After switching to cljs-test-runner I had none of these problems.

Hope this helps somebody.