How to run tests properly

[yuhan0]

I'm not sure if something has been changed recently with the testing setup, but it's a lot less friendly than I recall (might be a Eldev / Cask thing?)
Buttercup test failures are being output as elisp error traces instead of printed in a readable format

$ eldev test
Buttercup test failed
Run with ‘--debug’ (‘-d’) option to see error backtrace

Which in turn gives this:

$ eldev --debug test
Debugger entered--Lisp error: (buttercup-failed . "Expected `(cider-jar-find-or-fetch \"cider\" \"cider-...")
  signal(buttercup-failed "Expected `(cider-jar-find-or-fetch \"cider\" \"cider-...")
  buttercup-fail("%s" "Expected `(cider-jar-find-or-fetch \"cider\" \"cider-...")
  buttercup-expect((closure (t) nil '(cider-jar-find-or-fetch "cider" "cider-nrepl" "0.20.0") (buttercup--mark-stackframe) (cider-jar-find-or-fetch "cider" "cider-nrepl" "0.20.0")) :to-match (closure (t) nil '(rx "cider-cache/repo.clojars.org/cider/cider-nrepl/0.2...") (buttercup--mark-stackframe) "cider-cache/repo\\.clojars\\.org/cider/cider-nrepl/0..."))
  (closure (t) nil (buttercup-expect #'(lambda nil '(cider-jar-find-or-fetch "cider" "cider-nrepl" "0.20.0") (buttercup--mark-stackframe) (cider-jar-find-or-fetch "cider" "cider-nrepl" "0.20.0")) :to-match #'(lambda nil '(rx "cider-cache/repo.clojars.org/cider/cider-nrepl/0.2...") (buttercup--mark-stackframe) "cider-cache/repo\\.clojars\\.org/cider/cider-nrepl/0...")))()
  buttercup-describe("cider-jar-ensure-cider-nrepl-jar" (closure (t) nil (buttercup-expect #'(lambda nil '(cider-jar-find-or-fetch "cider" "cider-nrepl" "0.20.0") (buttercup--mark-stackframe) (cider-jar-find-or-fetch "cider" "cider-nrepl" "0.20.0")) :to-match #'(lambda nil '(rx "cider-cache/repo.clojars.org/cider/cider-nrepl/0.2...") (buttercup--mark-stackframe) "cider-cache/repo\\.clojars\\.org/cider/cider-nrepl/0..."))))
  load-with-code-conversion("/Users/yuhan/emacs-dev/cider/test/cider-jar-test.e..." "/Users/yuhan/emacs-dev/cider/test/cider-jar-test.e..." nil t)
  load("/Users/yuhan/emacs-dev/cider/test/cider-jar-test" nil t nil t)
  #f(compiled-function () #<bytecode 0x9353206b1df85b1>)()
  eldev--test-autoinstalling-framework(t #f(compiled-function () #<bytecode 0x9353206b1df85b1>))
  eldev-test()
  apply(eldev-test nil)
  eldev-cli(("--debug" "test"))
  (kill-emacs (eldev-cli (append (cdr (member "--" command-line-args)) nil)))
  command-line-1(("--execute" "(let ((eldev--emacs-version (format \"%s.%s\" emacs-..." "--execute" "(kill-emacs (eldev-cli (append (cdr (member \"--\" c..." "--" "--debug" "test"))
  command-line()
  normal-top-level()

Am I missing something in this dev workflow? It's nearly impossible to figure out what the actual test failure is, buried in all these escaped and truncated strings. Or are we supposed to run tests within Emacs?

[bbatsov]

For me eldev test works fine (besides the current failures that I still have to fix this week). Maybe @doublep can suggest some way to debug your problem?

[doublep]

The problem appears to be in this piece of code:

(let ((expected (string-join '("-Sdeps '{:deps {nrepl/nrepl {:mvn/version \"0.9.0\"} "
                               "cider/cider-nrepl {:mvn/version \"0.28.3\"}} "
                               ":aliases {:cider/nrepl {:main-opts [\"-m\" \"nrepl.cmdline\" \"--middleware\""
                               " \"[cider.nrepl/cider-middleware]\"]}}}' -M:cider/nrepl")
                             "")))
  (describe "should remove duplicates, yielding the same result"
            (expect (cider-clojure-cli-jack-in-dependencies nil nil '(("nrepl/nrepl" "0.9.0")
                                                                      ("nrepl/nrepl" "0.9.0")))
                    :to-equal expected)))

which invokes expect outside of it, so, as far as I understand, can fail at the point tests are being defined, but not yet "officially" executed. If this is removed I still get some failures, but in the nice expected way.

[vemv]

Thanks! Sorry about that, tests were red already so this nuance didn't call our attention.

[yuhan0]

Thanks - this should probably have been on the issue tracker, I wasn't aware that the tests were currently being fixed and thought it was a problem on my end.

[bbatsov]

My bad. I meant to address those problems faster.

[vemv]

Fixed #3165