Run test suites with cabal run

[dag]

I don't know if this could make sense for detailed, but for exitcode-stdio I find myself trying to run them with cabal run which fails with cabal: Couldn't find any executables..

You can of course specify which test-suite to run with cabal test but:

1. To pass in options to the executable you have to use --test-option[s] which is much clunkier than what you do with cabal run.
2. By default you don't get to see the output unless the test failed, or you manually view the log file.
3. Even with --show-details=always (again, clunky) this just prints the log file at the end, which means you don't get to see it until the whole test has completed, and you get no colors or progress bars if your test framework does that.

So I either have to do cabal build test && ./dist/build/test/test which is clunky, or cabal repl test and :main which means interpreted and can be really slow for things like quickcheck properties.

It's the small things. :-)

[23Skidoo]

Seems a nice idea to me. @ttuegel, what do you think?

[tibbe]

We will also add cabal test --show-details=streaming which will use pipes to stream the output to both stdout and to the test log file.

[dag]

Perhaps streaming then could be made default for cabal test (but not cabal install --enable-tests) and then support for -- starting test options could be added to cabal test a la cabal run? Then, this ticket would feel less important to me. That also raises the question if #1119 could work for cabal test too, though.

[ttuegel]

I think --show-details=streaming should definitely be the default; that's obviously what most people expect. I'm not sure if cabal run or cabal test -- [options] should run tests with options, but I'm leaning toward the latter. We will also need to distinguish between exitcode-stdio tests and detailed tests; the user should not be able to send command line options to the stub executable. (Is anybody even using detailed?)

[tibbe]

@ttuegel supporting cabal test -- [options] in the same way as we do for cabal run sounds like a good idea (we should do it for cabal bench too).

I'm kinda split about --show-details=streaming being the default. It works reasonable well now when cabal is very single package focused, but imagine if we support running many different packages' test suites in one run. This will happen once we support having many source packages inside one cabal "package environment". Lets say I run a Haskell company and I have this directory structure checked out from source control:

src
  |-- pkgA
  |-- pkgB
  |-- ...
  \-- pkgZ

I now change pkgA and want to check that none of my other package (which all depend on pkgA) broke:

$ cd src
$ cabal test *  # made up syntax to glob match many different packages

Streaming the output would now be too much. I only want to see failing cases. This is why I designed the initial command line UI like I did (based on what we have at work).

Aside: I don't think many people are using detailed. I don't even know which state it's in. Does it work? Did we end up blocked on test-framework or something else?

[ttuegel]

Let's keep the default behavior for now, and have streaming be optional. Later, we may want to do something smarter, like stream by default when there's only one package in the environment.

As for detailed: it works, but there are to implementations of the interface. I want to redesign the stub runner the get better logs, but that would entail no user-visible changes.

[dag]

I suppose it could default to streaming if there's only one package, or the package is implicit (by working directory)? Bit like -j1 (or did that change?). Or, if you explicitly name a test-suite (not sure how that even would work with multiple packages)? Or if it's made the default unconditionally, we can still do --show-details=failures when we want that. I suspect the single-package or single-test case is more common, especially since running multiple test suites in multiple packages will take a while so it's not something you invoke manually every little edit you make.

Yet another option would be to make this configurable. I tried adding show-details: always to ~/.cabal/config and ran a cabal test but it had no effect, not even a warning for unknown configuration option. :-)

In any case I'd be happy if I could use either test or run as I originally described, but I don't really care which of them it ends up being, or if I'll have to add a few configuration lines for it. :-)

[dag]

Actually, how is streaming supposed to work? Wouldn't that need do act like a pseudotty to get the test frameworks to not dumb down their output? If I configure the test framework not to do that then it instead breaks in e.g. a CI environment where you do want the plain-text output. I'd like a solution that "just works" here, which is what I originally was expecting from cabal run testname.

[tibbe]

Good question re. color output. I haven't thought about it a lot. I think
we do something to avoid writing terminal escapes to the log file today.
Perhaps we can just filter escapes before writing to the file but not
filter them when writing to stdout? I'm sure there's prior work here.

On Wed, Dec 4, 2013 at 10:35 PM, Dag Odenhall notifications@github.comwrote:

Actually, how is streaming supposed to work? Wouldn't that need do act
like a pseudotty to get the test frameworks to not dumb down their output?
If I configure the test framework not to do that then it instead breaks in
e.g. a CI environment where you do want the plain-text output. I'd like
a solution that "just works" here, which is what I originally was expecting
from cabal run testname.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1601#issuecomment-29848004
.

[dag]

Well the real issue is that the test frameworks typically check if stdout is a TTY and try to handle this automatically, so they'll strip coloring before it even gets to cabal.

[ttuegel]

@dag @tibbe It's actually much worse than that! If it detects a TTY, test-framework does its output normally, but if it detects that it's being redirected, it saves up all its output and prints it as it's exiting! QuickCheck does the same thing. So, if we don't properly emulate a terminal, streaming test output is going to be impossible.

[ttuegel]

OK, it looks like the unix package provides pty support on *nix systems, so I just need to find something that works on Windows, too.

[dag]

Regarding detailed; I'd love for that to replace the test frameworks, or be something they could build on. What is missing for that to happen? I'd like to see:

• Fancy terminal output
• Run tests in parallel (there seems to be stuff for this in Distribution.TestSuite already, is it used by the test runner?)
• Extensible options (again in Distribution.TestSuite, but I can't figure out how to pass in options via cabal test?)
• Running a subset of the provided tests (I'm guessing this is what tags is meant for but again, can't figure out how to actually select tags!)
• Have an interface for options and tags in cabal test that isn't clunky like --test-option[s] and that has bash completions (this is tricky since we currently have to compile the test module to get the options and tags, and if we change that then it's instead tricky to make the options extensible by external dependencies)

I imagine others would also want:

• XML output for CI/IDE

This is probably the wrong place to discuss this, but it doesn't quite feel like a ticket in its own right either. On the other hand it's somewhat relevant to this ticket because if the above is implemented I wouldn't feel much need for using cabal run with test suites anymore. :-)

[ttuegel]

detailed is just an interface; it wouldn't replace test-framework because the idea is to offload most or all of the work away from Cabal. The interface separates "test providers" (test-framework, QuickCheck, HUnit) from "test runners". This fits into the very-long-term goal of having a Cabal library which provides various architectural features and a cabal command-line tool which is merely the (simplest) client of said library.

The interface is ready, but we lack conforming test providers and test runners. On the test provider side, we need test-framework or tasty (or even just QuickCheck and HUnit) to export values of the types in Distribution.TestSuite.

On the test runner side, we have cabal test which is supposed to be the simplest possible test runner. As such, it may never support all the features you describe. (It has to ship with GHC, so external dependencies must be minimal.) The hope was/is that someone who wants a user interface with more advanced features would use the Cabal library to read the package description and run the tests as they see fit.

[tibbe]

That being said, I don't think writing a good test runner in cabal-install
would be a bad idea. In theory we want a plethora of test runners (and we
should support that). In practice people want only one.

On Thu, Dec 5, 2013 at 4:37 PM, Thomas Tuegel notifications@github.comwrote:

detailed is just an interface; it wouldn't replace test-framework because
the idea is to offload most or all of the work away from Cabal. The
interface separates "test providers" (test-framework, QuickCheck, HUnit)
from "test runners". This fits into the very-long-term goal of having a
Cabal library which provides various architectural features and a cabalcommand-line tool which is merely the (simplest) client of said library.

The interface is ready, but we lack conforming test providers and test
runners. On the test provider side, we need test-framework or tasty (or
even just QuickCheck and HUnit) to export values of the types in
Distribution.TestSuite.

On the test runner side, we have cabal test which is supposed to be the
simplest possible test runner. As such, it may never support all the
features you describe. (It has to ship with GHC, so external dependencies
must be minimal.) The hope was/is that someone who wants a user interface
with more advanced features would use the Cabal library to read the package
description and run the tests as they see fit.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1601#issuecomment-29907652
.

[dag]

I forgot to mention: should there be something like detailed for benchmarks?

[tibbe]

Eventually yes, for the same reason we have a detailed test interface,
namely to give cabal more visibility into what was run.

On Thu, Dec 5, 2013 at 5:51 PM, Dag Odenhall notifications@github.comwrote:

I forgot to mention: should there be something like detailed for
benchmarks?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1601#issuecomment-29914718
.

[amigalemming]

I would also like to run test-suites with cabal run (additionally to cabal test). cabal build and cabal repl already work with test-suite executables.

[phadej]

cabal v2-run runs also test executables