cabal test --test-show-details=streaming should support colour output

[mietek]

As discussed in #1601 (comment)

Paging @ttuegel, @dag.

[ttuegel]

I investigated this when I originally implemented test log streaming. Here is what I discovered:

• On Unix-like platforms, you need to use a PTY instead of a pipe to communicate with the child process. Libraries for doing color terminal output detect if they're connected to a pipe and disable color.
• Windows has no notion of a PTY or anything like it. I do not even know if this can be done on Windows.
• Assuming you have the child process connected to a PTY, Cabal would need to emulate the terminfo of the terminal it's connected to in case the child process uses features besides just ANSI color.
• Finally, to write the log file, you have to remove the control codes yourself. But, that's not enough: if the test output used control codes that change the contents of the screen, the log file will be garbled.

Of course, this is all dictated by the logging requirement. If we abandon logging test suite results for --show-details=streaming, we can just passthrough stdout and stderr. I opted not to drop the test logs because that would be a significant departure from the way Cabal has worked until now, but we can use this issue to discuss that option.

[Rufflewind]

There is an easier workaround via the script utility, which does all the tricky TTY emulation and stuff. (It doesn't fix all the problems though; color codes will still get recorded to file, but you could argue that's a "feature" too: you can view the colored output using less -R.)

To make it work, there needs to be a way to invoke the test executable via a wrapper of some kind. This would have applications beyond colored output and should be relatively easy to implement.

Therefore, the easiest way to get colored output is if Cabal supported a way to indirectly invoke the test executable via a flag or .cabal setting (or both?), e.g.

cabal test --test-runner=./faketty.sh --test-option=opt1

which would call the test executable like this:

./faketty.sh dist/build/test/mytest opt1

You can then put this in faketty.sh:

#!/bin/sh
p="s/'/'\\\\''/g"
cmd=
for arg; do
    cmd="$cmd '"`printf "%s" "$arg" | sed "$p"; printf "'"`
done
exec script -e -q -c "$cmd" /dev/null 2>/dev/null

[ttuegel]

To make it work, there needs to be a way to invoke the test executable via a wrapper of some kind. This would have applications beyond colored output and should be relatively easy to implement.

Not yet, but your contribution would be welcome!

For this particular case, I think it would be best if this worked out-of-the-box. If we added script to the list of configurable programs, we could build this function into the test runner (while having the ability to fallback to something else if script isn't available).

[chshersh]

Does this issue affect cabal new-test as well? I don't see colored test output with cabal new-test (Cabal version: 2.1.0.0, testing framework: tasty, os: Ubuntu 16.04). Though with stack build tool I see colored test output. In practice it's really useful to see colors in output. When tests are failing it's much easier to find red test results rather than searching for FAIL keyword.

[erikd]

Does this issue affect cabal new-test as well?

Yes it does!

I have dug into the stack source code and it too uses a pipe (via createSource from typed-process) but I can't see that its doing anything special to get the colour coming through

[erikd]

Colored test output works with cabal test --test-show-details=direct.

See also: #6129

[yaitskov]

Why command line flag is so long? I would expect something like -d

[gbaz]

Given that we have direct does this ticket really still make sense?

[Mikolaj]

@gbaz: no, the ticket seems to be outdated. Let's close.

@yaitskov: you can set test-show-details: direct in your cabal.project file and then the length no longer hurts. But feel free to review our commandline options and propose a sensible scheme of one-letter counterparts for all (most?) in a new ticket, especially if you'd like to implement that. That may be ripe for some overhaul.