Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-127221: Add colour to unittest output #127223

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

hugovk
Copy link
Member

@hugovk hugovk commented Nov 24, 2024

Use the existing _colorize module already used for tracebacks and doctest so output can be controlled with the PYTHON_COLORS, NO_COLOR and FORCE_COLOR environment variables.

This PR does a couple of things to add colour:

  • Enable colour for tracebacks by passing in colorize to msgLines = list(tb_e.format(colorize=can_colorize()))
  • Add colour for pass/fail/errors, using pytest as a guide
  • Plus some unrelated tidy-up (use more specific test methods, new-style objects, underscore variables)

Tested with some small demo scripts:

No tests ran

run-unittests-none.py
import unittest

if __name__ == "__main__":
    unittest.main()
Before After pytest
image image image

Passed

run-unittests-pass.py
Mode Before After pytest
Regular image image image
Verbose image image image

Failing

run-unittests.py
import unittest


class TestThings(unittest.TestCase):
    def test_pass(self):
        self.assertTrue(True)

    def test_fail(self):
        self.assertTrue(False)

    def test_error(self):
        raise Exception("Manually raised exception")

    def test_pass2(self):
        self.assertEqual(1, 1)

    @unittest.skip("demonstrating skipping")
    def test_skip(self):
        self.fail("shouldn't happen")


class ExpectedFailureTestCase(unittest.TestCase):
    @unittest.expectedFailure
    def test_fail(self):
        self.assertEqual(1, 0, "broken")

    @unittest.expectedFailure
    def test_pass(self):
        self.assertEqual(1, 1, "not broken")


if __name__ == "__main__":
    unittest.main()
Mode Before After pytest
Regular image image image
Verbose image image image

Subtests

run-unittests-subtests.py
import unittest


class DoSubTest(unittest.TestCase):
    def test_subtest(self):
        """Test with subtest"""
        for i in range(0, 4):
            with self.subTest(i=i):
                if i == 2:
                    raise Exception("Manually raised exception")
                self.assertEqual(i % 2, 0)


if __name__ == "__main__":
    unittest.main()
Mode Before After pytest
Regular image image image
Verbose image image image

📚 Documentation preview 📚: https://cpython-previews--127223.org.readthedocs.build/

@hugovk hugovk added type-feature A feature request or enhancement stdlib Python modules in the Lib dir labels Nov 24, 2024
@hugovk hugovk changed the title Add colour to unittest output gh-127221: Add colour to unittest output Nov 24, 2024
@nineteendo
Copy link
Contributor

Looks good, I see you're taking some inspiration from pytest, but I had to zoom in all the way to be able to read the text. :)

@hugovk
Copy link
Member Author

hugovk commented Nov 24, 2024

You can click the images to open them full size.

@nineteendo
Copy link
Contributor

nineteendo commented Nov 24, 2024

Well, I wanted to compare the images so I had to zoom in to see the difference. Maybe we could only show the comparison between pytest? The uncoloured images don't add much value.

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great!! With the coloured tracebacks as well, though, I sort-of feel like there might be a bit too much colour now when tests fail. It feels slightly overwhelming!

What about making only the "ERROR"/"FAIL"/"UNEXPECTED SUCCESS" lines coloured red, and not the ======= line above or the ------- below?

@@ -106,6 +107,7 @@ def cleanup2(*args, **kwargs):
self.assertTrue(test.doCleanups())
self.assertEqual(cleanups, [(2, (), {}), (1, (1, 2, 3), dict(four='hello', five='goodbye'))])

@force_not_colorized
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have to apply this decorator to so many methods that I almost wonder if it's worth having a custom metaclass that automatically adds it to each method on the class... or we could not use the @force_not_colorized decorator, and instead duplicate the logic in setUp and tearDown methods on the TestCleanup class here.

Neither feels ideal; it might be that what you have now is in fact best!

@hugovk
Copy link
Member Author

hugovk commented Nov 24, 2024

This is great!! With the coloured tracebacks as well, though, I sort-of feel like there might be a bit too much colour now when tests fail. It feels slightly overwhelming!

What about making only the "ERROR"/"FAIL"/"UNEXPECTED SUCCESS" lines coloured red, and not the ======= line above or the ------- below?

Thanks! Here's without coloured separators:

image

And verbose:

image

What do you think?

Also these demo scripts are a bit contrived; hopefully most of the time, most of your tests pass and just one or two fail. But then again, sometimes a lot do fail!

@AlexWaygood
Copy link
Member

Nice, that looks great to me now!

@nineteendo
Copy link
Contributor

nineteendo commented Nov 24, 2024

How about swapping the rows and columns in the table? It makes the text better readable:

show table
Mode Regular Verbose
Before image image
After image image
pytest image image

@AlexWaygood
Copy link
Member

How about swapping the rows and columns in the table? It makes the text better readable:

To be clear @nineteendo, this is a comment on the table in Hugo's PR description rather than the code he's changing in this PR, correct?

@nineteendo
Copy link
Contributor

Correct, with 3 images on the same row I can't read the text without zooming in.

@AlexWaygood
Copy link
Member

I see.

@hugovk, do you think we should add a link to the using-on-controlling-color somewhere in the unittest docs? I see we didn't when we added color support to doctest, but maybe we should have done

@hugovk
Copy link
Member Author

hugovk commented Nov 24, 2024

@hugovk, do you think we should add a link to the using-on-controlling-color somewhere in the unittest docs? I see we didn't when we added color support to doctest, but maybe we should have done

Yeah, good idea. Something like 5a1de7c?

Let's also add it to doctest and traceback.

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Doc/library/unittest.rst Outdated Show resolved Hide resolved
Copy link
Member

@sobolevn sobolevn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I like the idea. But, I personally feel like it is still a bit too colorful. Right now almost nothing is white.

One more idea: can you please test this with the white theme as well?

@hugovk
Copy link
Member Author

hugovk commented Nov 24, 2024

Thank you! I like the idea. But, I personally feel like it is still a bit too colorful. Right now almost nothing is white.

Almost nothing is white because these demo scripts are intentionally crafted to generate lots of failures :) It's very similar to pytest.

One more idea: can you please test this with the white theme as well?

Sure:

unittest pytest
image image
image image
image image
image image
image image

Note I've not adjusted my local colours for this light theme. If I was using it normally, I may have adjusted it already for pytest.

Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting merge stdlib Python modules in the Lib dir type-feature A feature request or enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants