Plans for mypy.myunit

[o11c]

Since driver is now merged, mypy.myunit is a package instead of a module; however all the logic is still in __init__.py and has irritating limitations.

Note that in these plans I am not yet making myunit and the driver aware of each other, just laying out a framework.

• Document which parts of myunit are actually used from elsewhere.
• Remove the unnecessary use of Any.
• Split "collect tests" logic from "run tests" logic, add -l to only collect and print.
• Split "collect results" logic from "print results" logic.
• Allow fetching a single test by name (see below) instead of collecting everything and then filtering even when there are no wildcards.
• Use fnmatch instead of hand-written glob functions (likely via regex conversion, so we can add a command-line option to take a regex directly).
• Add tests for myunit itself.
• Move assertions so you can do from mypy.myunit.assertions import *
• Add decorators in mypy.myunit.annotations, see https://gist.github.com/o11c/ef8f0886d5967dfebc3d
• Also add an decorator for "this is an assert_foo function" to replace the current traceback-cleaning logic, which is wrong if something goes wrong in the assert function itself.
• Add a check that some assert_foo function is actually run for every testcase.
• Record results of a test run to file so that diffs can be displayed from previous runs.
• Move mutable globals to mypy.myunit.ick
• Move main function and related logic to mypy.myunit.main
• Allow free functions to be used as tests instead of requiring them to be wrapped in a class.
• Support opting out of the tempdir logic.
• Support fixtures by looking up (name.strip('_'), annotated_type) in current module and in mypy.myunit.fixtures (use this for tempdir maybe?)
• Change data-driven testcases from overriding cases to overriding init, likely in a base class that takes care of calling parse_tests.
• Support a "keep going" mode of assertion failure.

Anyway, the big change is global test naming.

---

New test names have more dots, and will look like mypy.test.testlex.LexerSuite.test_empty. It is not always as simple as getattr - modules need to be imported, classes need to be instantiated, data-driven cases need to be parsed, etc.

The first component name will always be a module, and most likely always be just 'mypy' depending on what is done about #724

I introduce two new classes, TestFactory and Test, to avoid refactoring the responsibilities of Suite and TestCase. There will be a bridge of course, or maybe the old classes will just be subclasses of the new ones. Possibly the old classes will go away entirely.

API will look something like:

TestOrTestFactory = Union['Test', 'TestFactory']

class Test:
    name = ''
    # Basically just TestCase without all the cruft,
    # but also stores the annotations.
    # It is now forbidden to create your own subclasses, use fixtures instead.

class TestFactory:
    name = ''
    @abstractmethod
    def list_immediate_children(self) -> Iterable[str]: ...
    @abstractmethod
    def get_immediate_child(self, name: str) -> TestOrTestFactory: ...

    @staticmethod
    def list_recursive_children(name: str) -> Iterable[str]: ...
    @staticmethod
    def get_recursive_child(name: str) -> TestOrTestFactory: ...

class ModuleTestFactory(TestFactory):
    def __init__(self, module: types.ModuleType) -> None: ...
    # implementation uses `getattr`, `__path__`, and `__import__`
    # to find both contained modules (but not imported ones from elsewhere!)
    # and test objects (functions, classes, or instances) from the current module.

class SuiteTestFactory(TestFactory):
    def __init__(self, suite: Suite) -> None: ...
    # implementation uses `suite.cases()`, now restricted to only return `TestCase`s.
    # (The only remaining suite with suites as children is the temporary `ListSuite`).

Bikeshed: what do we call the module in which Suite, TestCase, TestFactory, and Test reside? Same module or separate ones?

[o11c]

Sweet, just realized that assert_raises isn't used at all outside of my local stuff.

[JukkaL]

Most things look reasonable (though I didn't quite understand how the fixtures would work) but some of these probably aren't the highest-impact things we could be working on and probably not worth the extra complexity, such as the different test failure modes. Having just PASS, FAIL (maybe XFAIL) and SKIP would be enough, and the rest would be of marginal utility. I've worked with much larger test suites that did just well with only a few test modes.

My recommendation is to create individual issues for the ones you think are the most useful (you can leave this open as an umbrella task), and with a little more detail on each issue. It's better to discuss specifics outside this umbrella issue. Please send separate pull requests for each of the issues, prefarably not stacked on top of each other. That way you'll get the fastest review turnaround times (unless I'm traveling or something and can't get this done -- I'll try to post an update to your open PRs if that happens so that at least you know what's going on).

My personal favorite thing to fix related to running tests would be #910.

[emmatyping]

I believe this can be closed as it is out of date and we are moving away from myunit (see #1673)