Add unit test runner

[jacob-carlborg]

This will allow to use the compiler as a library to implement more unit test like tests. These tests will be able to inspect the internals of the compiler to perform new kinds of tests that are not possible today.

Unit tests live in the test/unit directory. They are written using the built-in unittest blocks. The unit test framework supports callbacks executed before and after each test. The unit test runner allows to limit the tests executed either by file(s) and/or by UDAs. Example:

module self_test;

import support : afterEach, beforeEach;

@beforeEach initializeFrontend()
{
    import dmd.frontend : initDMD;
    initDMD();
}

@afterEach deinitializeFrontend()
{
    import dmd.frontend : deinitializeDMD;
    deinitializeDMD();
}

@("self test")
unittest
{
    import std.algorithm : each;
    import dmd.frontend;

    findImportPaths.each!addImport;

    auto t = parseModule("test.d", q{
        int a = 3;
    });

    assert(!t.diagnostics.hasErrors);
    assert(!t.diagnostics.hasWarnings);
}

• To run all unit tests, run: ./run.d -u
• To run only the unit tests in a single file, run: ./run.d -u unit/self_test.d
• To run only the unit tests matching a UDA, run: ./run.d -u --filter "self test"

[dlang-bot]

Thanks for your pull request and interest in making D better, @jacob-carlborg! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please verify that your PR follows this checklist:

• My PR is fully covered with tests (you can see the annotated coverage diff directly on GitHub with CodeCov's browser extension
• My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
• I have provided a detailed rationale explaining my changes
• New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

---

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub fetch digger
dub run digger -- build "master + dmd#9333"

[WalterBright]

I don't understand how this is different from unittests.

[jacob-carlborg]

I don't understand how this is different from unittests.

This is unit tests. What we currently have in DMD are end-to-end tests. Also, if this is unclear, this is only internal, to be able to test DMD. It doesn't add any new language features.

[jacob-carlborg]

I was asked to make this a separate PR: #8528 (comment).

[jacob-carlborg]

Is blocking: #8528.

[CyberShadow]

This is unit tests. What we currently have in DMD are end-to-end tests.

It might be worth clarifying the difference between this and the existing unittest blocks, like

dmd/src/dmd/root/aav.d

Lines 256 to 266 in 51ecc6f

	unittest
	{
	AA* aa = null;
	Value v = dmd_aaGetRvalue(aa, null);
	assert(!v);
	Value* pv = dmd_aaGet(&aa, null);
	assert(pv);
	*pv = cast(void*)3;
	v = dmd_aaGetRvalue(aa, null);
	assert(v == cast(void*)3);
	}

. IIRC, there should already be some machinery in the build system to run them.

[wilzbach]

IIRC, there should already be some machinery in the build system to run them.

Yup, e.g.

cd src
make -f posix.mak unittest

Gotchas:

• we only test this on Posix, Windows fails
• this is part of the global test target

[wilzbach]

Still looks very nice 👍

[wilzbach]

CCing @ibuclaw and @kinke as I'm not sure whether they would be interested in running these tests as part of their testsuite (though they are mostly tests for DMD as a library).

[jacob-carlborg]

CCing ibuclaw and kinke as I'm not sure whether they would be interested in running these tests as part of their testsuite (though they are mostly tests for DMD as a library).

I really hope they will do that. This is indented as an alternative to the kinds of tests in the compilable and fail_compilation directories. It will also be possible to test new things that are not testable today. Hopefully we can eventually test the gluelayer and backend as well. For example testing that an optimization works. Running the tests using the test runner in this PR will hopefully be quicker since it will compile all tests into a single executable. Compared to the current approach, especially fail_compilation, which require compiling and running one executable per test.

[wilzbach]

Running the tests using the test runner in this PR will hopefully be quicker since it will compile all tests into a single executable.

BTW one thing that I wanted to ask is whether you know how well the GC version of DMD works?
In other words: if we have > 100 tests, will there be so much memory leaked that the process crashes?

[jacob-carlborg]

BTW one thing that I wanted to ask is whether you know how well the GC version of DMD works?
In other words: if we have > 100 tests, will there be so much memory leaked that the process crashes?

I don't know. I haven't experienced any issues with my tool DLP (that uses DMD as a library) [1], but I have 16 GB of memory.

[1] http://github.com/jacob-carlborg/dlp

[jacob-carlborg]

All tests are passing here.

[jacob-carlborg]

Thanks.

[rainers]

Looks like these changes only pass intermittently on Win32: https://auto-tester.puremagic.com/platform-history.ghtml?projectid=1&os=Win_32

[jacob-carlborg]

@rainers is the error: Error 45: Too Much DEBUG Data for Old CodeView format? Why does it pass sometimes?

[rainers]

is the error: Error 45: Too Much DEBUG Data for Old CodeView format? Why does it pass sometimes?

It seems it is. I guess it fails for one build server because they have different versions of optlink installed:

power: 8.0.16 works
win-farm1: 8.0.17 works
win-farm2: 8.0.13 fails

@braddr please update at least win-farm2

[jacob-carlborg]

Would removing the -g help?

[rainers]

Yes, I think it would.

[jacob-carlborg]

@rainers #9355

[WalterBright]

@jacob-carlborg can you please add documentation for tools/paths.d ?