test suites/apps are sprinkled around the repository (and not run)

[hannesm]

apart from finding a good way where to have those, they should also be run on a regular basis and output compared (addresses #6)

the general idea should be that test suites and test apps (to run the suites) should be part of the specific library they test - and thus live in the same directory (well, a subdirectory)

any comments? carl? peter? bruce?

[cgay]

Sorry, I missed this issue when it was created. I agree, and it's on my list. Assigning to myself.

[cgay]

The original comment is a bit vague, but I think the "sprinkled around" part of this has been addressed.

I've also changed testworks such that test "apps" are no longer necessary. What I would like to do is to make the workspace-tool find, build, and run tests in any library named .*-test(s|-suite), and put output in a standard location. Not sure the best way to store the results permanently for later comparison; perhaps a git repo. Then running all tests should be just

$ workspace-tool new run-tests --all  # gets all packages, not just opendylan
$ workspace-tool test --all 

and forgetting to add tests to combined test suites / apps will be a thing of the past.

[cgay]

We're getting closer to this. Currently working on making the existing test suites pass by fixing the simple issues, then by marking long-standing, difficult issues as expected to fail. Then we'll be able to know "if it isn't green, my change broke it."

[cgay]

Status:

• I'm considering the "sprinkled around" problem to be solved enough that remaining problems can be addressed one-by-one.
• I've made a start at GitHub CI with GitHub Action in https://github.com/dylan-lang/install-opendylan. We can close this bug once a workflow is set up to run libraries-test-suite.

[cgay]

The CI depends on #1373, and fixing #1373 will cause the libraries-test-suite build to fail due to tests like this:

.../opendylan/sources/common-dylan/tests/common-extensions-tests.dylan:154.37-54:
  Serious warning - Reference to undefined binding {this-is-undefined in common-dylan-test-suite}.
                                          -----------------
        assert-signals(<error>, ignorable(this-is-undefined),
                                          -----------------

It looks like its possible to get rid of a few such warnings by tricking the compiler with dynamism. The ones that can't be avoided (like the one above) will need a different solution.

[cgay]

This is done.