SIGNAL a condition on test failures, for use with ASDF:TEST-SYSTEM #58
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PLEASE DO NOT MERGE YET
See https://gitlab.common-lisp.net/asdf/asdf/merge_requests/124 and ASDF mailing list discussion https://mailman.common-lisp.net/pipermail/asdf-devel/2019-September/thread.html for details.
Hello!
I recently wrote a script to run FiveAM tests for one of my
libraries on many different implementations on your own machine:
https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-tests.lisp
I do not want to copy-paste that script for all of my libraries. It
would be nice to contribute a generalized version of the script to
Roswell (on which the script is based), and just be able to say "ros
test my-system in all installed implementations" for any system.
The first step to doing that is to get ASDF:TEST-SYSTEM to report
errors using a common interface.
Because of the way ASDF is designed, ASDF:TEST-SYSTEM needs to use
conditions to signal test failures. I like this idea. Since you do
not have to handle conditions raised by SIGNAL, the functionality can
be added without impacting existing use or interfaces of FiveAM (the
use of WARNING instead of ERROR as the parent condition of
TEST-SPEC-FAILURE also ensures this, as many people tend to abuse
HANDLER-CASE to indiscriminately catch ERRORs).
With this patch, all projects that use FiveAM and define ASDF test-op
should be testable from a generic version of the Roswell script
without any changes.
The basic mechanism for test automation with FiveAM is then very
simple:
(handler-case (asdf:test-system "system")
(fiveam:test-spec-failure (condition)
(princ condition uiop:stderr)
(uiop:quit 1)))
Other test libraries can have supported added as well:
(handler-case (asdf:test-system "system")
((or fiveam:test-spec-failure prove:test-failure) (condition)
(princ condition uiop:stderr)
(uiop:quit 1)))
The goal is to introduce a parent condition in ASDF that FiveAM, etc., will then inherit from:
(handler-case (asdf:test-system "system")
(asdf:test-failure (condition)
(princ condition uiop:stderr)
(uiop:quit 1)))
Ultimately, any system using any test library should be testable this way, ensuring that test automation becomes trivial to build.
I am going to propose adding the parent condition to ASDF, but in the
meantime (until the ASDF patch makes it to the repository, another
release comes out, release becomes widely available…), I would like to get this code into FiveAM.
Vladimir