-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathBUGS
167 lines (126 loc) · 5.92 KB
/
BUGS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
Open
----
25 autorun doesn't allow rebinding special variables, notably
*standard-output*
26 UX failure: After renaming/deleting a test, it's still run and
there's no obvious way to remove it.
27 asdf:test-system after clear-tests doesn't necessarily re-add
tests; just returns ok after not running anything. Maybe make
clear-tests touch the asd file to force recompilation?
29 Pie in the sky idea: use emacs compilation machinery to highlight
failing tests in a buffer. Probably not worth the effort?
-- See slime-create-compilation-log and things manipulating the result
-- of make-overlay for example of how slime works.
35 Considering commenting all non-result printing.
38 Updating the test file does not trigger autorun.
39 nil *handle-errors* should probably just decline to handle instead of
explicitly triggering the debugger.
Closed
------
1. When LS tests were autorunning and autorun and we used
autorun in SDB, the LS test suit seemed to trigger two or
three dozen times in a row.
-- Went away with queue driven redesign
2. Is handler aborting the rest of the clauses to a test and on error
instead of running them?
-- can't reproduce
3. Argument values aren't shown in any way.
-- added printing of function arguments for failed tests
4. No option to drop into the debugger. Could expose through a global
special.
-- added *handle-errors*
5. Need a way to stop autorun
-- Fixed; call AUTORUN again to toggle it.
6. Somehow we managed to deadlock sbcl when stress testing
autorun (this may have been related with running
autorun multiple times for the same system and manually
killing threads)
-- Can't reproduce. Probably eliminated when AUTORUN was
rewritten.
9. Compilation errors triggered by AUTORUN open the debugger,
this is probably undesirable. The undesirability is exacerbated by
the fact that the test running seems to be sometimes triggered at
random.
-- Fixed by adding a handler-bind in yarty:test-system and adding
*handle-autorun-compilation-errors*
10 When an error is thrown by evaluation of a function's argument
during EACH, the function's arguments are erroneously reported as
returning nil.
-- Fixed
14 TEST-ON-CHANGE is a clunky name. Maybe call it AUTORUN or AUTOTEST
or simply AUTO?
-- AUTORUN, as a parallel to RUN-TESTS.
8. Starting AUTORUN failing with FAMOpen2 error code 3, this
seems to be related to either little free memory or having already
run fam before (possibly with an ungraceful exit/hanging)
-- removed cl-fam dependency
11 Under AUTORUN tests seem to sometimes trigger at
random. Initial hypothesis: emacs autosave files are triggering
fam.
-- Using asdf to choose relevant files to watch.
12 Ran into a situation where autorun kept triggering
constantly while trying to write a large cdb file.
-- Using asdf to choose relevant files to watch.
15 It should be possible to YARTY:RUN-TESTS or (asdf:test-system 'foo)
/without/ loading autorun.lisp -- especially /without/ bring in the
cl-fam dependency. This is important so that end users of libraries
which use YARTY can run the test suite with hopefully minimum
overhead. (The C FAM/Gamin dependency is particularly egregious,
and so is requesting that the user manually install my fork of
cl-fam, but it might be nice to make BT and LPARALLEL loading
contingent on AUTORUN use as well.)
-- removed cl-fam dependency
16 AUTORUN should only trigger when a _relevant_ file is
changed. Related to 11 and 12.
-- Using asdf to choose relevant files to watch.
18 Computing files-to-watch every .5 seconds is taking significant
compute power (~8% processor of a small digital ocean server).
-- Only recompute them when a change is detected; also watch the .asd.
-- Same fix again. This time comment the code.
20 *handle-autorun-compilation-errors* also causes test errors to be
handled.
-- *handle-autorun-compilation-errors* is no longer an option, specify
that autorun also binds *handle-errors* to true.
19 Error handling should instead be serious condition handling.
-- change from *handle-errors* to *handle-serious-conditions*
-- REVERTED: errors are more appropriate. see cltl2 29.3.12
21 In autorun, results of test-system are being added to a channel but
never recieved; growing the channel indefinitely
-- Flush the channel
13 During *12 calling autorun from entering the debugger with
an interrupt failed to kill the autorun threads. After the
file was finished writing we were able to kill the threads.
-- Can't reproduce. Hopefully fixed by the refactoring that has
happend since it was reported.
7. Sending autorun output to the repl is unsatisfactory, and probably
requires setup not described in the readme.
-- For now, we consider the output as good enough and mention in the
README that slime may need to be configured to redirect output.
22 Tests should be restartable when conditions aren't handled.
-- Add continue restart.
23 Stand alone each shouldn't print "In nil"
-- Fixed.
24 RUN-TESTS should take an arbitrary number of packages.
-- Fixed.
30 README should include an example .asd.
-- Included.
17 Files for a test system aren't watched to trigger AUTORUN.
-- Fixed, when using asdf setup described in README
31 Bad asdf files chokes autorun
-- Fixed, add error handling during computation of files to watch
32 EACH doesn't recognize flet / labels functions when they start a form
-- Fixed
33 Feature request: start a child image (of any impl.) to return test
results and exit.
-- Added for sbcl and ccl as FRESH-TEST
34 Our return value doesn't actually match CL-TEST-GRID
-- Changed to mach CL-TEST-GRID: failures are now their own list (and strings)
36 Considering exporting a stream to which test output is printed.
-- Add *OUTPUT*
36 Enhancement: Export test functions
-- Added
37 Allow docstrings in deftest
-- Added, as well as declarations
28 autorun starts threads and leaves them running even when autorun is
not passed a valid system name.
-- Fixed.