Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the golden tester even further #160

Open
2 of 13 tasks
fmease opened this issue Aug 11, 2022 · 0 comments
Open
2 of 13 tasks

Improve the golden tester even further #160

fmease opened this issue Aug 11, 2022 · 0 comments
Labels
A-meta-test-infrastructure Area: Meta: Test infrastructure

Comments

@fmease
Copy link
Owner

fmease commented Aug 11, 2022

Split from #140.

  • comprehensibly document UI testing in the currently empty README.adoc
  • introduce a format option -f/--format which accepts pretty or terse (taken from rust's libtest) where the terse format prints single tests single characters without space and line breaks (. for passing tests, i for ignored ones, g for gilded ones, I for invalid ones, a for auxiliary ones (not shown by default)) and maybe even make the terse the default
  • syntactic or preferably semantic highlighting for test configurations by the VS Code extension
  • improve the looks of the generated diffs by diff'ing diff'ed lines by character (only those lines where it makes sense, I think that's single + lines preceded by single - lines) and highlight them (changing their background color)
  • check how much the current architecture of auxiliary files impacts the (parallel) performance of the test runner. I suspect since we are skipping so many files, every thread wastes a lot of time and performance is not as good. basically the division of work is suboptimal. also for packages, we heavily rely on auxiliary files and the more package tests we add the more the performance will probably degrade. could the auxiliary/ folder design from rustc's test suites help us out?
  • revision / matrix system
  • support putting the test configuration into a separate file named <testname>.testconfig which allows us to test files where we don't want to have any comments e.g. to test everythings ok with truly empty files or lexing and spans are fine without comments when lexing (…), you get what I mean
  • smh. (somewhere) record the modification date of passing tests and skip them (printing ignored or ideally smth. more descriptive) when they are tested and if they were indirectly referenced (e.g. through a folder)
  • use proper lexers to obtain the comments (to get rid of false-postive test config params) (via Lexer-based UI tester #162)
  • ensure that the unit tests of the testing framework are executed in CI (haven't checked yet)
  • check if the framework actually works on Windows (we currently use the POSIX? timeout command for which I am not sure if it is available)
  • add a flag to test all git-unstaged or all git-staged test files
  • skip / ignore files that have already been tested & passed: record modification time / hash of test files & the compiler; if any of them differ from the current run, don't skip; this might not be very useful right now; add a flag to force-run all tests
@fmease fmease added the A-meta-test-infrastructure Area: Meta: Test infrastructure label Aug 11, 2022
@fmease fmease changed the title Improve to the golden tester even further Improve the golden tester even further Aug 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-meta-test-infrastructure Area: Meta: Test infrastructure
Projects
None yet
Development

No branches or pull requests

1 participant