SCSS testing

[MartijnCuppens]

Now that our flexibility of our scss is going to extended with PRs like #28445, we should have a look at automated tests for scss. With these tests I mean: check the output if scss configuration is changed (like adding utilities, etc..)

I used MochaJS for doing these tests with RFS, that worked pretty well, but maybe we can use the same "thing" for js and (s)css tests.

[mdo]

I'd love to see some examples of how this helps us and others. Never have found a good solution that didn't seem superfluous to writing CSS, but I haven't actively sought that out :).

[MartijnCuppens]

Here is how it works in RFS:
https://github.com/twbs/rfs/tree/master/test

For example for sass, there are 10 different configurations that are tested with the same main file(https://github.com/twbs/rfs/tree/master/test/sass).

In the expected folder the expected output is shown (https://github.com/twbs/rfs/tree/master/test/expected).

These tests are triggered by git hooks for each commit and also in travis to make sure we don't introduce breaking changes.

For Bootstrap we can something alike to test if changes to the utilities api won't break anything. Other tests could be the output of functions or mixins with specific configurations.

[joneff]

If I may chime in, at the end of the day CSS controls how things look. In terms, facing a similar problem, we decided to test the visual output of static pages. A screenshots is generated build time (on travis) and if it doesn't match up, we commit it and then trigger the build again.

There were some caveats, like specifying the correct tolerance; dealing with browser updates (we are using firefox for the screenshots), spinning animations prove impossible (need to be disabled). I mean it's not the best solution but it works for us and gives us nice visual diffs as well.

[romaricpascal]

Hello,

As part of implementing mixins and functions to help manipulate the utilities map, I wanted to look into having some tests (as there some end up being a bit more than just a chain of map-get).

@mdo mentionned two efforts that had already started on the subject:

1. His work on Basic Sass tests for $theme-colors customization #34967: from what I gathered, it uses ad-hoc @if and @warn(or possibly @error further down that route) to output whether exxpectations are met
2. The gs/test-sass-unit-tests branch by @GeoSot and @XhmikosR : this one uses [sass-true] to write the tests with custom JS scripts to gather the tests to run and interpret their output, if I understood correctly.

I tried a 3rd way to get start getting a sense of safety for the functions I was building using:
a. [sass-true], still, to write the tests, as it already provides mixins for structuring the tests and making assertions
b. Mocha for collecting the test files, running them and their results

The logic behind this approach is to let each tool do what they're good at so there's no need to maintain either a library for structuring the test code (which is more or less where 1. would lead, I think), nor scripts for collecting, running tests and reporting their results (which is what 2. does).

Setup is in this commit on my fork and here is a commit with some test (though that one is just plain sass-true test, nothing out of this world). It works by making Mocha understand that the project uses .test.scss files (instead of the regular .js extension) and turn each them into a JS module that runs sass-true.

It's not all rainbows and unicorns, though, and there's a few downsides:

• its reliance on require.extensions for letting NodeJS require .scss files and transform them into a JS module on the fly. The feature is [marked as deprecated by NodeJS][require-extensions], however, it's what tools like babel-register use to hook onto NodeJS. As Node shifts to ESModule, they're looking at a [loader API for doing that kind of transpilation][node-loader-api], but it's experimental for now. Cleanest alternative may be to use [Jest and its transform option][jest-transform]. If you're aware of another testing tool with that kind of opening, that could work too.
• watching the files require using nodemon rather than Mocha's --watch, likely because of some NodeJS internal caching of the module created programmaticaly. I could look for a way to cache bust it, but a watch-css-test script is much simpler.
• its reliance on Mocha in the current state rather than Jasmine (which is already used by Karma, maybe?, so that would be some extra dev dependency), but I think the same approach may be achievable with Jasmine (or other test libraries). I did see some comments about Jasmine issue on windows 10 in the branch for 2. though. So maybe not as straightforward. On that topic, I'm working on Linux, so would need to check that it works on other platforms too.

Hopefully I'm not stepping on anyone's toes, but thought I'd describe the approach I'd take (especially as I went through it to give myself some help) 😊 Obviously happy to discuss in more details.

[sass-true]: https://www.npmjs.com/package/sass-true)
[require-extensions]: https://nodejs.org/dist/latest-v16.x/docs/api/modules.html#requireextensions
[node-loader-api]: https://nodejs.org/docs/latest/api/esm.html#esm_experimental_loaders
[jest-transform]: https://jestjs.io/docs/configuration#transform-objectstring-pathtotransformer--pathtotransformer-object

[mdo]

As mentioned elsewhere, love this @romaricpascal! I'd love @XhmikosR's feedback on the additional testing dependency. I'm fine with it, but if there's a way to keep dependencies down, I'm always for that more :).

[GeoSot]

The gs/test-sass-unit-tests branch by @GeoSot and @XhmikosR : this one uses [sass-true] to write the tests with custom JS scripts to gather the tests to run and interpret their output, if I understood correctly.

To be honest I cannot recall the exact reason I used custom js. I can just remember (if that helps) that couldn't find a solution using jasmine, and the js script , is a copy of true-sass internals.
Plus the time the script was written dart-sass didn't provide much configuration through js, so I used exec to parse the files.

At least the written tests are using the sass-true functions 😋

So, if you need any help I'll try to give you a hand

[mdo]

We shipped this finally!

[XhmikosR]

@mdo can you remove any unused related branches too?