[WIP] Test: Uses Criterion framework to test LibSass

[am11]

[Work in progress]

Issue-URL: sass/sassc#90.

Summary of current progress:

• Users Criterion as a testing framework: https://github.com/Snaipe/Criterion.
• Revokes SassC and Ruby dependencies for running tests.
• Use Cmake to DRY make logic (only targets tests: test/CMakeLists.txt).
• Clone dependencies within CMakeLists.
• Uses Theory and TheoryDataPoints to test.
• Uses array to capture all the input.scss | input.sass.
• Controls TODOs and unexpected mode with environment variables.
• Use SLRE for regex replacements: https://github.com/cesanta/slre.

Besides all the non-TODO tests are running and passing (with both debug and release configurations).

Rationale

The idea is to first make the test runner (close to) the drop-in replacement of Ruby test runner, then make enhancement and incorporate ability to test those features of LibSass which ruby-runner wasn't able to test.

Scope of this PR (checklist):

Existing infrastructure features:

• Ability to collect all specs from sass/spec repo.
• Ability to filer TODO specs.
  • Implemented using environment variable: LIBSASS_SKIP_TODO.
• Ability to alter unexpected passes for TODO tests.
  • Implemented using environment variable: LIBSASS_UNEXPECTED_PASS.
• Ability to select sass-spec branch.
  • Set LIBSASS_TEST_SPEC_BRANCH to the name of the desired branch (default is master).
• Ability to compile all spec files and compare with the expected output.
• Ability to test all styles of output: expanded, nested, compact, compressed.
  • Currently only nested output is being compiled and tested.
• Buildable and run-able on Windows.
  • Compiles with VS2015 and MinGW.
• Account for each spec individually.
  • Blocked by Feature: User-defined dataset driven tests Snaipe/Criterion#43.
• Configure and report coverage.
• Configure CI to run tests.

Non-existing in ruby-runner, but most wanted feature:

• ability to test error messages.

Usage:

Currently, I have only tested on Windows 10 with VS2015. I intend to test on Unix before merging it.

# in powershell..
cd somewhere
git clone https://github.com/sass/libsass

# build libsass with VS2015
&"${env:ProgramFiles(x86)}\MSBuild\14.0\Bin\MSBuild" libsass\win\libsass.sln `
/p:Configuration=Release

mkdir blah
cd blah
cmake ../libsass/test
cmake --build . --config Release

# following step will be automated:
# * either by (somehow) setting the compile target of subdirectory project to run.exe's.
# * OR with execute_process( COMMAND cp ..) in CMakeLists
cp Criterion/Release/criterion.dll Release/

# disable TODO specs
$env:LIBSASS_SKIP_TODO=1

# need to unify the path of output executable (in test/bin for instance)
Release/run --ascii
# in *nix Shells, --ascii switch can be skipped

Questions:

• Using too many dependencies: is it good / bad / ok?
  • IMO it is an ok working solution at least for the starters and aligns with separation of concerns and DRY principles.
  • With LIBSASS_TEST_SKIP_DOWNLOAD_DEPS set, re-downloading all dependencies with cmake libsass/test can be skipped.
• SLRE based Regex has negative performance:
  • Should we update the spec/**/*.css files to exactly match libsass output?
    • I have a feeling this is not going to happen. :)
  • OR Should we invest time in optimizing sanitize function?
    • I discourage this option very much in favor of the next one.
  • OR Should we tap into LibSass's AST and find out the absolute equality of given CSS outputs? :)

---

Special thanks to @Snaipe for all the advises + assistance and for building the great testing framework for C: Criterion. 🏆

[saper]

I don't think that replacing ruby with C for the sake of replacing ruby is a most important goal in itself; however with a good C/C++ testing framework one could actually start tearing apart libsass code internally and test various pieces in isolation (probably very difficult at this). Any ideas how to approach this? To have more white-box testing of the C++ code? Is Criterion suitable for this?

[am11]

To have more white-box testing of the C++ code? Is Criterion suitable for this?

Yes it is possible with Criterion.

I am interested in getting the checklist done and get answers to the questions I raised above; which lie within the scope of this PR.

For the other aspects, like why ruby-runner is not a viable option for libsass moving forward, lets keep it to the related threads: sass/sass-spec#136 & sass/sassc#90.