Provide language tests in liquid repo itself

[dylanahsmith]

Problem

Generating tests from liquid is a flawed approach, which implicitly couples to those tests following a set of rules to allow the tests to be generated properly. This results in issues like #30 and #31 and plenty of updates to the upstream liquid repo to get the right test coverage.

This test generation approach is also quite inconvenient, since it means that adding a test requires changes to upstream liquid to add the test as well as a PR to liquid-spec to regenerate the tests. This also means that these two can easily get out of sync, without CI giving feedback of problems from changes to liquid that could break test generation.

The test generation approach also relies on cloning the liquid repo into tmp/liquid and patching it, leading to issues like #33 and requires hacks to test against a liquid branch (e.g. to use the test immediately in liquid-wasm).

Proposal

Have the end goal be to have an in-repo gem in the upstream liquid repo with yaml liquid language tests, along with the code to run those tests as part of its test suite. This would only includes tests that don't rely on language extensions (e.g. custom drops, tags or filters) which aren't really testing the core language itself, which could continue to be tested in test/integration folder.

As part of that transition, the liquid tests can continue to be refactored to improve test generation, which will also simplify the final conversion of those ruby tests to YAML tests. For instance, that can include

• Converting more language tests to use tests helpers decoupled from the liquid library API, such as assert_template_result or assert_match_syntax_error
• Separating language tests from library tests, such as by not using those decoupled test helpers assert_template_result or assert_match_syntax_error in those library tests
• Splitting assertions into separate test methods so they can each have a unique and descriptive name, without needing a digest hash for uniqueness

so that we can get to the point where the generated YAML tests could then be committed into the liquid repo and the ruby tests/assertions they were generated from removed.

[dylanahsmith]

Separating language tests from library tests, such as by not using those decoupled test helpers assert_template_result or assert_match_syntax_error in those library tests

Note that we should probably try to rewrite the language extension tests into Shopify liquid language extension tests, so that we can have test coverage for these features in a way that will be relevant despite liquid implementation changes. These tests could live within this repo.

It should also be noted that liquid-c should similarly be modified to make sure it tests against any integration tests that get moved out of liquid's test/integration folder.

[dylanahsmith]

I've opened Shopify/liquid#1621 to have a place to reference this work from upstream liquid PRs.