Implement Testsuite

[cmarqu]

It would be interesting to see how this implementation fares with https://github.com/nst/JSONTestSuite (see http://seriot.ch/parsing_json.html).

[Paebbels]

Nice new project. I'll see if I can run the tests with some of the tool chains.

[Paebbels]

Added JSONTestSuite testcases in 7e8b053.

[Paebbels]

@eine as you added vunit support, do you think it's possible to test these cases with vunit in a quick way?

---

/cc @LarsAsplund

[eine]

I pushed https://github.com/Paebbels/JSON-for-VHDL/tree/JSONTestSuite. A single testbench is used, where a JSON file is loaded and the raw content is printed. No further checks are done. Using VUnit, 319 configurations are created, one for each file in https://github.com/nst/JSONTestSuite/tree/master/test_parsing. 303 of 318 tests pass: https://github.com/Paebbels/JSON-for-VHDL/commit/8df831cbf8b8c0bcbe20c38df6c7c84442fd31c6/checks?check_suite_id=396976882 Nevertheless, this is expected, because I ignored y_, n_ or i_ prefixes.

[Paebbels]

@eine I never imagined, that testing this suite would be so easy... Because of your generic parameter. Python can read all test cases externally.

Currently, You're cloning the testsuite. Should we either link as submodule or always clone and remove the source files from tests/JSONTestSuite? If we do a clone, on the CI server, we should also consider removing these source entirely from the repo to reduce the repo size.

[eine]

At first, I thought about adding it as a submodule. However, later I thought that this repo is likely to be used as a lib in many projects, thus added as a submodule to them. This is the case in VUnit. It doesn't seem to make much sense to have all those projects clone a large testsuite that is not useful. That's why I decided to clone it in CI, and to wait for this discussion.

we should also consider removing these source entirely from the repo to reduce the repo size

What do you mean with "these sources"? The run.py and TestSuite.vhdl together are less than 50 lines. I think it is worth keeping them in the repo, even if the test JSONs are to be cloned/retrieved separately. Otherwise, where should we locate these scripts?

An alternative is to keep these sources along with JSONTestSuite added to the repo but in a feature branch. That would allow to test it periodically, while keeping the "distributable branch" clean. Moreover, we can use a workflow/action to keep the feature branch a commit ahead of master.

[Paebbels]

@eine

we should also consider removing these source entirely from the repo to reduce the repo size

What do you mean with "these sources"?

Some time ago I copied all files from JSONTestSuite into this repo. I like the idea of clone and test on CI as well as having just the test-script to execute in the repo.

• We should atleast delete the files
• I'll investigate about the nuclear option.

[eine]

Some time ago I copied all files from JSONTestSuite into this repo.

Oh. I completely overlooked that. I agree, we should delete them.