Contract based unit tests

[jfiala]

Description

Currently there are no unittests ensuring that the contracts (i.e. the outcome of the language-generators) change only when they are supposed to change. What about adding at least one unittest for each language to secure this and to track all the generated output files (better their hashes)? For different configuration options of the language maybe additional tests make sense. I think this would also track breaking changes a lot easier...

[jfiala]

Automatically generating the output for all the petstore-variations and also allow for different language configurations would make the results more reliable. The current process is unreliable because if someone forgets to update the generated files they will not be validated. IMHO the expected outcome should be fixed, but the actual outcome should always get generated dynamically.

@wing328 can you please review the idea? If you want I can setup a separate module for this. It would also allow to replace the shellscripts with an integrated maven build.

[wing328]

@jfiala is your suggestion similar to https://github.com/civisanalytics/swagger-diff, which shows the differences between 2 Swagger specs?

[jfiala]

No, I think the manual process of running the shellscripts is tiresome and leads to outdated generated sample files (I've had this numerous times and then the updater is forced to deal with any bugs that are due to outdated sample files...), which forces the updated to also fix other bugs that have creeped into the language. Also, the shellscripts use different contracts which adds additional complexity.
Finally, the shellscripts are platform specific and make it hard for windows users to work with them. As we use Java we should use unittests, not shellscripts to verify outcomes IMHO.

I'd suggest to setup a separate integration-test project which will automatically generate all the languages including test variations with every build, forcing language updaters to be always consistent and otherwise the build will fail.

Currently forgetting to update the samples leads to no build errors and therefore bugs creep into the languages without getting noticed.

So my suggestion basically should be:

1. add a separate test module which generates all samples automatically using unittests and which will fail if the expected outcome differs from the actual (generated) outcome, forcing language updates to update the expected outcome and thus maintain the tests.

2. add additional tests for the test variations (e.g. security-contract petstore-securityy-test.yaml etc.).

3. maybe also implement cross-language consistency checks, as some features should be consistent throughout all languages

4. remove the checkbox (Ran the shell/batch script under ....) in the PR template because the build will fail if the outcomes are not up to date.

What do you think?

[jfiala]

@wing328 please take a look at the PR #4721, if you like it we can migrate all the shellscripts into unittests and also add the directory layout comparison to make sure everything is equal, maybe also improve the error reporting.

Then you can also remove the "Ran the shell/batch script under ./bin/ ..." task from the template, as this will be run automatically. Developers can then simply copy the generated output into the expected output and then check if the difference is intended.

[jfiala]

@wing328 can you pls have a look at this suggestion to replace the shellscript and to enforce always up-to-date generated samples? Nearly every time I want to improve a language the samples are out of date and would cause compile errors if they are generated correctly. It is no fun having to generate the samples, fix errors due to outdated samples and then being able starting the real work. Just wondering why nobody else complains about this... (latest example e.g. #4924)