Proposal: Create new repo for openapi tests, similar to json schema test suite

[spacether]

This issue is a proposal that a new repo be created that stores tests of openapi payloads, request, responses, and documents. These tests will help implementation verify that they have built features correctly.

There is existing prior work here that applies to json schema only in the json schema test suite: https://github.com/json-schema-org/JSON-Schema-Test-Suite

Need for this solution:

• there are few tests here: https://github.com/OAI/OpenAPI-Specification/tree/main/tests/v3.1 and they do not provide full coverage for openapi features, they do not describe what is acceptable and unacceptable for openapi-only schema formats, they may not be well known in the community, they do not have coverage tests for different versions of the openapi spec
• what is allowed for int32/int64/float/double is not explicitly defined in the spec or the format registry (for example is X.0 allowed in for int32?). Implementers need this info when building.
• I attempted to add int64 tests into the json schema test suite and was told that that repo is not the right place for that test, because the format is openapi-specific and is not listed in the json schema spec.

Benefits:

• tooling providers could have a verifiable way of proving that their implementation is conforming to the requirements
• long term this should reduce the number of what is the correct way to implement x feature questions on this repo because you can point them to the existing test
• including these tests in a separate repo will allow implmenters to include that repo as a subpackage and easily test against it. (I currently do this with the json schema test suite here)
• General json schema 2020-12 tests should not be included here because they already exist in the json schema test suite. Refer users to that suite also. This reduces the maintenance workload.

Proposal Short Term:

• create a new repo, Openapi Test Suite in the https://github.com/OAI org
• have one folder for each version of the openapi test suite
• move the existing tests into that
• perhaps organize the tests in the same directory structure as the openap spec?
• add tests in components/schemas/format.json which includes the int64 format tests in: Adds int64 optional format test json-schema-org/JSON-Schema-Test-Suite#711

I would be happy to add additional test for int32/float/double/int formats
@karenetheridge mentioned that she could contribute tests from https://github.com/karenetheridge/OpenAPI-Modern/tree/master/t

Open Questions:

• How thorough should these tests be?
  • My proposal is to incrementally add coverage bit by bit starting with easiest tests, which may be component schema formats.
• Are you okay using the json schema test suite format for the schema test files?
• What openapi versions will have tests?
• Do these tests mean that the tooling MUST/MAY/verb implement these features? (How does json schema handle this Julian Berman?)

[Julian]

Do these tests mean that the tooling MUST/MAY/verb implement these features? (How does json schema handle this Julian Berman?)

The JSON Schema test suite has two "buckets" historically:

• all the top-level tests (tests which live in tests/<version>/*.json represent MUST behavior, both where the spec explicitly uses the word MUST but also where it informally seems plainly to mean that. For example if it says "the foo keyword passes if the instance bars", that is clearly intended to be required definitionally for the keyword foo.
• a second catch-all bucket, optional/, which has created lots of confusion over the years, mostly because it simply represents "everything else" -- they're tests where the specification either: uses MAY language, or where it proscribes behavior which is language or environment specific for external reasons (e.g. because the JSON spec itself does not require a specific precision for integers, so tests which rely on that are not strictly required behavior under the JSON Schema specification either)

The latter was even until now not broken out further mostly because it never was super high priority, and that secondly doing so is disruptive for users of the suite who might be (and in fact do) rely on the layout of tests. But if the suite was (re-)started today I would have started by having explicit directories for MAY, SHOULD, and NOT SPECIFIED THE SPEC DOESN'T SAY ANYTHING AT ALL (and we'll get there eventually).

So if you're going to tackle OpenAPI I'd start there immediately (with explicitly representing what level of requirement the tests represent).

But really my personal unsolicited opinion is to start very concretely on scope -- specifically, if it were me, I would start with a repo whose entire scope is specifically "have format tests for all the OpenAPI Format Registry" and nothing else at all. That has specific well-defined scope, and essentially matches the existing JSON Schema test suite tests, so it'd seem like there should be no need to wait for anything to do that.

All the rest (having suites for the rest of the OpenAPI behavior) seems like it'd be way more open to other opinions -- and the JSON Schema test suite format is unlikely to work for all of what such a suite might want to test, much as it's also clear to me too it'd be useful. So again as I say, if it were me, even if I wanted to tackle both, I'd probably then start with 2 separate repositories, format-registry-suite and openapi-overall-test-suite (or whatever).

But just my 0.02 (and hopefully answered the question there). Happy to help with other things if you all need the hand, or else to answer any other questions.

[handrews]

See also #3510 regarding the existing linked in the 2nd paragraph of the original comment.

I agree that the format registry is the best scope to start with. You could expand that with tests of other OAS extensions to JSON Schema, but these are really different suites with different expectations of conformance.

Tests for the OAS as a whole are the provenence of OAI/oascomply, and are a far more complex problem because the OAS does not define measurable output behavior.

But format registry and JSON Schema extensions (which were excluded from the scope of OASComply) make sense to track here.

[spacether]

This came up for format: double and format: float here: openapi-json-schema-tools/openapi-json-schema-generator#426
And it came up in openapi-generator here: OpenAPITools/openapi-generator#14499