MP OpenAPI 3.1 Feature Test Summary

[Azquelt]

Test Strategy

Describe the test strategy & approach for this feature, and describe how the approach verifies the functions delivered by this feature.

For any feature, be aware that only FAT tests (not unit or BVT) are executed in our cross platform testing. To ensure cross platform testing ensure you have sufficient FAT coverage to verify the feature.

If delivering tests outside of the standard Liberty FAT framework, do the tests push the results into cognitive testing database (if not, consult with the CSI Team who can provide advice and verify if results are being received)?

List of FAT projects affected

Test strategy

• New and updated functionality:

  • extensions attribute added to most annotations 387
    • TCKs cover using the extensions attribute on one standard annotation, one container annotation and applying the @Extension annotation directly to a method.
    • More TCKs exhaustively cover each new extensions attribute
  • Improvements to the definition of security requirements 483, 468
    • Define behavior of @SecurityRequirementsSet and make it repeatable
    • Clarify that a individual @SecurityRequirement annotation applied to a class or method is equivalent to a @SecurityRequirementsSet annotation containing that @SecurityRequirement annotation
    • Add securitySets attribute to @OpenAPIDefinition and @CallbackOperation
    • TCK tests added
      • Use of @SecurityRequirementsSet, includes requirement for multiple auth methods and for optional authentication
      • Use of @SecurityRequirementsSet on a resource method
      • Use of @SecurityRequirementsSet in the securitySets attribute of @OpenAPIDefinition
      • Use of @SecurityRequirementsSet in the securitySets attribute of @CallbackOperation (added in #539)
  • Add additionalProperties attribute to @Schema 423
    • TCK tests added for values True, False and String
  • Allow @APIResponse to be applied to a class, indicating that every resource method on that class has that response 417
    • TCK tests added for
      • @APIResponse on a resource class
      • @APIResponse on an exception mapper class
  • Add processing of some Jakarta Bean Validation annotations 482
    • TCK tests added to test all supported annotations when applied to a field
    • TCK test added to test an annotation applied to a resource method parameter
  • Define the precedence of the mp.openapi.scan.* config properties 422
    • TCK tests added which test all combinations of config properties

• Which FAT projects are affected?

  • TCK - io.openliberty.microprofile.openapi.3.1.internal_fat_tck
  • Repeated FAT buckets - io.openliberty.microprofile.openapi.2.0.internal, com.ibm.ws.microprofile.openapi_fat
    • No new tests added but verifies we haven't broken existing functionality

• What manual tests are there (if any)? (Note: Automated testing is expected for all features with manual testing considered an exception to the rule.)

  • We have existing manual tests for the UI but there are no changes to them for this release.

Confidence Level

Collectively as a team you need to assess your confidence in the testing delivered based on the values below. This should be done as a team and not an individual to ensure more eyes are on it and that pressures to deliver quickly are absorbed by the team as a whole.

Please indicate your confidence in the testing (up to and including FAT) delivered with this feature by selecting one of these values:

3 - We have delivered all automated testing we believe is needed for the golden paths of this feature and minimal coverage of the error/outlying scenarios. There is a risk when the feature is used outside the golden paths however we are confident on the golden path. Note: This may still be a valid end state for a feature... things like Beta features may well suffice at this level.

We currently only have manual testing for the UI, though it checks all of our customizations and touches all of the core functionality as well as some more obscure bits and we perform the testing every time we make any change to the UI projects.

We think we have good coverage of all new functionality added in this release including outlying scenarios, though there are a few unlikely error cases not tested.

[ayoho]

Coverage sounds good to me 👍