Clarify how to proceed with requirement "Test result statement" for stable APIs

[jlurien]

Problem description

API-Readiness-Checklist indicates that for stable APIs a mandatory check is

Nr	API release assets	Explanation
9	Test result statement	A statement in a discussion issue of the API Sub Project by at least one of the API Sub Project members that the Gherkin feature files have been successfully executed against their (lab) API implementation.

Which would imply passing the enhanced test plan to the last RC after M3 in a lab environment, but this is not really feasible for this meta-release, due to the reduced period between M3 and M4, and the delay in the availability of enhanced test plans.

Possible evolution

Discuss at release and project management level how to deal with the exceptionality for this first meta-release, and give guidelines to APIs intending to release stable versions about how to proceed and fill the API-Readiness-Checklist

[tanjadegroot]

In the TSC of 05/09/24, it was decided that EXCEPTIONALLY the 5 stable APIs can be published without having done the test plan execution, as they have been implemented in various industry contexts.
Not having done the execution of the test plans, means that there can be bugs found in the test plans. In this case, these errors should be notified to the Sub-Project which should then prepare a patch release of the API.

The following information is proposed t be added in the M4 release PR in the API Readiness checklist:

• In row with point 9 "Test result statement":
  - in Status column put "N"
  - in Comments column add: "Fall24 EXCEPTION: Test results not available (*)"
• Add the following footnote under the table: "(*) If you encounter issues with the provided test files (.feature), please create an issue in the API Sub-Project to signal these issues so they can be fixed in a patch release."

Please comment on the proposal here, by latest tomorrow friday EOB

[jlurien]

Thanks @tanjadegroot, what about the Status column? Should APIs fill it with "N"?

[hdamker]

Thanks @tanjadegroot, what about the Status column? Should APIs fill it with "N"?

My take would fill it with "N", allows later in a patch release to change it to "Y" (if bugs had to be fixed).

[tanjadegroot]

Thanks @tanjadegroot, what about the Status column? Should APIs fill it with "N"?

yes, let me update the description accordingly

[tanjadegroot]

Implemented in the API-Readiness-Checklist.md of the Fall24 meta-release stable APIs and closed as completed