Feature Test Summary - Jakarta Enterprise Beans 4.0

[tkburroughs]

Test Strategy

Describe the test strategy & approach for this feature, and describe how the approach verifies the functions delivered by this feature.

For any feature, be aware that only FAT tests (not unit or BVT) are executed in our cross platform testing. To ensure cross platform testing ensure you have sufficient FAT coverage to verify the feature.

If delivering tests outside of the standard Liberty FAT framework, do the tests push the results into cognitive testing database (if not, consult with the CSI Team who can provide advice and verify if results are being received)?

As indicated in the UFO, there are really only 4 differences between EJB 3.2 and Enterprise Beans 4.0 that are relevant to FAT:

1. the feature names have changed from ejbXxxx-3.2 to enterpriseBeansXxxx-4.0 (except for mdb-3.2 -> mdb-4.0).
2. a few methods were removed from the EJBContext and/or SessionContext
3. the @Schedule annotation is now repeatable
4. there is a liberty specific EJB container API, which relies on javax packaging, so will be transformed and renamed (com.ibm.websphere.appserver.api.ejbcontainer.jar -> io.openliberty.ejbcontainer.2.0.jar)

FAT coverage will be accomplished as follows:

1. Since the function is otherwise identical, the Enterprise Beans 4.0 feature will provide FAT coverage primarily by using the Jakarta EE 9 Repeat Action, which will run the existing FAT with the new enterpriseBeansXxxx-4.0 feature names. FATs will now repeat for Jakarta EE 7, Jakarta EE8, and Jakarta EE 9. Some of each variety will be run in "Lite" mode, and others in "FULL" mode.
2. Existing FAT that covers the removed methods will be updated such that they continue testing the removed methods only for the older spec levels.
3. A new FAT test will be added to cover the fact that @Schedule is now repeatable.
4. FAT that covers the EJB Container API will be repeated for Jakarta EE.

List of FAT projects affected

Most EJB FAT projects cover multiple EJB features, but they are separated below by the primary feature they test to demonstrate that all features are covered:

• enterpriseBeanLite-4.0

  • com.ibm.ws.ejbcontainer.async_fat
  • com.ibm.ws.ejbcontainer.exception_fat (also Home & Remote)
  • com.ibm.ws.ejbcontainer.interceptor_fat (also PersistentTimer & mdb)
  • com.ibm.ws.ejbcontainer.interceptor.v32_fat (also Home)
  • com.ibm.ws.ejbcontainer.session.async_fat (also enterpriseBeans-4.0 & mdb)
  • com.ibm.ws.ejbcontainer.session.passivation_fat
  • com.ibm.ws.ejbcontainer.timer.auto_fat
  • com.ibm.ws.ejbcontainer.timer.calendar_fat (also enterpriseBeansHome & enterpriseBeansPersistentTimer)
  • com.ibm.ws.ejbcontainer.timer.np.config_fat
  • com.ibm.ws.ejbcontainer.timer.np_fat
  • com.ibm.ws.ejbcontainer.timer.v32_fat
  • com.ibm.ws.ejbcontainer.tx_fat (also mdb-4.0)
  • com.ibm.ws.ejbcontainer.war_fat
  • com.ibm.ws.injection_fat (also Remote & mdb)
  • io.openliberty.ejbcontainer.v40_fat

• enterpriseBeansHome-4.0 (i.e. EJB 2.x)

  • com.ibm.ws.ejbcontainer.bindings_fat (also Remote)
  • com.ibm.ws.ejbcontainer.exception_fat (also Lite & Remote)
  • com.ibm.ws.ejbcontainer.legacy_fat (also Remote)

• enterpriseBeansPersistentTimer-4.0

  • com.ibm.ws.ejbcontainer.timer.persistent_fat

• enterpriseBeansRemote-4.0

  • com.ibm.ws.ejbcontainer.cdi_fat
  • com.ibm.ws.ejbcontainer.exception_fat (also Lite & Home)
  • com.ibm.ws.ejbcontainer.legacy_fat (also Home)
  • com.ibm.ws.ejbcontainer.remote_fat (also Lite & Home)

• enterpriseBeansRemoteClient-2.0 (private feature in application client)

  • com.ibm.ws.ejbcontainer.remote.client_fat (also enterpriseBeans-4.0 convenience feature)

• mdb-4.0

  • com.ibm.ws.ejbcontainer.mdb.ra_fat
  • com.ibm.ws.ejbcontainer.session.async_fat (also enterpriseBeans-4.0 and Lite)
  • com.ibm.ws.ejbcontainer.tx_fat (also Lite)

• enterpriseBeans-4.0 (convenience feature including all of the above)

  • com.ibm.ws.ejbcontainer.remote.client_fat (also RemoteClient)
  • com.ibm.ws.ejbcontainer.session.async_fat (also Lite and mdb-4.0)

• test that contains removed methods (new in EJB 4.0)

  • com.ibm.ws.ejbcontainer.legacy_fat

• test for @Schedule as repeatable (new in EJB 4.0)

  • io.openliberty.ejbcontainer.v40_fat

• test that covers the EJB Container API (repeated for Jakarta)

  • com.ibm.ws.ejbcontainer.bindings_fat
  • com.ibm.ws.ejbcontainer.injection_fat
  • com.ibm.ws.ejbcontainer.remote.client_fat

Test strategy

• What functionality is new or modified by this feature?
• What are the positive and negative tests for that functionality?
• What manual tests are there (if any)? (Note: Automated testing is expected for all features with manual testing considered an exception to the rule.)

The general test strategy for the Jakarta EE 9 Features, including Jakarta Enterprise Beans 4.0, is to repeat all tests run for Java EE 8 in open-liberty by transforming them to use the Jakarta packages.

Confidence Level

Collectively as a team you need to assess your confidence in the testing delivered based on the values below. This should be done as a team and not an individual to ensure more eyes are on it and that pressures to deliver quickly are absorbed by the team as a whole.

Please indicate your confidence in the testing (up to and including FAT) delivered with this feature by selecting one of these values:

0 - No automated testing delivered

1 - We have minimal automated coverage of the feature including golden paths. There is a relatively high risk that defects or issues could be found in this feature.

2 - We have delivered a reasonable automated coverage of the golden paths of this feature but are aware of gaps and extra testing that could be done here. Error/outlying scenarios are not really covered. There are likely risks that issues may exist in the golden paths

3 - We have delivered all automated testing we believe is needed for the golden paths of this feature and minimal coverage of the error/outlying scenarios. There is a risk when the feature is used outside the golden paths however we are confident on the golden path. Note: This may still be a valid end state for a feature... things like Beta features may well suffice at this level.

4 - We have delivered all automated testing we believe is needed for the golden paths of this feature and have good coverage of the error/outlying scenarios. While more testing of the error/outlying scenarios could be added we believe there is minimal risk here and the cost of providing these is considered higher than the benefit they would provide.

5 - We have delivered all automated testing we believe is needed for this feature. The testing covers all golden path cases as well as all the error/outlying scenarios that make sense. We are not aware of any gaps in the testing at this time. No manual testing is required to verify this feature.

Based on your answer above, for any answer other than a 4 or 5 please provide details of what drove your answer. Please be aware, it may be perfectly reasonable in some scenarios to deliver with any value above. We may accept no automated testing is needed for some features, we may be happy with low levels of testing on samples for instance so please don't feel the need to drive to a 5. We need your honest assessment as a team and the reasoning for why you believe shipping at that level is valid. What are the gaps, what is the risk etc. Please also provide links to the follow on work that is needed to close the gaps (should you deem it needed)

Answer : 4

All existing EJB container FAT in open-liberty is being repeated for Jakarta EE 9, and one additional FAT has been added for the one unique Jakarta EE 9 capability (repeatable @Schedule). Note that several FAT projects that previously existed only for WebSphere Liberty have now been moved to open-liberty and repeat for Jakarta EE 9.

While additional EJB container FAT exists that only runs for WebSphere Liberty and or WebSphere traditional that is not currently repeated for Jakarta EE 9, since the code base is common and the fat present in open-liberty does cover all of the EJB container features, our review of the FAT concludes that the FAT coverage is at a level 4.

[ayoho]

Coverage sounds good to me 👍