LG-293: Add ability for requestTiming hungRequest detection to not collect javacores

[donbourne]

Provide a means to disable the collection of javacores during hungRequest detection in requestTiming-1.0.

When ready, add links to the Upcoming Feature Overview document and Feature Test Summary issue:

• UFO: https://ibm.box.com/s/dsxkj60s6s29vso5piys5sk9wwhjd0nd
• FTS: Feature Test Summary for LG-293: Add ability for requestTiming hungRequest detection to not collect javacores #16104
• ID: RequestTiming-1.0 doc update for LG-293: Add ability for requestTiming hungRequest detection to not collect javacores docs#3764

---

List of Steps to complete or get approvals / sign-offs for Onboarding to the Liberty release (GM date)

Instructions:

• Do the actions below and mark them complete in the checklist when they are done.
• Make sure all feature readiness approvers put the appropriate tag on the epic to indicate their approval.

Design

Before Development Starts or 8 weeks before Onboarding

• POC Design / UFO Review Scheduled (David Chang) or N/A.
• POC Design / UFO Reviewed (Feature Owner) or N/A.
• Complete any follow-ons from the POC Review.
• Design / UFO Approval (Alasdair Nottingham) or N/A.
• No Design / No UFO Approval (Arthur De Magalhaes - cloud / Alasdair Nottingham - server) or N/A.
• SVT Requirements identified. (Epic owner / Feature owner with SVT focal point)
• ID Requirements identified. (Epic owner / Feature owner with ID focal point)
• Create a child task of this epic entitled "FAT Approval Test Summary". Add the link in above.

Legal

3 weeks before Onboarding

• Identify all open source libraries that are changing or are new. Work with Legal Release Services (Cass Tucker or Release PM) to get open source cleared and approved. Or N/A. (Epic Owner). New or changed open source impacts license and Certificate of Originality.

Translation

3 weeks before Onboarding

• All new or changed PII messages are checked into the integration branch, before the last translation shipment out. (Epic Owner)

Feature Complete

2 weeks before Onboarding

• Implementation complete. (Epic owner / Feature owner)
• All function tests complete. Ready for FAT Approval. (Epic owner / Feature owner)
• Review all known issues for Stop Ship. (Epic owner / Feature owner / PM)

Focal Point Approvals

2 to 1 week before Onboarding

You MUST have the Design Approved or No Design Approved label before requesting focal point approvals.

All features (both "Design Approved" and "No Design Approved")

• FAT - (Kevin Smith). SOE FATS are running successfully or N/A . Approver adds label focalApproved:fat to the Epic in Github.
• Demo - (Tom Evans or Chuck Bridgham). Demo is scheduled for an upcoming EOI. Approver adds label focalApproved:demo to the Epic in Github.
• Globalization (Sam Wong - Liberty / Simy Cheeran - tWAS). Translation is complete or N/A. TVT - complete or N/A. Approver adds label focalApproved:globalization to the Epic in Github.

"Design Approved" features

• Accessibility - (G Scott Johnston). Accessibility testing is complete or N/A. Approver adds label focalApproved:accessibility to the Epic in Github.
• ID - (Kareen Deen). Documentation work is complete or N/A . Approver adds label focalApproved:id to the Epic in Github.
• Performance - (Jared Anderson). Performance testing is complete with no high severity defects or N/A . Approver adds label focalApproved:performance to the Epic in Github.
• Serviceability - (Don Bourne). Serviceability has been addressed.
• STE - (Swati Kasundra). STE chart deck is complete or N/A . Approver adds label focalApproved:ste to the Epic in Github.
• SVT - (Brian Hanczaryk- APS). SVT is complete or N/A . Approver adds label focalApproved:svt to the Epic in Github.

Ready for GA

1 week before Onboarding

• No Stop Ship issues for the feature. (Epic owner / Feature owner / Release PM)
• Ship Readiness Review and Release Notes completed (Epic owner / Feature owner / Release PM)
• Github Epic and Epic's issues are closed / complete. All PRs are committed to the master branch. (Epic owner / Feature owner / Backlog Subtribe PM)

Other deliverbles

• OL Guides - (Yee-Kang Chang). Assessment for OL Guides is complete or N/A.
• WDT - (Leonard Theivendra). WDT work complete or N/A.
• Blog - (Laura Cowen) Blog article writeup (Epic owner / Feature owner / Laura Cowen)

[cthigh]

UFO review - Monday Dec 7, 2020.

Presenter: Prashanth Gunapalasingam
Moderator: Cindy High

Notes:

1. Slide 2 - What do we do on non-OpenJ9? Will this work on any Java? OpenJ9 has Java cores. Hotspot has thread dumps.

   Suggestion: Rename enableJavacore to enableThreadDumps to be more generic for both OpenJ9 and Hotspot.

2. Slide 2 - Add default behavior to the end user overview. Enabled by default. (default is described on slide 5)

3. Slide 6 - All messages on this slide include “Java dump”. Should the attribute be “Java dump”. Can the message text be changed to say “Java thread dump” or something else? The message is from kernel (framework manager) and there are questions whether it should change but it would be nice to have consistency between message and configuration parameter. Take off-line.

4. Page 7 - By default, this feature will not help the first problem of filling the file system. Jared noted that a concept was created to manage file system space using a rolling list of JavaCores (https://github.com/OpenLiberty/open-liberty/pull/3363/files ) that might help manage the first issue.

5. Slide 18 - Confusion: appName, ServletName are bold but not new. Perhaps clarify that bold indicates changed use or behavior.

6. Slide 19 - Comment - JDBCTiming - “enableJavacore=“true” is redundant since true is the default

7. Slide 30 - The new message says “A series of java cores will not be created for the hung request…..”. Update the message using the final terminology for “thread dump” to have generic context for OpenJ9 and Hotspot. Resolve with issue Update local JDK wiring #3.

[pgunapal]

UFO has been updated with comments from the UFO socialization on Dec 7th and is ready for Design Review Approval.

[pgunapal]

@samwatibm The new messages have been translated and will be checked in this week. May i get the globalization approval for this feature?

[pgunapal]

@steven1046 There are no UI-related changes/updates for this feature, as it does not have any UI. May I get the Accessibility approval for this feature?

[pgunapal]

@jhanders34 Performance testing is not needed for this feature, as we are providing an option to disable the creation of thread dumps for hung requests in the requestTiming-1.0 feature and will not regress performance. May I get the Performance approval for this feature?

[pgunapal]

@samwatibm The translated messages are in master. May I have the Globalization approval for this feature?

[pgunapal]

@hanczaryk It was decided in the UFO review that we don't need SVT for this feature, as the capability is easily tested and covered from the existing FAT tests. Would you mind reviewing the SVT approval for this feature?

[pgunapal]

@donbourne Serviceability Approval Comment :

UFO -- does the UFO identify the most likely problems customers will see and identify how the feature will enable them to diagnose and solve those problems without resorting to raising a PMR? Have these issues been addressed in the implementation?
- Yes, if the enableThreadDumps="false" is configured and a hung request is detected. a debug trace message will be logged, indicating that a series of java thread dumps will not be generated for that particular hung request. Also, if users incorrectly specify the enableThreadDumps attribute in the server.xml, a warning message will also be logged, and the default value will be used. Also, we will thoroughly document the behaviour of how the sub-element configurations will be override the root element configurations in the requestTiming-1.0.

Test and Demo -- As part of the serviceability process we're asking feature teams to test and analyze common problem paths for serviceability and demo those problem paths to someone not involved in the development of the feature (eg. L2, test team, or another development team).
a) What problem paths were tested and demonstrated?
   - If users specify the attribute name or value incorrectly, a warning message was logged and the default value was used.
   - If the attribute is not specified in server.xml, it will use the default value, and create thread dumps.
   - If both the root and sub elements have the same attribute, but different values, the sub-element value will be used.
b) Who did you demo to?
   - Observability squad
c) Do the people you demo'd to agree that the serviceability of the demonstrated problem scenarios is sufficient to avoid PMRs for any problems customers are likely to encounter, or that L2 should be able to quickly address those problems without need to engage L3?
   - Yes, they did. Only concern was that they found that the sub-element configuration overriding the root element was irregular, however, the other requestTiming configuration attributes behave similarly, hence, we decided to thoroughly document this.

SVT -- SVT team is often the first team to try new features and often encounters problems setting up and using them. Note that we're not expecting SVT to do full serviceability testing -- just to sign-off on the serviceability of the problem paths they encountered.
a) Who conducted SVT tests for this feature?
   - It was decided in the UFO review that we don't need SVT for this feature, as the capability is easily tested and covered from the existing FAT tests.
b) Do they agree that the serviceability of the problems they encountered is sufficient to avoid PMRs, or that L2 should be able to quickly address those problems without need to engage L3?
 - N/A

Which L2 / L3 queues will handle PMRs for this feature? Ensure they are present in the contact reference file and in the queue contact summary, and that the respective L2/L3 teams know they are supporting it. Ask Don Bourne if you need links or more info.
- WAS L3: Logging

[skasund]

L2 has requested a live STE presentation for this feature. The STE templates can be found at the links below. You can use either one to create the education.

Slide Template: https://ibm.box.com/s/1an42g7zdgmaj84w7dft0indqfgi8ffm

Github Template: https://pages.github.ibm.com/WASL3/site/STE/about

Please upload the completed slides to the same 'STE Archive' BOX folder or provide me the Github link, and I will provide the STE approval. The live presentation can be scheduled with the L2 team at a later date. Thanks!

[chirp1]

Info for the the ID issue at OpenLiberty/docs#3764 contains the information needed to write the documentations. Approving.

[pgunapal]

@skasund The completed STE slides are uploaded to the STE Archive BOX folder.

[skasund]

I've provided the STE approval. L2 will work with you to schedule a live session.

[pgunapal]

Closing as this feature has been released and GA'ed in 21.0.0.4 (https://openliberty.io/downloads/#runtime_releases)