Update Onboarding feature to automate the identifying APEX release scenarios

[yogananda0517]

This work is a pre-requisite to completing the requirement of identifying APEX release plans - #6262.

GitHub issues related to this epic:

• DataScope and MgmtScope  #8841
• Feedback onboarding APEX feature #8837

APEX out of scope scenario: When onboarding the product that are out of scope for Data & Management Plane. There is no option to indicate this, you can only choose the option "I am not sure which of the above options apply to my product". Feedback from service partners who do not have any REST APIs is that they are frustrated in onboarding to the Azure SDK team using the Release Planner app. Since we change the roles of "who" completes the onboard survey in fall 2022, nearly 100% of the scope is correctly identified in the triage work item.

• Add aka link for determining REST API scope to the API details UI screen - https://aka.ms/azsdk/rest-api-scope

JP's recommendation:
mockup.html (2).zip

Requirement:

APEX out of scope scenario UI changes

Preview Give feedback

1. Update UI to add a button for "The product is not enabled by ARM (management plane) or non-ARM (data plane) REST APIs".
2. Update the text in the first bullet to add reference to ARM REST APIs so that is clear. "This product creates, or updates ARM Manifest resource types (ARM REST APIs)"
3. Create a link to the "Determining management plane (ARM) and/or data plane scope" section of the onboard doc and create an aka link to it (service partners are having difficulty finding this information from the release planner app).
4. If the new option (not in scope) is selected change the datascope and mgmtscope fields of the triage and product work item to No.
5. Update the unsure option. The unsure option should be specific for either plane not grouped together. In most cases, teams are going to be unsure about data plane. There needs to be an unsure option for data and management in the onboarding UI.
6. Management and Data Plane need to have an option to identify when there are no changes to the REST APIs/SDKs for a new product. For example, a SKU that does not require any changes to the ARM REST API, such as adding new resource types or properties.
7. After user reviews & submits: If the product is out of scope for APEX, then display a message to the user that provides information needed for the user to mark all of the Azure SDK APEX requirements as "Not Applicable". Message content: "<Product Name/> is out of scope for the Azure SDK APEX requirements. Access the Cloud Lifecycle (CLC) entity for <Product Name/>: <link to the CLC entity/>. Mark all Azure SDK APEX requirements as Not Applicable (N/A). "

APEX out of scope scenarios data automation/back-end changes

Preview Give feedback

1. IF user selects the option "The product is not enabled by ARM (management plane) or non-ARM (data plane) REST APIs", THEN epic-product type DataScope and MgmtScope values are set to "No". See this spreadsheet tab - https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/ETD9YCt4cfxHlL4mC1KlVygBIQss1mm6NMIpu3_BdOJhJQ?e=lRzfw3&nav=MTVfe0YwRTQ2MjhCLTBCNzktNDVFNC1CQzU0LUFFRTJBRDM5QTM5OX0
2. IF user selects the option "The product is not enabled by ARM (management plane) or non-ARM (data plane) REST APIs", THEN Triage Work Item state = "Out of scope APEX" See this spreadsheet tab - https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/ETD9YCt4cfxHlL4mC1KlVygBIQss1mm6NMIpu3_BdOJhJQ?e=lRzfw3&nav=MTVfe0YwRTQ2MjhCLTBCNzktNDVFNC1CQzU0LUFFRTJBRDM5QTM5OX0
3. IF user selects the option "The product is not enabled by ARM (management plane) or non-ARM (data plane) REST APIs", all launch criteria fields in the epic type = product need to have value set to "N/A". See this spreadsheet tab - https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/ETD9YCt4cfxHlL4mC1KlVygBIQss1mm6NMIpu3_BdOJhJQ?e=lRzfw3&nav=MTVfe0YwRTQ2MjhCLTBCNzktNDVFNC1CQzU0LUFFRTJBRDM5QTM5OX0
4. IF user selects the option "The product is not enabled by ARM (management plane) or non-ARM (data plane) REST APIs" PrivPrev State, PubPrev State, and GA State in the epic type = product need to have value set to "Out of scope". See this spreadsheet tab - https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/ETD9YCt4cfxHlL4mC1KlVygBIQss1mm6NMIpu3_BdOJhJQ?e=lRzfw3&nav=MTVfe0YwRTQ2MjhCLTBCNzktNDVFNC1CQzU0LUFFRTJBRDM5QTM5OX0
5. If the user selects option that there are no changes to the REST APIs/SDKs for a new product for either management or data plane, then all launch criteria fields in the epic type = product need to have value set to "N/A". See this spreadsheet tab - https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/ETD9YCt4cfxHlL4mC1KlVygBIQss1mm6NMIpu3_BdOJhJQ?e=lRzfw3&nav=MTVfe0YwRTQ2MjhCLTBCNzktNDVFNC1CQzU0LUFFRTJBRDM5QTM5OX0
6. If the user selects option that there are no changes to the REST APIs/SDKs for a new product for either management or data plane, then the relevant lifecycle state lifecycle fields values need to be updated. See this spreadsheet tab - https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/ETD9YCt4cfxHlL4mC1KlVygBIQss1mm6NMIpu3_BdOJhJQ?e=lRzfw3&nav=MTVfe0YwRTQ2MjhCLTBCNzktNDVFNC1CQzU0LUFFRTJBRDM5QTM5OX0

Identify APEX release scenario: Instead of manually determining if a product that onboarded is in scope for APEX, optimize process by identifying and confirming the product onboarded is in scope for APEX.

Identify if product is in scope for APEX UI changes

Preview Give feedback

1. All changes for APEX out of scope scenarios (above) must be completed first because the data is used to determine the value of the isAPEX field in the epic type = product.
2. Add option to show user if they are potentially in scope for APEX. Allow user to edit/confirm APEX scope.
3. This option should show up after the use has selected options on first step in onboarding flow.

Identify if product is in scope for APEX data automation/back-end changes

• If during the first step of the onboarding wizard the user identified that ""The product is not enabled by ARM (management plane) or non-ARM (data plane) REST APIs" . then the isAPEX field value in the epic type = product is set to "no".
• If during the first step of the onboarding wizard the use identifies that either management or data plane is in scope for the product, then check the product lifecycle. If the product lifecycle is not GA, then display to user "The product current lifecycle is <product lifecycle">, confirm that this product is in scope for APEX. If product lifecycle is GA, then check isAPEX field value in the epic type = product. If the value is "no", then do not show anything. APEX is complete. If the value is "yes", the display message to user "The product lifecycle is GA and APEX has not been completed. Ask user to confirm."

Azure SDK Info Meeting Required Updates

• Review & update rules for determining when an Info Meeting is required - https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/ETD9YCt4cfxHlL4mC1KlVygBIQss1mm6NMIpu3_BdOJhJQ?e=ThDqfB&nav=MTVfezUwQ0FENzNDLTlEN0YtNDkzRi1BNTEzLTY5MjBBRjZFNTg3Rn0

Updated onboarding email templates based on scenario

• All new onboarding email templates - https://microsoft-my.sharepoint.com/:f:/p/ladonnaquinn/EnTlkCIg0JdIqFuSLVJUpK4BlPHf9nlYfYli0dAg0B3oKg?e=EDFEvz

Scenario Testing

• Onboard scenario testing - https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/EQOYpXEI_V5Em4S2atNe0JABJn3HmJyJP8_f5kSepo6EgQ?e=4tseIt
• Overall test results as of August 19, 2024 - https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/EQOYpXEI_V5Em4S2atNe0JABJn3HmJyJP8_f5kSepo6EgQ?e=vUl6jT&nav=MTVfezI5RDA3NTA3LUJENjgtNEU0QS04RkEzLTc2NkU1MUM0NzdFM30

[maririos]

When onboarding the product that are out of scope for Data & Management Plane

This seems specific to APEX, but this question is generic enough that only asks if the Product targets ARM or data plane.

@ladonnaq could you clarify?

[ladonnaq]

When onboarding the product that are out of scope for Data & Management Plane

This seems specific to APEX, but this question is generic enough that only asks if the Product targets ARM or data plane.

@ladonnaq could you clarify?

There are products that are in scope for APEX that have no RP (not deployed via ARM, not cloud app) and do not have non-ARM REST APIs (data plane), so they are out of scope for the Azure SDK APEX requirements but they are in scope for other APEX requirements (i.e. security, ...). They still have to onboard with us so we can validate they are out of scope and we still have to keep records. For example, all of the virtual machine skus that are products for the Compute Platform. We still have to ask them to onboard so we have a triage work item to document they are out of scope, then ask them to mark the Azure SDK requirements as Not Applicable, and then we have to go to Cloud Lifecycle and approve all of the Azure SDK APEX requirements as Not Applicable. Right now it seems strange that they have to choose the option of "I am not sure which of the above options apply to my product" because they already likely know.... none of the options apply. :-)

[maririos]

What do we do with those teams though? they don't have a REST API right? or an SDK? because they are not ARM or data plane...
And how often does this happen? I don't want to confuse our main constumers if this is a very specific case

[ladonnaq]

What do we do with those teams though? they don't have a REST API right? or an SDK? because they are not ARM or data plane... And how often does this happen? I don't want to confuse our main constumers if this is a very specific case

There are products that are in scope for APEX but they are not in scope for the Azure SDK APEX Requirements. They still need to onboard with us so that we can validate that they are truly not in scope. In fact, we just had a service team send us an email on this exact issue. They are not in scope and were blocked from onboarding. I told Yoga to tell them to answer unsure and we would change the scope in the triage work item and mark it as out of scope APEX.

Of the 100 products now showing up to complete a lifecycle milestone by Dec 2023, about 15 of those are out of scope for Azure SDK APEX requirements. There are scenarios where a service introduces a new product and they are not making any changes to the management plane and/or data plane REST APIs. I see this happen from time to time with SKUs mostly. There are also all of the VM SKUs which are almost always out of scope for us but we still need to keep history (so we know talked to them and they are out of scope) and we still have to go to Cloud Lifecycle and approve their attestation of N/A (Not Applicable).

[maririos]

If I understand correctly, either way PM team needs to talk to them to understand the scenario and leave the record of the conversation. We can go ahead get full requirements for this and then prioritize or we can keep asking this teams to use the unsure option like they are already doing. Open to whichever option

[ladonnaq]

If I understand correctly, either way PM team needs to talk to them to understand the scenario and leave the record of the conversation. We can go ahead get full requirements for this and then prioritize or we can keep asking this teams to use the unsure option like they are already doing. Open to whichever option

I will schedule some time with @ccbarragan this week to discuss and a few other gaps/new requirements that will require UI changes and feedback for overall UX. I believe that we should do this because it is a valid use case and at this time service teams are sending us emails or finding us on teams to ask us "What to I do because the new product will not change any of the existing ARM REST API Spec for to enable so therefore would be not applicable for any of the Azure SDK APEX requirements". We this with many of the VM skus under Compute RP and at times with new SKUs that are only making changes to billing or something that does not require any changes to the ARM REST API Spec.

FYI - Here is the triage work item breakdown by state for all, data plane, management plane. So at this time we have 29 products that we have evaluated and determined out of scope for APEX Azure SDK requirements and they are mostly associated with the Compute RP. Here are some metrics from ADO query for triage so you can see the breakdown.

[ladonnaq]

@maririos I will target looking at this again in Feb 2024. In ADO we have around 31 products (from 2023) that we identified as not in scope for APEX. Our process requires that all products in scope for APEX onboard with us so we have a record we spoke with them and identified the product as out of scope for the Azure SDK APEX requirements. We had several service partners confused because they knew they did not have any ARM or data plane REST APIs but there was no option. We told them to just choose the "I don't know" option so they could complete onboarding but that is really a hack. We should add a checkbox for the user's to self-attest that they have no ARM or data plane REST APIs/SDKs required to enable the new product.

[ladonnaq]

@maririos I have had several teams mention that not having a "none of the above option" is frustrating and confusing. Remember that we updated our instructions/guidance and the onboarding form is typically completed by engineering or a product PM (who is usually involved in the design of the REST APIs).

This feedback from a senior software engineering manager is for a scenario where the product is in scope for APEX but not in scope for ANY of the Azure SDK requirements -

I would like this prioritized to be done by beginning of March and a new option added for "My product is not enabled by ARM (management plane) or non-ARM (data plane) REST APIs and SDKs.

I would like a link to https://aka.ms/azsdk/onboard as help/guidance for the bullet. It would be good if we could perhaps make the section that describes how to determine API scope a link so it would take user's directly to where they need to go on the docs page.

I will update the descriptions so we have a summary of the todos.

[maririos]

I will update the descriptions so we have a summary of the todos.

This will be helpful as of right now, reading the comments I am unclear on how to proceed. For example:

• Besides adding a new option with the text you want, what does that mean for the back end? does that change the email templates?
• What are the documentation links you want to add and to where in the UI

[ckairen]

Just a couple of questions to clarify things~~

• Should we have a separate step for APEX eligibility? Right now it's under REST API details and it seems a little bit confusing for me. plus there's no text specifying this is for APEX scope.
• For SDK details tab, would it be better if we have 2 checkboxes of "this product has public SDK for mgmt" and "this product has public SDK for data plane"
• Right now there's also no backend on ado for separate "isPublicSDK" for mgmt and data plane, right now it's just both or neither. We'll need to add backend support for that
  • Once we change that to be separate between mgmt and data plane, where did we reference the old value and what would be affected?
  • What happens to the outdated data in the database after we make this change? How do we want to present it?
• For review and submit tab, shouldn't mgmt and data plane APEX eligibility be shown separately?
• If the product is in GA, is APEX still able to be in scope? I thought APEX is only for non GA cycles?

[ladonnaq]

Just a couple of questions to clarify things~~

1. Should we have a separate step for APEX eligibility? Right now it's under REST API details and it seems a little bit confusing for me. plus there's no text specifying this is for APEX scope.
   • Yes please add a APEX details section for the conditional statements.
2. For SDK details tab, would it be better if we have 2 checkboxes of "this product has public SDK for mgmt" and "this product has public SDK for data plane"
   • 2 checkboxes is fine. If the user already identified data or management is out of scope, then don't show the checkbox on the SDK details step. For example, if data plane has been identified as out of scope, then don't show the data plane SDK checkbox in the SDK details tab.
3. Right now there's also no backend on ado for separate "isPublicSDK" for mgmt and data plane, right now it's just both or neither. We'll need to add backend support for that
   • Once we change that to be separate between mgmt and data plane, where did we reference the old value and what would be affected?
     • True there is only one field in triage "Existing SDKs" with picklist values of "Yes" and "No". We could change the picklist values to be "data", "management", "both", "None", "Not Applicable". Not Applicable is for the "out of scope" scenarios. I also think we should move the "Existing SDKs" field to the epic work item. We should also change the wording - "The product already has public SDKs". For new products, the answer would always be no. What we need to know is "Are there any SDKs currently available for the service that will be updated by this product?". If there are no SDKs, then the product will require a new initial SDK, which would be in scope for TypeSpec. We want to make sure that the "new initial SDK" scenario triggers the user to schedule an Information Meeting before creating a release plans.
   • What happens to the outdated data in the database after we make this change? How do we want to present it?
   • Only triage work items created after Sept 2023 have any values for "Existing SDKs". The triage work items created earlier have no data value. Let's discuss how to best address the data quality issue for all existing products that have onboarded. We might be able to leverage Ronnie's Inventory dashboard and manually update the field values.
4. For review and submit tab, shouldn't mgmt and data plane APEX eligibility be shown separately?
   - Good point. In the APEX details section we can specify the scope data, management or both in the statement.
5. If the product is in GA, is APEX still able to be in scope? I thought APEX is only for non GA cycles?
   - In theory yes but there have been products that had a lifecycle of GA in service tree and none of their lifecycle milestones requirements for APEX launch readiness were signed off. I have seen this scenario for SKUs. That is why I recommended "If product lifecycle is GA, then check isAPEX field value in the epic type = product. If the value is "no", then do not show anything. APEX is complete. If the value is "yes", the display message to user "The product lifecycle is GA and APEX has not been completed. Ask user to confirm." I will manually update the isAPEX field that was added to the epic-product. I am waiting for Yoga to finish resolving the data discrepancies between ADO and S360/CLC that he caused. He was not consistent in updating the ADO lifecycle and launch criteria fields in the epic-product type when he approved APEX requirements in S360/CLC. As soon he is done, then I can use the APEX lifecycle state fields to determine the value of the isAPEX field.

[maririos]

Implementation feedback:

• From engineering team: Feedback onboarding APEX feature #8837
• From PM team: https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/EQOYpXEI_V5Em4S2atNe0JABJn3HmJyJP8_f5kSepo6EgQ?e=Le4mU8

[maririos]

When working on implementation, we missed 2 requirements: .trigger o schedule an Informational meeting and changing the onboading email templates.

@ladonnaq Here are the follow up questions about them:

Azure SDK Info Meeting Required Updates
Review & update rules for determining when an Info Meeting is required - https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/ETD9YCt4cfxHlL4mC1KlVygBIQss1mm6NMIpu3_BdOJhJQ?e=ThDqfB&nav=MTVfezUwQ0FENzNDLTlEN0YtNDkzRi1BNTEzLTY5MjBBRjZFNTg3Rn0

1. In conclusion, there is an Informational meeting pop up when:

• The product doesn't have an existing published SDK, right?
• What happens when people select unsure?

Updated onboarding email templates based on scenario
All new onboarding email templates - https://microsoft-my.sharepoint.com/:f:/p/ladonnaquinn/EnTlkCIg0JdIqFuSLVJUpK4BlPHf9nlYfYli0dAg0B3oKg?e=EDFEvz

2. I summerized the onboarding templates and the scenarios. Could you verify my summary is correct?

• If management and data place in scope values are No: https://microsoft-my.sharepoint.com/:w:/p/ladonnaquinn/ETbBCace8mBMmfp9sA5ACVsBTl3u4a5Orfb--QvRr52HPQ?e=f9581c
• If there are no existing SDKs (initial text block will change): https://microsoft-my.sharepoint.com/:w:/p/ladonnaquinn/ETNzgI3-dq5AmxlZevniuL8B8ZXk2runxj6jtDUid_9cqQ?e=Q8a0v5
• If existing SDKs (initial text block will change): https://microsoft-my.sharepoint.com/:w:/p/ladonnaquinn/ETLLMiSrBzBBu8BGdwMlwhEBbRzoXSAL_1LAGVCD9Vpyrw?e=D9OPen

APEX complete

3. I looked at the excel spreadsheet trying to understand when APEX is complete, but couldn't figure it out. Could you explain again?

[ladonnaq]

When working on implementation, we missed 2 requirements: .trigger o schedule an Informational meeting and changing the onboading email templates.

@ladonnaq Here are the follow up questions about them:

Azure SDK Info Meeting Required Updates
Review & update rules for determining when an Info Meeting is required - https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/ETD9YCt4cfxHlL4mC1KlVygBIQss1mm6NMIpu3_BdOJhJQ?e=ThDqfB&nav=MTVfezUwQ0FENzNDLTlEN0YtNDkzRi1BNTEzLTY5MjBBRjZFNTg3Rn0

1. In conclusion, there is an Informational meeting pop up when:

• The product doesn't have an existing published SDK, right? Correct Info Meeting is required for Greenfield Services (no existing published SDKs).
• What happens when people select unsure? . (La Donna) Good catch, I was focused on all of the in scope and out of scope scenarios, I forgot to identify what to do with the "unsure" scenarios. I have only encountered the "unsure" scenario once this year, the rest of the times that "Unsure" has been seen is really "No" but there was no way for the user to indicate "No". It is rarer scenario now, but still valid, so we need to cover it. I will add details to the spreadsheet but in general if DataScope or MgmtScope = "Unsure", then the Info meeting pop-up should be displayed and Info meeting state = "Required". The email template would be a bit different. I will look at your summarized versions and suggest edits for the "Unsure" scenario.

Updated onboarding email templates based on scenario
All new onboarding email templates - https://microsoft-my.sharepoint.com/:f:/p/ladonnaquinn/EnTlkCIg0JdIqFuSLVJUpK4BlPHf9nlYfYli0dAg0B3oKg?e=EDFEvz

2. I summerized the onboarding templates and the scenarios. Could you verify my summary is correct?

• If management and data place in scope values are No: https://microsoft-my.sharepoint.com/:w:/p/ladonnaquinn/ETbBCace8mBMmfp9sA5ACVsBTl3u4a5Orfb--QvRr52HPQ?e=f9581c
• If there are no existing SDKs (initial text block will change): https://microsoft-my.sharepoint.com/:w:/p/ladonnaquinn/ETNzgI3-dq5AmxlZevniuL8B8ZXk2runxj6jtDUid_9cqQ?e=Q8a0v5
• If existing SDKs (initial text block will change): https://microsoft-my.sharepoint.com/:w:/p/ladonnaquinn/ETLLMiSrBzBBu8BGdwMlwhEBbRzoXSAL_1LAGVCD9Vpyrw?e=D9OPen

APEX complete

3. I looked at the excel spreadsheet trying to understand when APEX is complete, but couldn't figure it out. Could you explain again?

[ckairen]

linking #8841

[ladonnaq]

When working on implementation, we missed 2 requirements: .trigger o schedule an Informational meeting and changing the onboading email templates.
@ladonnaq Here are the follow up questions about them:

Azure SDK Info Meeting Required Updates
Review & update rules for determining when an Info Meeting is required - https://microsoft-my.sharepoint.com/:x:/p/ladonnaquinn/ETD9YCt4cfxHlL4mC1KlVygBIQss1mm6NMIpu3_BdOJhJQ?e=ThDqfB&nav=MTVfezUwQ0FENzNDLTlEN0YtNDkzRi1BNTEzLTY5MjBBRjZFNTg3Rn0

1. In conclusion, there is an Informational meeting pop up when:

• The product doesn't have an existing published SDK, right? Correct Info Meeting is required for Greenfield Services (no existing published SDKs).
• What happens when people select unsure? . (La Donna) Good catch, I was focused on all of the in scope and out of scope scenarios, I forgot to identify what to do with the "unsure" scenarios. I have only encountered the "unsure" scenario once this year, the rest of the times that "Unsure" has been seen is really "No" but there was no way for the user to indicate "No". It is rarer scenario now, but still valid, so we need to cover it. I will add details to the spreadsheet but in general if DataScope or MgmtScope = "Unsure", then the Info meeting pop-up should be displayed and Info meeting state = "Required". The email template would be a bit different. I will look at your summarized versions and suggest edits for the "Unsure" scenario.

Updated onboarding email templates based on scenario
All new onboarding email templates - https://microsoft-my.sharepoint.com/:f:/p/ladonnaquinn/EnTlkCIg0JdIqFuSLVJUpK4BlPHf9nlYfYli0dAg0B3oKg?e=EDFEvz

2. I summerized the onboarding templates and the scenarios. Could you verify my summary is correct?

• If management and data place in scope values are No: https://microsoft-my.sharepoint.com/:w:/p/ladonnaquinn/ETbBCace8mBMmfp9sA5ACVsBTl3u4a5Orfb--QvRr52HPQ?e=f9581c
• If there are no existing SDKs (initial text block will change): https://microsoft-my.sharepoint.com/:w:/p/ladonnaquinn/ETNzgI3-dq5AmxlZevniuL8B8ZXk2runxj6jtDUid_9cqQ?e=Q8a0v5
• If existing SDKs (initial text block will change): https://microsoft-my.sharepoint.com/:w:/p/ladonnaquinn/ETLLMiSrBzBBu8BGdwMlwhEBbRzoXSAL_1LAGVCD9Vpyrw?e=D9OPen

APEX complete

3. I looked at the excel spreadsheet trying to understand when APEX is complete, but couldn't figure it out. Could you explain again?

Here is link to the "Unsure" scenario email template - https://microsoft-my.sharepoint.com/:w:/p/ladonnaquinn/EfFiMuJHXm5Ekw5ZzF4pvsIBac9s9n44YuoklZBczdrGRw?e=bxYyhx