Discovery: pull request configuration, yaml, and debt

[codecovdesign]

TLDR? previously, in phase I of this epic we did some housekeeping to understand our issues better, identify yaml cleanups, and ensure that our intended displays of pr comment formats was displaying as intended across the different plans and settings (in case of deprecated yaml inputs). Next we are looking to 1) understand and better identify default yaml, 2) identify out of the box yaml configs, 3) understand and identify yam usage data, 4) ideation of longer terms yaml builder/helper in UI

Problems to solve, discovery phase II

• What is the systems default yaml?

  • Once identified: identify improvement/cleanups/deprecations/etc AND audit existing UI notification showing default yaml (next steps)

• What is the current yaml usage?

  • such as: custom vs yaml, pro vs team vs free
  • what are the most common settings / what are the least common settings
  • flags usage, how many CFF, components
  • what are next steps? (deprecation, docs improvements)
  • explore qualitative feedback, such as: Proper yaml setup to skip files in coverage testing feedback#493 to identify trends and common challenges

• What do our input mean/translate to?

  • Translate existing inputs from docs into more clear outcomes if used
  • audit inputs and rewrite/coordinate to what would happen

• What could longer term yaml building look like?

discovery phase I: housekeeping

## Problem to solve

The pull request comment formatting and options have changed through different iterations over time. Along the way, some yaml settings are now duplicative and/or unknown internally and/or outdated. This discovery is aimed at documenting the issues and customer feedback and doing some housekeeping:

Solution ideation

• Audit the yaml/configuration states
• Revisit the yaml in-app discovery
  • Ideate on follow up iterations
• Identify ux/eng debt to address

WIP: view main figma file

Tasks

Preview Give feedback

1. codecov yaml configurations audit and updates (per PR comment updates) #987
   in discovery +1
   
2. Unable to see overall patch % for PR changes #986
   Dev-Ready Polish +2
   
3. PR comment documentation image updates #985
   in discovery +1
   
4. Default yaml audit issues #1042
   in discovery +1
   
5. yaml config: header (old format), headercondensced, and newheader #1041
   Dev-Ready bug +2
   
6. Remove reach from default yaml #1223
   Dev-Ready +1
7. Discovery: Make indirect changes and project changes check opt-in #1058
   Dev-Ready P0: must do +2
   
8. Discovery: removed code behavior #1020
   3 of 4
   Dev-Ready +1
   
9. Remove base % from report source #1072
   Dev-Ready +1
   
10. PR comment solicitation cleanup #1073
    Dev-Ready Polish +2
    
11. Annotations audit: how/when they show + user sentiment #1074
    in discovery +1
12. Update source blockquote to be after missing files #1222
    Dev-Ready +1
    
13. Add components to default
14. Align files, flags, and components table display in PR comment and in-app #1189
    Dev-Ready +1
    
15. Investigation: commit yaml in UI is not commit yaml that it's processed under #1225
    +0
16. Remove "Codecov Report" header #1141
    in discovery +1
    
17. Only show comment when coverage/bundle is negatively changed (configurable) #1350
    6 of 6
    epic in discovery +2
    
18. Update table showing hits/misses to covered/uncovered #1554
    in discovery +1
19. Missing lines table not showing #1129
    Dev-Ready bug +2
    
20. Remove "impacted files" copy from <details> element #2136
    Dev-Ready +1
21. Default yaml in UI #2232
    in discovery +1
22. Discovery: configuration without a yaml #2234
    in discovery +1
    
23. Update files table header #2237
    Dev-Ready in discovery +2
    
24. Improving Positive Reinforcement for Coverage Increases #2265
    in discovery +1
    
25. Update copy in summary #2266
    in discovery +1
26. Data questions of yaml usage
27. Transformed yaml for codecov #2488
    Low bug +2
28. investigation: application side yaml analysis #2789
    in discovery investigation +2
    
29. Audit: repo level yaml section #3078
    in discovery +1

[codecovdesign]

sync on sept 9th, default yaml kickoff:

What is the systems default yaml? Once identified: identify improvement/cleanups/deprecations/etc AND audit existing UI notification showing default yaml (next steps)

• bug creation for yaml display: Transformed yaml for codecov #2488

• investigation issue for default yaml: Default yaml audit issues #1042
  - identify read yaml
  - identify the k8 yaml (is this the production.yaml?)

• ToDo kyle setup weekly sync to touch base over next four

[codecovdesign]

checkin on sept 17th

• default yaml parts, so far
• will need to con't investigation of read only areas
• null is seen as a placeholders that are worth investigating more
• there could be different default based on where in the code you are, depending on what you are doing
• first step continues to be documenting the yaml file
• kyle ToDo design turn for default yaml

[codecovdesign]

checkin on oct 1

• updated the notion doc for the default yaml with a paste of all the places where read_yaml_field is called in the worker (this might not be totally accurate)
  • consideration around refactoring, since this may not be optimal
• not sure how the yaml is being used in the API
• two things to consider separatly 1) product related issues, 2) eng work
  • with product it's what are the implications of default ux given the yaml
  • with eng work it's about what is the default yaml
• ToDo add commit yaml issues if relevant
• ToDo kyle to create slack channel

[codecovdesign]

working session:

• created agenda doc
• what is the process of the system evaluating the yaml values
  • what is the mechanism to which/what makes up the yaml (sources)
  • of the different yamls, which one is evaluated as the priority