From 63a7b11e7e9241763c7f6f5dbc17105f167f2d0c Mon Sep 17 00:00:00 2001 From: Nicholas Shaddox Date: Mon, 18 Sep 2023 13:23:34 -0700 Subject: [PATCH] add: delivery metrics markdown page w/ newer info --- docs/delivery-metrics.md | 727 ++++++++++++++++++++++++++------------- 1 file changed, 487 insertions(+), 240 deletions(-) diff --git a/docs/delivery-metrics.md b/docs/delivery-metrics.md index b052844..23cda6d 100644 --- a/docs/delivery-metrics.md +++ b/docs/delivery-metrics.md @@ -1,256 +1,503 @@ -# Delivery Metrics + + + + + + +#### Our recommendation is to collect the following metrics for team effectiveness & productivity. + +**Delivery Metrics (Product & Squad level)** + +1. Lead Time - Tracks the time it takes for a code change to go from commit (branch or main) to production +2. Deployment Frequency - Measures how often code is deployed +3. Change Failure Rate* - Percentage of changes that result in failures after it is in production or released to end user +4. Minutes of Impact - Calculates the down time from a service disruption over set time + +**Engineering Metrics (Squad level)** + +5. Branch & PR Age - Evaluates development best practices on repos +6. Code Coverage - Provides percent of unit testing covering the lines of code +7. Code Quality - Assesses high or low quality of code as well as security related defects + +**Joy Index (all level)** + +8. Team & Leadership Joy Index - Benchmarks individual developer experience, satisfaction, and happiness + +**Product Metrics (Product level)** + +9. Business OKRs defined & measurable - Verify the squads have established high quality OKRs with clearly defined measures - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricsWhat is MeasuredConsiderations
- Deployment Frequency
- Source(s): CircleCI, DataDog		 - Deployment frequency, to production, is a good indicator for how small of batches teams are working.

- IN PROGRESS - Implemented but the solution is not sustainable. - 		-
    -
  • - For teams who are not fully in control of their production deployments showing deploy frequency to the lower environments is beneficial (Dev, QA, etc.) -
    • -
  • - Teams should set a minimum target a production deployments per day. -
    • -
-
Lead Time for Changes - The amount of time it takes a commit to get into production -
Mean Time to Restore Service (MTTR) - How long it takes an organization to recover from a failure in production -
Change Failure Rate - The percentage of deployments causing a failure in production -
+Theme + Metrics + Focus Relentlessly + Excel Technically + High Performing Team +
Delivery Metrics +

+(Product & Squad level) +

Lead Time + ✔️ + ✔️ + ✔️ +
Deployment Frequency + + ✔️ + ✔️ +
Change Failure Rate* + + ✔️ + ✔️ +
Minutes of Impact + + ✔️ + ✔️ +
Engineering Metrics +

+(Squad level) +

Branch & PR Age + + ✔️ + ✔️ +
Code Coverage + + ✔️ + ✔️ +
Code Quality + + ✔️ + ✔️ +
Joy Index + + + ✔️ +
Product Metrics (Product level) + Business OKRs defined & measurable + ✔️ + + ✔️ +
-## Sensible Defaults -The sensible defaults are the engineering practices & standards for product delivery. The below metrics are signals for product teams to see how they are tracking against the engineering practices. +_* Currently **Change Failure Rate** is not measurable both at Product or Squad level in AA_ + + +# Performance Pulse Check + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Sensible DefaultsWhat is MeasuredConsiderations
- Trunk Based Development
- Source(s): GitHub Insights		 -
    -
  • - DONE Number of active branches - should be far less than the number of engineers on the team. -
    • - Once a change is merged into main delete topic branches locally and on remote. -
    -
    • -
  • - DONE Age of open branches - other than main - branch life should be measured in minutes and hours not days. -
    • -
  • - DONE PR Duration - life cycle should be measured in minutes and hours not days -
    • -
- 		-
    -
  • - How to leverage network insights to show commits to main over time like GitHub Insights does? -
    • -
-
- Small Stories
- Source(s): JIRA - 		- Cycle time in JIRA from In Progress (work started) to Done (work complete) - start out by targeting 1 story per engineer per business day (3 days max) until you find the right pace for your team. -
    -
  • - Track average and standard deviation of cycle time. -
    • -
  • - Log the WIP -
    • -
-
- Team Ownership of Quality
- Source(s): CodeClimate - 		- Unit Testing (TDD) -
    -
  • - DONE Code Climate - Unit Test code coverage - target 80% for net new code -
    • -
  • - Cyclomatic Complexity - I’m not sure this is being calculated by CodeClimate, rather this is more of a feature of SonarQube -
    • -
  • - Technical Debt Ratio -
    • -
- 		-
  • - For existing code bases (mono-repos) where multiple teams are pushing code changes, start by measuring and gating on net new code changes. -
-
- Build in Security
- Source(s): Synk - 		- Snyk - Number of vulnerabilities over time - Critical & High. -
    -
  • MTTR on Critical and High - Identified to remediated
    • -
  • Critical should be resolved in ~ 14 days
    • -
  • High should be resolved in 30 - 45 days.
    • -
  • Date of discovery of CVE not date of identified by Snyk
    • -
  • Fixed vs opened ratios of issues.
    • -
  • Running summary from the last 30 days, fixed vulns, (newly) reported, ignored (what are they choosing not to fix)
    • -
-
- Pair Programming
- Source(s): GitHub - 		-
    -
  • Co-Authored commits
    • -
  • Huddles
    • -
  • Open channel communication
    • -
-
- Automated Deployment
- Source(s): CircleCI - 		-
    - From pushing a code change, an automated build process provides feedback that the build is production-deployable in minutes. -
  • - Pipeline MTTR - when you have a failed build what’s the time to recover -
    • - The main branch is your production candidate branch and should be in good running order to facilitate frequent production deployments. -
    -
    • -
-
- Fast Automated Builds
- Source(s): CircleCI - 		-
    -
  • - From pushing a code change, an automated build process provides feedback that the build is production-deployable in minutes. -
    • -
  • - Pipeline Duration by Stage - Can be pulled from CircleCI api target < 10 from commit to deploy anything more is not fast feedback. -
    • -
-
- Build for Production
- Source(s): DataDog? - 		-
    -
  • - Definition of done for a feature includes at minimum: -
      -
    • - ensuring that the application is observable and supportable in production -
      • -
    • - logs, traces, and metrics are available and useful -
      • -
    • - monitoring and alerts are configured -
      • -
    • - on-call team members are prepared to respond quickly to any failures or outages (e.g. production access, runbooks in place) -
      • -
    -
    • -
-
Metrics + Run + Walk + Crawl +
Lead Time + Less than 1 hour + Between one +

+week and +

+one month +

Between one +

+month and +

+six months +

Deployment Frequency + On Demand/Multiple times per day + Between once per day to once per week + Between 2 weeks and a month +
+ + + +
Change Failure Rate + 0-7% (<15 min) + 7-15% (1+ hours) + 16-30% (2+ hours) +
Minutes of Impact + Less than 15 minutes + More than 1 hour + More than 2 hours +
Branch & PR Age + Less than 1 day + Less than 3 days + 1 week to 1 month +
Code Coverage + 80-90%+ for existing & new code + 20-50% for existing code +

+80%+ for new code +

0-50% for existing code +

+80%+ for new code +

Code Quality +

+(vulnerabilities) +

0 critical +

+0 high +

+0 medium +

+0 low +

0 critical +

+ 0 high +

0 critical +
Joy Index + eNPS 30 to 80+ + eNPS -10 to 30+ + eNPS -30 to 10+ +
Business OKRs defined & measurable + Defined with clear and inspiring objectives that align with the company's vision + Defined but room for improvement regarding alignment with company's vision + Defined but challenging for squads to understand how their work contributes to the company's vision +
-| Sensible Defaults | What is Measured | Considerations | -|:---|:---|:---| -| **Trunk Based Development**
Source(s): GitHub Insights | | | -| **Small Stories**
Source(s): JIRA | | | -| **Team Ownership of Quality**
Source(s): CodeClimate | | | -| **Build in Security**
Source(s): Synk | | | -| **Pair Programming**
Source(s): GitHub | | | -| **Automated Deployment**
Source(s): CircleCI | | | -| **Fast Automated Builds**
Source(s): CircleCI | | | -| **Build for Production**
Source(s): DataDog? | | | + +# + + +# Why Metrics Matter + + +### Quantifiable Performance + + + +* Metrics provide objective measurements of squad performance, fostering a data-driven culture. They enable squads to track progress, set benchmarks, and identify areas for improvement. + + +### Continuous Improvement + + + +* By identifying bottlenecks and inefficiencies, metrics drive continuous improvement efforts. Squads can refine processes, optimize workflows, and enhance collaboration to accelerate delivery. + + +### Risk Mitigation + + + +* Lower change failure rates and shorter Minutes of Impact indicate increased software stability and reliability. This minimizes business risks associated with service disruptions and costly downtimes. + + +### Informed Decision-Making + + + +* Metrics facilitate informed decision-making by offering a comprehensive view of the software delivery pipeline. Squads can make strategic choices to align with business goals. + + +### Cross-Functional Collaboration + + + +* These metrics encourage collaboration between development, operations, and quality assurance teams. Silos are broken down, fostering a holistic approach to software delivery. + +Software powers innovation and competitive advantage and metrics have emerged as the compass guiding software delivery squads toward success. They provide a roadmap to accelerated delivery, enhanced quality, and collaborative excellence. By leveraging the insights gleaned from these metrics, organizations can transform their software delivery processes and stay at the forefront of their industries. + + +# + + +# Delivery Metrics (Product & Squad level) + + +## Lead Time + +Measures the time it takes for a code change to go from commit (branch or main) to production. Provides insights into workflow efficiency and bottlenecks. Shorter lead times indicate smoother processes and quicker value delivery. + + + +* _How to measure:_ Conduct squad level VSM to gather time code change goes from commit to production +* _Example: _Squad’s lead time is 509.15h (~64 days). Working time is 163.85h (~20 days) + + +## Deploy Frequency + +Measures how often code is deployed. Enables rapid iteration and faster time-to-market. Encourages small, incremental changes, reducing the risk of failures. + + + +* _How to measure:_ Divide the total number of deployments made in a given time period (e.g., a month) by the total number of days in that period +* _Example:_ If a squad deployed code 10 times in a month with 31 days, the deployment frequency would be 10/31 = 0.32 deployments per day + + +## Change Failure Rate + +Gauges the percentage of deployments that result in failures. Offers visibility into code quality and stability. Low failure rates signify robust testing and higher software reliability. + + + +* _How to measure:_ Percentage of changes that were made to code that then resulted in incident/rollback/or any type of prod failure. Calculated by counting the number of deployment failures and then dividing by number of total deployments. +* _Example:_ If your squad deployed five times this week and one of them resulted in a failure, your change failure rate is 20% + + +## Minutes of Impact (MoI) + +Calculates the down time from a service disruption over set time. Highlights the squad’s ability to detect and resolve issues swiftly. Shorter MoI reflects strong incident response and system resilience. + + + +* _How to measure:_ Measures time it takes for service to recover from failure. Calculated by tracking the average time between a bug report and the moment a bug fix is deployed. +* _Example: _If your squad had four incidents in a 40-hour workweek and spent one total hour on them (from alert to fix), your MoI for that week would be 15 minutes + + +# Engineering Metrics (Squad level) + + +## Branch & PR Age + +Evaluates development best practices on repos. + + + +* _How to measure: _Calculated by the overall average age of branches & pull requests across each repo +* _Example: _Number of commits a branch is behind or ahead of main. Hours or days a PR has existed before merging into main. + + +## Code Coverage + +Provides percent of unit testing covering the lines of code. Assesses the test performance. + + + +* _How to measure: _3rd party tooling which runs in your automated CI/CD builds +* _Example: _If the code were testing has a 100 lines of code and the number of lines of code that is actually validated in the same software is 50, then the code coverage percentage of this software will be 50% + + +## Code Quality + +Assesses high or low quality of code as well as security related defects. Identifies potential risks of compromising code and indicated where to fix software defects + + + +* _How to measure: _3rd party tooling which runs in your automated CI/CD builds +* _Example: _1 aspect of code quality is reusability. This is best measured by counting the number of interdependencies. The more there are, generally the less reusable the code. + + +## Joy Index + +Benchmarks individual developer experience, satisfaction, and happiness. + + + +* _How to measure:_ Employee net promoter score tracks employee engagement and measures the impact of their people and culture initiatives. The eNPS shows how workers feel about your organization. +* _Example: _Question -_ _“How likely are you to recommend us as a place to work for your family, friends, co-workers?” + + +# + + +# Product Metrics (Product level) + + +## Business OKRs defined & measurable + +OKRs are defined, with clear and inspiring Objectives that align with the company's overall mission and vision. Key Results are specific, measurable, and quantifiable, providing a clear path towards achieving the Objectives. OKRs are regularly reviewed and updated as needed, with a strong commitment to achieving them. + + + +* _How to measure:_ All team members understand the OKRs and how their work contributes to their achievement. The OKRs are logged in the company's OKR tracker. +* _Example:_ Baggage Services Product; Reduce the percentage in manual edits to optimizer suggested solutions from 86% to 43% by Q3, 23 inorder to increase reliance on the TLink optimizer + +