In today’s fast-paced digital landscape, software development is about delivering high-quality, secure features that offer the most possible value to customers, and delivering these features as quickly as possible. Software development organizations need to improve efficiency, optimize flow, and adapt quickly to ever-changing market demands. But how can organizations measure these abstract concepts to ensure they are on the right track?
Metrics play a crucial role in guiding development teams, product managers, and business leaders toward informed decision-making. There are various categories of essential software development metrics including business agility, DevOps, application security, and productivity. By understanding these metrics, teams can measure their progress, optimize processes, and enhance their overall performance. I’ll provide a brief overview of these categories in this blog. In subsequent posts in this series, I’ll dive deeper into each metric and highlight successful usage and anti-patterns.
I would be remiss not to provide a note of caution here - while metrics help organizations understand how they are performing and identify areas for improvement, they are only a starting point for understanding the full picture of what is happening. Metrics allow you to identify where you should drill in further. Quantitative measures should be supplemented with qualitative measures, including live conversations and open-ended survey questions.
Metrics also can have undesired effects. I recently revisited a talk given by Mike Carey, who at the time was the Enterprise DevOps lead for Freddie Mac, at the 2020 Business Agility Conference - What You Measure Matters. He clearly articulated how your metrics signal your personal, team, and organizational values and how metrics can drive unintended attitudes, behaviors, and culture within and across teams. You should ensure that metrics emphasize desired business outcomes.
Finally, remember Goodhart’s law - "When a measure becomes a target, it ceases to be a good measure". Use metrics to identify trends and look for areas of success and areas needing improvement. Do not use metrics to indicate that all of development needs to achieve some absolute, rigid goal.
Here are the metrics for each category that tend to be impactful for organizations:
1. Metrics for Business Agility
Business agility refers to an organization’s ability to adapt, innovate, and respond to changes in the market quickly and efficiently. Agility in software development enables teams to pivot based on customer feedback, market trends, or new technologies without sacrificing product quality.
I like to think of two distinct subcategories of business agility metrics:
Flow metrics - these metrics assess how efficiently organizations deliver value to customers; I like Dr. Mik Kersten’s flow framework that utilizes these metrics:
Flow time - time elapsed from start to completion for a given work item
Flow efficiency - the total time spent in value-added work activities as a percentage of total flow time
Flow load - number of work items currently in progress (active or waiting)
Flow velocity - number of completed work items over a time period
Flow distribution - the distribution of work items by type in a system
Since flow metrics are intended to measure the flow of value, they are most useful for measuring the flow of features through the system, where each feature defines an increment of value delivery to the customer that is typically completed within a 1-3 month period. As you will see later in this blog, flow metrics are also useful as team productivity metrics.
Outcome metrics - these metrics address whether the features delivered drive the expected business outcomes; there are two types of metrics that I like in this subcategory:
Key performance indicators (KPIs) articulate specific and measurable business results. They tend to be context-specific based on the business and solutions delivered and may include metrics such as: Net Promoter Score (NPS), customer experience score, revenue, and retention rates.
Objectives and key results (OKR’s) define specific outcomes that the organization is working towards to achieve future success
Additionally, organizations should regularly assess how well they are embracing the various tenets of business agility. Self-assessment tools such as the AgilityHealth Enterprise Business Agility Health Radar and the Scaled Agile Framework (SAFe) Business Agility Assessment can provide useful metrics about business agility competencies.
2. DevOps Metrics
DevOps is all about fostering collaboration between development and operations teams to deliver software faster, with higher quality, and in a more reliable manner. Measuring the performance of DevOps practices through specific metrics helps assess how well teams are meeting these goals and ensures the pipeline is optimized for efficiency, security, and reliability. In this category, the DORA metrics are most commonly used:
Change lead time - this throughput metric measures the time it takes for a code commit to be successfully deployed to production; it measures the efficiency of the development pipeline
Deployment frequency - this throughput metric measures how often application changes are deployed to production
Change fail percentage - this stability metric measures how often a production deployment causes failures in production
Failed deployment recovery time - this stability metric measures how long it takes to rollback or patch a failed production deployment
To the DORA 4, I would add three additional metrics on DevOps practices:
Infrastructure as Code (IaC) Coverage - assesses how much of the infrastructure is managed using Infrastructure as Code (IaC) tools like Terraform, Ansible, or CloudFormation.
Automated Test Success Rate - measures the percentage of automated tests (unit, integration, performance, etc.) that pass in each build or deployment cycle. In DevOps, automating tests is critical for ensuring that code is reliable, bug-free, and ready for production.
Time Spent on Unplanned Work - tracks the percentage of time spent on unplanned or emergency work, such as dealing with incidents, bugs, or hotfixes vs. planned development activities. High amounts of unplanned work indicate underlying issues in the pipeline, such as inadequate testing, deployment bottlenecks, or infrastructure instability.
3. Metrics for Team Productivity
Whereas business agility flow metrics measure the flow of features across the entire system, team productivity metrics provide insight into how efficiently a development team is working to identify potential areas for improvement within each team. It’s important to balance these with quality metrics to avoid creating a culture of overwork at the expense of long-term success. Productivity should be measured holistically to ensure both output and quality are optimized.
Key productivity metrics include:
Cycle Time - measures the duration from when a work item is started to when it is deployed to production; reflects the efficiency of the team to complete work.
Throughput - tracks the number of work items (e.g., user stories, tasks, or bugs) completed in a given period. This differs from velocity in that it focuses on quantity rather than the complexity or size of work items.
Average WIP - measures the average number of work items in progress; useful for determining a team’s workload and potential bottlenecks.
Code Churn - measures how often code changes after it has been written. High churn, especially if it occurs late in the development process, can signal potential problems such as unclear requirements, poor communication, or technical debt.
Automated Test Coverage - tracks the percentage of the codebase that is covered by automated tests. More extensive test coverage generally leads to fewer bugs and faster feedback on code quality, which increases productivity by reducing rework.
Notice that some of the team-level productivity metrics are similar to the value stream-level flow metrics:
Flow time = cycle time
Flow load = average WIP
Flow velocity = throughput
By measuring similar behaviors at the individual work item (task, bug fix, story) and feature level, we gain different insights into bottlenecks within and across development teams.
Team-level productivity metrics provide the current state of productivity and trends over time. But they are only a starting point for understanding what is happening within the team. Teams should conduct regular retrospectives and capture improvement items, prioritize them, and implement the most impactful items. Additionally, qualitative data can be captured via team surveys. Team surveys are available via many tools including Agility Health radars, Swarmia, Jira Align, and Lean Agile Intelligence.
4. Application Security Metrics
As organizations increasingly “shift left” to have software development teams focus on application security and the prevention of cyberattacks, they need to incorporate security metrics into their metrics suite. This will allow you to balance speed and security to optimize for specific development goals, drive collaboration between development and app sec teams, and ensure continuous improvement of application security and development remediation practices.
Key application security practices include:
Mean Time to Detect (MTTD) - measures the average time it takes to identify a vulnerability after it’s been discovered or reported
Mean Time to Remediate (MTTR) - measures the average time it takes to remediate a vulnerability once it has been detected
Effort Toward Remediation - measures the total development effort required to remediate a detected vulnerability
Code Scanning Coverage - measures how much of a code base is covered by static application security tests
In today’s competitive software development environment, measuring the right metrics is essential for maintaining agility, improving productivity, and aligning development efforts with the broader value stream. By tracking metrics for business agility, DevOps, team productivity, and application security, organizations can move beyond simple output measurement, focusing instead on delivering high-quality software that meets customer needs while continuously improving development, operations, and security processes.
Comments