4 Ways to Measure the Success of Your Agile Teams

I’ve been working on Agile and Scrum certifications for several years now. One of my certifications is called Certified Scrum Professional Product Owner or CSP-PO. I took this course through Scrum Alliance with the good folks at Applied Frameworks and learned quite a few techniques I use regularly in my product management work day to day. A blog post of theirs that I revisit regularly and share with all of my clients is called Agile Metrics: 4 Balanced KPIs to Measure Success. I’ve seen what happens when teams don’t follow these guidelines and what happens when they do. Companies can suffer under vanity metrics that become work to track and provide meaningless information. While everything has a meaning, the problem with vanity metrics and tracking a laundry list of organizational stats is the overhead and the lack of actionable insight. Joel Bancroft-Connors at Applied Frameworks breaks down the 4 most important metrics to measure an Agile team’s success and presents a very strong argument for why 4 is enough. I’d like to review that here and expand upon it with my thoughts and experiences, but please do see the original and thank you for the insightful article Joel!

Setting metrics for a team relies on organizational success levers. These can be summarized as the major goals or most important elements of business that an organization cares about in a general sense.

Value: Are we solving the customer’s problem?

Predictability: What’s the accuracy of our planning and delivery?

Productivity: How can we maximize resource allocation and time available?

Quality: How can we ensure our product is bug-free and scalable?

Stability: How can we stay in the race long term and maintain momentum?

Growth: Are we growing and learning as teams and individuals?

With those defined, he poses the question, “How do we measure Agile teams to determine this?” I’d like to add to that informally to ask “How do we measure Agile teams to determine this without wasting everyone’s time and energy on vanity metrics?” I’ve found that many clients have been allured by vanity metrics and find it hard to sway away from them once they start down that path. A PM shouldn’t be shy about educating their team’s and stakeholders on vanity metrics and information radiators. Joel elaborates on this by pointing out humanities limitations with focus and preference for working with 3-5 concepts at a time. This led him to narrow down the six above and come to the conclusions and 4 metrics outlined below. 

1. Cycle Time (Productivity)

2. Escaped Defect Rate (Quality)

3. Planned-to-Done-Ratio (Predictability)

4. Happiness Metric (Stability)

Cycle Time (Productivity)

This is the time it takes to create value as a team. Some may consider cycle time to be a measurement of the time from ideation to delivery which is definitely true in some contexts and needed for larger product management conversations in an organization. The cycle time being referred to here is measuring the specific team, so it’s a measure of time from the moment the team starts developing until reaching the definition of done. Different teams will have a different definition of done, but typically this means developed, reviewed, QA’d and ready for release planning.

Measuring cycle time is often made simple by using tools like Jira that do it automatically and by considering your sprint start dates as the start date for the cycle time as well. Different orgs may need different levels of granularity here, but measuring productivity will show you trends and lagging indicators that can help you troubleshoot and solve organizational or process related problems.

Escaped Defect Rate (Quality)

There are so many ways to analyze a system and process that “improvement” and “clarity” can be lost in the sauce, but the escaped defect rate metric is the beacon in the storm and gauge on quality. A team is only as good as their last release. Getting work done quickly needs to be balanced with getting work done correctly. Escaped defects are issues found after the team has decided they were done with an issue or after dev review, QA and release in some orgs.

Escaped defect rate doesn’t lie. It reveals the dev, product and communication practices in the org and can help highlight areas of future improvement for scrum coaches and leaders alike. Without quality products, we can’t solve the problems of our users in software. This number needs to be watched closely and kept as close to 0 as possible for stability purposes as well.

Planned-to-Done Ratio (Predictability)

No Product Manager can escape the fateful day when an internal stakeholder requests a roadmap. Roadmaps are misunderstood beasts, check out my article covering them here. The biggest mistake that happens with them is that they are set in stone, when they should be agile and variable. To make a point in time roadmap, you need to be confident in the team’s predictability. Without predictability, you’re missing accuracy and timelines/roadmaps and the like are sure to disappoint and set you and others up for awkward conversations down the road.

Planned-to-Done is a great metric as it boils down to what the team plans to do during a sprint increment vs what they actually complete and provides a percentage measuring the overall predictability of your team. This number needs to be as high as possible, but it’s easier said than done as many variables can impact this number. Starting to analyze the trends here will allow you as a PM to work with leaders and stakeholders to make business decisions and better guide the product and organization.

Happiness Metric (Stability)

The last metric and the most important and often overlooked for Agile teams is the happiness metric. This is simply the satisfaction, morale and vibes of the team. This is unique in the list as the only leading indicator in that typically happiness will decrease before any of the other metrics are impacted for an organization. I’ve had a hard time implementing this for clients and to have it taken seriously by the teams at times. It can be a little “Summer Campy” for lack of better words, but the value is apparent and impactful.

The advice given is to incorporate this into Sprint Retrospectives as a weekly survey checking in with the team. Shameless plug here, but I intend to use my SaaS company QuikReview’s services to create surveys to measure this going forward. There are other methods of happiness measurement like the Crisp happiness index that can be used for teams. Remember Project Aristotle by Google, psychological safety and happiness is key to team success and innovation. This one needs to be front and center when measuring teams and scaling businesses.

Conclusion

With all four of these metrics, you now have two highly quantifiable measures for productivity and quality in cycle time and escaped defect rate. You also get the increased accuracy and predictability provided by measuring planned-to-done ratio and, last but not least, the x-factor, the happiness metric, that can make or break a team and provide a heads-up before issues impact an organization or product. While different teams, departments and industries need to track different metrics for different reasons, starting from these four will build a real actionable foundation for your Agile teams. I’ve returned to this time and time again and will continue to expand on its use as I build great products and better teams and encourage you to try it out as well.