Metrics and Agile Coaching

You need to know how you’re doing, how your team is doing, how your organization is doing and how your coachees are doing. This means having clear ways of measuring progress: metrics.

There are several ways of categorizing metrics which help us understand their uses and their applicability. For each categorization, we will look at “velocity” (the number of story points per sprint) and see how it fits in.

Quantitative vs. Qualitative Metrics

Many people intuitively understand quantitative vs. qualitative metrics but, when pressed, find it difficult to define the difference. At a high level, we think of quantitative as empirical and qualitative as subjective. As a coach, you will need to understand this distinction clearly so that you can help with effective use of metrics and avoid mis-use.

• Quantitative: A quantitative metric has a standardized unit of measure that is well-defined and universally agreed upon. As well, single measurements with this metric can be compared (>,=,<), and have accuracy (usually expressed with significant figures). E.g. 5.10 seconds, 310 kilometers, 895 dollars per day.
• Counting: Counting metrics are a grey area between quantitative and qualitative. Counting metrics only count whole units, but each unit does not need to be identical or standardized. Like quantitative metrics, single measurements can be compared for quantity. Unlike quantitative metrics, counting measurements can, in theory, be done to an arbitrary number of significant figures up to single units. E.g. 26 people, 231 apples, 17 retail outlets, 165,231 pieces of Lego.
• Qualitative: Qualitative metrics lack a standardized unit of measure, but they are still expressed with numbers. This use of numbers is what sometimes confuses people about qualitative metrics. These metrics are comparable only statistically. One cannot compare single measurements. And of course, these metrics are based on subjective perception. E.g. 4-star rating, 9/10 net promotor score.

The particular pitfall with qualitative metrics is when people try to compare single measurements. For example, you might consider facilitating a Sprint Review in which customers give a star rating at the end of the review. It will be tempting to directly compare one customer’s rating with another’s. This should not be done. Instead, you should realize that in order to use star ratings for customer feedback, you need a statistically significant sample size, and you will probably need to compare these ratings over time, rather than in one specific sprint review. In other words, if you don’t have a good understanding of the mathematics behind statistics and the skill to apply that understanding, avoid using qualitative metrics!

Velocity is a qualitative metric. Both story points and sprints are non standardized units of measure.

Leading vs. Trailing Metrics

Leading metrics tell us about the future. Trailing metrics (also called lagging metrics) tell us about the past. But, interestingly, one person’s trailing metric is often another’s leading metric. A simple example of this relationship is between profit and number of employees. As a business person, profit is in the past; it’s a trailing metric. But as an HR person, it correlates well to future changes in staffing levels. High profit leads to more hiring; losses lead to layoffs. Another example is between marketing and sales: the rate of incoming leads is a trailing metric for marketing and a leading metric for sales.

In an Agile environment, the cost of your team, the business benefit realized from your team’s work, and the number of days off from work may be trailing metrics to some stakeholders and leading metrics to others. Consider the metric lead time for completing committed work items. As a customer, this is a leading metric that indicates how fast I’m likely to get my request completed once the team commits to the work. As a team, this is a trailing metric that we want to improve, but we only know we have improved after the fact.

Leading metrics tend to be more practically useful. A metric that can tell us something about the upcoming future gives us a way to control or influence the future; to respond proactively before the outcomes are measured. Trailing metrics only allow us to react, and then, only after the delay of measurement and reporting. You proactively adapt with leading metrics. You reactively adapt with trailing metrics.

Typically, trailing metrics are easier to measure, more limited in their implications, and therefore often easier to talk about. Unfortunately, they are also often easier to “game.” You can improve the metric without improving outcomes, and that means it is less likely to tell us anything about future performance.

Velocity is a trailing metric. Velocity tells us about the past work of a team, but doesn’t have any strong correlation to the future performance of that same team, particularly when looking at important outcomes.

Fitness, Health and Vanity

Fitness metrics indicate “yes” or “no” if your work is satisfactory. Health metrics give you warning signs. Vanity metrics make you feel good (or not).

A fitness metric involves measuring something in relation to a threshold. For example, can I deliver a pizza in 30 minutes or less? The delivery time is a fitness metric and my customer has chosen my services because I promise a quick delivery or it’s free. The customer has decided that 30 minutes is an acceptable time to wait for a pizza. If it arrives earlier, it doesn’t matter much. If it arrives later, it matters a lot.

A health metric needs to be in a certain range of measurement (for health), but doesn’t always fall in that range. Blood pressure is a common example of a health metric. A health metric is a diagnostic metric that can often give you clues as to what is or isn’t going on in a situation. But the range isn’t a strict threshold like with a fitness metric.

In an Agile environment, the amount of work items currently in progress in your team is often a health metric. Too few or too many is usually sub-optimal. Notice that this is a counting metric and a leading metric that predicts lead time.

Velocity is almost always a vanity metric; it makes people feel good to have an increasing velocity, but it doesn’t really mean much for outcomes and results. In some rare cases, velocity might be a fitness metric: if the velocity is insufficiently high, the sponsors of your team might cut funding and re-allocate it elsewhere.

Activity vs. Results

You are not solely responsible for results. As we discussed earlier, you can control some things, influence others, and some things are independent of your actions. This is where the distinction between activity and results becomes useful: you can measure both, but you can usually only hold people accountable for their activity.

• Activity metrics measure the frequency and/or intensity of actions performed by people. People have agency and choice about these activity metrics (at least in the sense of refusing to do an action). It is not alway possible to increase the frequency or intensity of an activity metric, but it is almost always possible to reduce it. E.g. 6 cold calls/hour (frequency), 142 minutes of cycling per week (intensity), 10 landing page A/B tests per month.
• Results metrics measure the outcomes that come from performing an action or activity. People usually aspire to improve results, but do not usually have direct control of results. E.g. 352 webinar registrations, $1.2M gross sales, blood pressure of 110/70, 340 website visitors per day.

As an Agile coach, your understanding of the various types of metrics helps your coachees enormously. For example, goal setting for quantitative, leading and fitness metrics is very different from goal setting for qualitative, trailing and vanity metrics.

And, by the way, just stop using velocity.