What are we measuring and why?

In the past weeks a few separate incidents converged and inspired me to write this blog post. First, as you might know from my last post, I’ve been doing a series of kick-offs with newly formed teams. When we get to the Team Working Agreements part, inevitably the question comes up about whether or not the team should estimate. Second, I also had an interesting exchange with a new colleague about what kinds of KPIs do we think Management (yes the captial-M kind) cares about. And to round it all out there was a refreshing twist on the concept of velocity in a twitter thread by Alan Cooper about just this topic.

For starters, I want to assure you that this won’t be a deep dive into #NoEstimates (even though I am a big fan of it :-)). Rather, I want to dig deeper into the question of team kpis and which ones are worth measuring and why.

What is our size gauge?

When I’m starting out with a new team I like to encourage them to keep things simple with as little overhead as possible. At the last team kick-off I facilitated, one of my colleagues proposed that the team measure velocity in story points so that the team could know how much work they’re capable of. Fair enough. You have to start somewhere. But actually, before even getting to velocity the first step is agreeing on the scale of measure.

As a new team, especially one that is on the larger side of overall size (8 developers) it’s inevitable that there will be a mix of opinions and experience to inform their decisions on how complex or big things are, which then in turn influences how much work they decide to take in.

If deciding whether a story is a 5 or an 8 surfaces those different opinions sooner then I’m all for it! Still, I try to steer them more towards simplicity and suggest t-shirt sizes to start with. In the early stages of team formation it’s less important to find out an exact point size, and more about agreeing on the general shape of things.

Throughput

After the team arrives at a general agreement on how big things are. What next?

They need to have a general sense of how much work they can get done together in a sprint. This is what the throughput measure is.

Over time, as the team gets used to working together you should be able to see an average throughput with a rough factor of variance. Here is where the “keep it simple” approach works well — if you just limit it to COUNT, to number of tickets instead of story points, it’s a LOT easier to understand. (Note: there is a LOT of info about how this happens in #NoEstimates). Of course, you can refine your throughput data by also including the different sizes.

But, even if you don’t break it down by size, by simply measuring the overall throughput sprint after sprint, you will still get to some kind of average. This is especially helpful if you extend your predictability measure to months or quarters rather than weekly sprints.

Predictability (Committed/Delivered)

The next metric to consider, especially when trying to do any kind of planning is predictability. Basically the product owner needs to know, with some sort of agreed variance, how much work the team can sustainably get done in a sprint. The main KPI I’d recommend to use here is Committed/Delivered. This one is important for the team as a trust factor. What I’m looking for here is that the team more or less delivers what they say they will deliver in a sprint. For sure you will see some pretty big variance in the beginning — either in the overcomitting side or the undercommitting side.

What happens if the team starts to game the system? I was asked this the other day from a manager who worried that a team would tend to undercommit ie; “not work hard enough” to ensure there wasn’t any variance between committed/delivered. I turned the question right back and asked what were the consequences when they overcommitted? And, what levels of variance were accepted in terms of commitment? I would venture to guess that if huge importance was placed on 0 variance or the team is somehow blamed for overcommitting without examining (ie retrospecting) on what caused the variance, then you will see teams staying on the conservative side when it comes to planning.

As a coach, you actually want to encourage the team to experiment with this variance. For example, if they are struggling with focus and delivery you might suggest they reign their commitment back in and take in less than they feel comfortable with. In fact one team that I am working with right now is experimenting with a series of 1-day sprints, each containing 1 increment or goal.

This idea of “undercommitting” is frequently more difficult for teams to accept than the overcommitting tendency. The idea that a team can (and should) have one thing to focus on, instead of maybe three competing priorities is easily forgotten once they’re in a sprint planning meeting. I’ve seen time and time again, teams planning stuff in to make sure everyone on the team has “enough stuff to work on”.

It’s not about that though, is it? Somewhere there needs to be a balance. And here is where our friend velocity shows up.

So what about velocity?

This excerpt from a super interesting twitter thread by Alan Cooper proposes a completely different way to view velocity. It’s not about speeding up, it’s about slowing down!

We need to remind ourselves that we are in this constant mode of discovery, of inspect and adapt. If we commit to too many competing priorities at the same time we lose focus, and if we try to speed out way through things we miss important details that end up costing us time in the end anyway.

Cycle Time

An alternative to velocity is cycle time. For each ticket simply count the number of days from when the ticket is first pulled in progress to the day when the team declares it is closed. That’s your cycle time. You can also keep a running average on this and even split by work types. For example, you can have a cycle time for bugs, for new features, for technical items. You can see how they differ from each other and draw some conclusions.

Additionally, once a team has a sense of cycle time they can apply what they know in some story mapping to decide with the product owner how to chunk out releases.

What about lead time?

After a while chances are the team’s backlog grows…and grows….and keeps on growing. Another metric, lead time, might help the team to think about how to improve their flow. Lead time is cycle time + the amount of time the issue was sitting in the backlog waiting to be taken in progress. And all that time it was sitting in the backlog it was getting more stale and probably less urgent, which begs the question, why even keep it there past its sell-by date?

Where does the time go?

Last but not least I sometimes like to measure where the team is putting their efforts. It’s amazing what you can find. This measure is particularly helpful when you have a team or stakeholders how have a nagging sense that things are going slow but can’t pinpoint exactly why. A lo-fi way to do this is to have a simple matrix of the different activities the team wants to measure.

One tick can be more or less equivalent to 1/4 day, and each person gets 4 ticks a day. The point is not to get too granular and count hours but to have an overall picture of what the team is spending their time on.

Final thoughts

What we’re actually striving for when we talk about any of these metrics is that the team and the product owner are talking with each other when the numbers start to look strange. Use the numbers to start conversations about what is actually happening instead of as a means unto themselves.