Measurement and Agile – Oil and Water? (Part 1)

In this four-part blog series, we’ll explore measurement and metrics in the Agile space to help organizations and teams be more successful.

The truth is that measuring things in software development is hard. As Edwards Deming clearly stated “The most important things cannot be measured.” We’ve all seen how metrics can be gamed and abused (velocity, anyone?) and many have heard of the “Law of Unintended Consequences” and experienced in practice how measurement led to unintended and counterproductive behaviors. So a lot of folks have jumped on the #NoEstimates bandwagon and we’re all better off not measuring anything at all, right?

I’m not sure it’s that easy. The reality is that we’re all living in the business world and we’re being asked to help answer certain, very valid questions, such as:

• What’s our customer satisfaction? How is it trending?
• How are our Agile teams doing? Are we getting better?
• How is our Agile / digital transformation progressing?
• Are we getting the value we were hoping for out of our transformation efforts, coaching, etc.?
• What ROI are we getting from the money we invest in our transformation?
• For consultants and coaches: Am I moving the needle and making an impact?
• Are we helping the organization deliver value faster?
• Where do we need to deploy our limited coaching resources?
• What bottlenecks in our value streams do we need to address?

While measurements are hard and there are certainly landmines to avoid, how can we even start to answer these and similar questions without any measurements? There is truth in Peter Drucker’s famous proclamation:” If you can’t measure it, you can’t improve it.”

So let’s stop arguing about whether we should measure and rather focus on how we can do it better, so we can get the value out of it while staying as clear as possible of its pitfalls.

Getting Started

Before we can decide on what to measure, we will have to ask ourselves first what we’re trying to achieve. Without going into too much detail, the GQM approach can be useful, which is to establish a goal first (what are we trying to do, e.g. improve our value delivery), derive appropriate questions (what do we need to answer, e.g. what is the time from ideation to value delivered) and finally, determine related metrics (how do we measure this, e.g. what is our average feature cycle time between creation in our application lifecycle management [ALM] system and code deployment).

The official GQM approach – as opposed to the simplified explanation provided here – can be heavy if followed in detail. For a different, more lightweight approach, please check out Larry Maccherone’s ODIM approach (outcome, decision, insight, metric).

A lot has been written about how to establish and rollout an organizational measurement and metrics framework, so for this post I’m going to focus on just some of the basic steps and recommendations based on my own experience:

1. Pick a few appropriate quantitative measurements

Given the goal(s) to be achieved and the question(s) requiring answers, select a few quantitative metrics that make sense. And yes, just a few. Start easy and don’t try to boil the ocean; you can always add additional metrics later on. Preferably, look for data that’s already (easily) available and being produced by existing processes and systems. Capturing new data via systems or even manually will always make it harder, so go there only if necessary. That said, don’t just use data that’s convenient but otherwise misleading or not meaningful.

Think about at which level this data needs to be captured: the individual, the team, the release train/program, the solution train or value stream, the portfolio, or even at the organizational level? Is rollup required?

If you’re lucky, your ALM (VersionOne, Rally, Jira, etc.) may already capture this data for you automatically. In some cases, minor configuration changes could help ensure this data is collected.

2. Pick a few appropriate qualitative measurements

Yes, in a way “qualitative measurements” are somewhat of an oxymoron. Think about them as qualitative (subjective) data, e.g. derived from surveys, quantified. Example: If you decided to measure team happiness or engagement every sprint (some teams do this with sticky notes during retrospectives), you could capture each person’s “vote” as 1 = unhappy, 2 = doing okay and 3 = happy. Then calculate the average for all team members every sprint and you just turned qualitative data into something quantifiable.

Since qualitative data is less likely to be captured by existing systems, you might have to resort to surveys, Excel spreadsheets, or – of course – look into platforms like AgilityHealth® (more on that later).

Why do we care about qualitative measurements? Because delivery metrics derived from our ALMs don’t tell the whole story. To illustrate: If you go to see your doctor, there’s more to your health than taking blood pressure and temperature, which can both be measured quantitatively. You may have frequent headaches or other areas of discomfort, but without you telling your doctor, he would not be able to diagnose you holistically.

Next Up

Hopefully this introduction to the first two recommended steps got you thinking or even started with measurements. In part two of this post, we’ll take a closer look at balanced views and achieving value from data.  
Photo by Sharon Pittaway on Unsplash
By René Rosendahl, AgilityHealth Product Strategist