The impact of Agile quantified with Larry Maccherone

LarryThis week’s guest blog is from Larry Maccherone, who is presenting a session on Agile practice at Agile Australia 2014.

Larry’s session, “The impact of Agile quantified: A de-mystery thriller“, introduces the first-ever quantified decision framework for targeting improvement and making Agile practice decisions.

Larry is the Director of Analytics for Rally Software and a frequent presenter at Agile conferences around the world. His research focuses on agile measurement, analysis, and visualisation for software and systems engineering.


The metrics systems we used before Agile had the wrong feedback loop emphasis, so Agilists fittingly threw them out. They replaced them with qualitative insight, which works great on smaller teams.

But Agile is going through another environmental shift. It’s scaling up to larger projects, and being adopted by larger organisations, where it’s infeasible for executives to go by feel alone when making decisions. In these environments, qualitative insight alone is insufficient. It must be complemented with appropriate quantitative insight.

So, the pendulum has swung and the value of appropriate measurement is clear, but we don’t want to make the same mistakes that caused early Agilists to throw out the old measurement regimes. Here are some high-level guidelines on how to successfully introduce measurement into an Agile development environment.

What to measure

The best metrics feedback is balanced. Rather than focusing on just one area of performance, it should come from all four key outcome areas of a project:

  • Do it fast (productivity, responsiveness)
  • Do it right (quality, customer/stakeholder satisfaction)
  • Do it on time (predictability) and
  • Keep doing it (employee satisfaction/engagement)

Application Lifecycle Management tools, such as Rally’s ALM, will help to identify the outcome dimensions of Productivity, Responsiveness, Quality and Predictability. The two remaining remaining dimensions – Customer Satisfaction and Employee Engagement – can be easily acquired using simple surveys.

Keep thinking

As the metrics begin to roll in, remember that numbers are not a replacement for thought. In the Agile environment, quantitative data should complement, rather than replace, qualitative insight. Ideally you want to create a cycle where results of analyses drive questions that lead to more analyses, all in support of making the best decisions.

The process of gathering metrics must be practical. If getting the data is too costly, time-consuming or onerous for your developers to record, you’re asking too much. Developers are a valuable resource. You want to keep them focused on the job at hand. The value of your metrics should always exceed the perceived burden.

To understand what you need to measure, we recommend you work backwards. We believe that better measurement leads to better insight which leads to better decisions and then better outcomes. But you should start with the Outcomes you have in mind, lest you pick convenient measures that don’t lead in the right direction. We call this the ODIM process.

Get the numbers right

Many organisations put a tremendous amount of effort into gathering data without first making sure that the model they are using to analyse that data and turn it into actionable insight is a fair representation of reality as well as appropriate for the context. To avoid such mistakes, invest in the right skills and expertise that will allow you to carry out context-appropriate analyses.

When forecasting, recognise that any mention of a project or activity completion date will be taken as gospel by the business. So be careful about giving hard dates. Remember to consider the risks and delays that could occur. In the Agile environment, projects are rarely fully planned upfront. Instead, they unfold over time thanks to feedback gathered along the way. Because of this, it’s better to give a probability distribution of completion dates that take into account risks that may cause delay.

Essential forecasting considerations are: what can go wrong, what would the impact be, and how long would it take to resolve? Before you ask your developers to prepare detailed work breakdown and estimation, consider these four basic forecasting truths:

  • All forecasts have multiple possible outcomes, some more likely than others.
  • The most pessimistic and optimistic forecasts are least likely.
  • Use probabilistic forecasting techniques like Monte Carlo to identify which results are more likely than others.
  • Risk factors probably pose a bigger chance of impact than the uncertainty of any one story estimate. Rather than asking your team “How long will it take?”, your time would be better spent asking “What can go wrong?” and “How likely is it?”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s