Some thoughts on metrics

I’ve found it helpful to keep a few fundamentals in mind when choosing and using metrics, and I want to share those in this post. Maybe you will find some of this useful.

Purposes and functions of metrics

To help me stay focused on the reasons to measure and the uses of metrics, I remind myself that metrics have two purposes and three functions. The purposes are:

  1. To steer work in progress
  2. To support continuous improvement efforts

Regardless of which purpose we are interested in with any given metric, it will perform one or more of the following functions:

  1. Informational function — provide plain information about progress, financials, risks, or other aspects of the work
  2. Diagnostic function — serve as an indicator when an aspect of the work deviates from expectations and might interfere with delivery
  3. Motivational function — influence people’s behavior, whether intentionally or inadvertently

Three dimensions of management

There are many variations in how software can be built and delivered. Accordingly, there are many choices for metrics that may or may not help us achieve the two purposes of measurement. We determine which metrics to use based on our operational context. One way of understanding our operational context is to use a model comprising three dimensions of management: the approach to the Iron Triangle, the process model, and the delivery mode. Different metrics may apply depending on where our organization stands on a spectrum of practice along each of these dimensions.

I list the three dimensions in this order for a reason. Variations in Iron Triangle management have the most profound effect on how the work flows. Variations in process model have a significant, but less profound effect. Variations in delivery mode can also have an effect. If we focus on, say, process model and we ignore Iron Triangle management, we will miss something.

I feel it necessary to emphasize this because I hear/read comments that indicate people underestimate the impact of Iron Triangle management on work flow. Also, the server statistics for this blog show that the post on process models is among the most-visited pages, while the post on Iron Triangle management receives only a moderate volume of visitors. That may mean that people consider process model to be more significant than Iron Triangle management. In my experience, when managers have had difficulty making sense of the numbers it has almost always boiled down to confusion about how the Iron Triangle is being managed in their environment. All this suggests that people generally underestimate the importance of Iron Triangle management for choosing metrics and for other elements of management. So I want to suggest that you consider this dimension of management first, before worrying about the details of the process model or delivery mode.

As you might imagine, delving into the operational context in detail can be a complicated exercise. It is easy to lose the forest amid the trees. We need to provide all the information necessary to support the business decisions each stakeholder must make. At the same time, we want to keep the quantity of information we report down to a minimum, to avoid causing a low signal-to-noise ratio in reporting. To help me keep my head in the right place, I like to refer to a couple of simple lists of principles.

Glen Alleman’s Five Irreducible Principles of Management

The first is a list of five basic questions that must be answered for any initiative. The list was created by management consultant Glen Alleman. He calls them the Five Irreducible Principles of Management (see the presentation slides, particularly slide 12, but read the whole thing for context). These five questions apply to any initiative regardless of the approach to the Iron Triangle, the process model, or the delivery mode in use. Whether responsibility for delivery lies with a steering committee, a designated manager, or a self-organizing team, the questions must be answered. Otherwise, you will have no idea what is going on. They are:

  1. Where are we going?
  2. How do we get there?
  3. Do we have enough time, resources, and money to get there?
  4. What impediments will we encounter along the way?
  5. How do we know we are making progress?

Metrics alone will not answer all these questions directly. However, it should be obvious that they apply directly to questions 3 and 5, and with a little thought you can probably see that metrics can help us with all five, both for purposes of steering the work and for guiding improvement efforts.

When I am considering whether a particular metric is relevant in a given context, I find it very useful to ask myself whether the metric will help me answer any of these questions. If not, then it’s not the right metric for the situation.

The Albert Einstein School of Management

The second list is a collection of quotes attributed to Albert Einstein. I call the list the Albert Einstein School of Management. The quotes are:

  1. A man should look for what is, and not for what he thinks should be.
  2. Insanity: doing the same thing over and over again and expecting different results.
  3. We cannot solve our problems with the same thinking we used when we created them.
  4. Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.
  5. Any intelligent fool can make things bigger and more complex. It takes a touch of genius – and a lot of courage – to move in the opposite direction.

I’ve read that at least one of these quotes may have originated with Benjamin Franklin. I’m okay with that, because I don’t want to muddy the waters with historical accuracy. I just want to have a catchy list of principles that’s easy to remember for practical reasons.

Measure outcomes, not activity

A simple rule of thumb that I find helps me stay focused on practical metrics is just this: Measure outcomes, not activity. This might sound pretty obvious, but many managers don’t measure outcomes. They depend on observations of activity to tell them whether they are making progress. They might measure, for instance, the number of hours each team member bills to a project per week; or they might track whether a development team is using some particular programming technique. Measurements like these can only be secondary indicators, at best.

Observations of outcomes provide a more direct indication of progress and earlier warning of problems than observations of activity. By tracking, for instance, throughput, cycle time, and other measures of results, we can tell whether we are delivering according to stakeholder expectations. When those measures trend beyond our defined limits of acceptability, they provide early warning that we need to conduct a root cause analysis and take corrective action.

It is conceivable that we could answer the five questions based on measures of activity, but it’s likely the indicators will be a bit fuzzy and red flags will appear too late for us to take corrective action.

4 thoughts on “Some thoughts on metrics

  1. Dave,

    You are right about metrics alone. But those 5 questions are not “metrics” questions, they are tangible evidence questions, with the follow on question of “show me.” Show me you know what done looks like in the form of some kind of tangible evidence. This can be a shopping list handed to a 16 y/o on her first trip to the store, with her freshly minted drivers license. Or the check list to get the Atlas V Heavy off to orbit on time. http://bit.ly/OJPtSp

    Some for the other 4 questions. Metrics “could” be found for the evidence. For example #, “what impediments along with way” starts with the “show me” answer of “I have a risk register, and that risk register is connected in some way to the cost and schedule probability of success.” But there are then specific metrics in Risk Retirement (waterfall flow downs, from RED, toe YELLOW, to GREEN) Reference Class Forecast models with “monetized” impact analysis on next quarters funding profiles. That “might” be a metric once Question #4 has an answer in the “show me” style.

    I’d suggest the following – ask the question, search for the evidence of an answer, then and only then ask “what metric could be used as a measure of quality for the answer?”

    Back to the 16 y/o’s now 22 y/o’s (we have two, 1 year apart). “Hey guys when are you coming home for dinner?” Answer “tonight” Me, “got any time in mind?” Them, “yea dad after work (summer jobs), but then we’re going out.” Me, “where you going?” “Them, “out with friends.” Me, now releasing there will be no unit of measure for the answer (since this week is the last week in their campus apartments before back to school and off to a real job), so my answer is “OK, see you tomorrow.”

    The units of measure of any metric must be meaningful to the decision maker.

    Also if you drop the PMI Iron Triangle (cost, schedule, quality) and replace it with cost, schedule and technical performance http://slidesha.re/tjKkU8 you can get around the limitations of the bad idea. In the end those three measures are immutable – you’re either on schedule or not, on budget or not, and it either works within the specific error bands or it doesn’t. So those are the three classes of metrics. BTW “works” MUST start with a capabilities statement. For example “we need the capability to process provider claims for $0.07 a transaction starting in the spring of 2014 versus our current $0.11 per transaction.” How to do it – technical and operational requirements – good question. But capabilities come first.

    1. Glen,

      Thanks for the comment, and for the excellent explanation as usual. Completely right. The 5 questions are great guidelines for all aspects of project management. My intention was to consider metrics specifically, and not all the other issues in this particular post. Your point about starting from capabilities is especially relevant. That seems to be poorly understood generally in the corporate IT world, for some reason.

  2. Dave,
    I’m on an assignment where agile is being integrated with federal acquisition. The notion of “capabilities based planning” http://bit.ly/N7YFlA and http://bit.ly/vfM74J are the basis of the process. The DoD 5000.02 and the picture you’ve provided other places around the W, speak to the capabilities first.

    One way to introduce the notion is to ask “if you had the system on Monday that is described in the plan and the requirements and the stories, and it was free and it worked as specified, what would you do with it in terms of moving your business (or mission) forward?”

    Sometime there is a big pause. This is where capabilities start. Here’s some examples from actual projects:

    ■ We need the capability to pre‒process insurance claims at $0.07 per transaction rather than the current $0.11 per transaction.

    ■ We need the capability to remove 1½ hours from the retail ordering process once the merger is complete.

    ■ We need the capability to change the Wide Field Camera and the internal nickel hydride batteries, while doing no harm to the telescope.

    ■ We need the capability to fly 4 astronauts to the International Space Station, dock, stay 6 months, and return safely.

    ■ We need the capability to control the Hell Fire Missile with a new touch panel while maintaining existing navigation and guidance capabilities in the helicopter.

    ■ We need the capability to comply with FAR Part 15 using the current ERP system and its supporting work processes.

    Then the first question “what does done look like,” has a place to look for the answer – in the capability. From that point on, it’s all a sleigh ride to the end.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s