1. Iron Triangle management
2. Process models
3. Delivery mode (this post)

Despite the many complexities of software work, we are always working in one of two modes:

• Discrete project
• Continuous support

By discrete project mode I mean a mode of operation in which an organization creates a special initiative that exists for a defined period of time whenever it needs to achieve a set of related objectives, such as creating a new software application or upgrading the routers on the corporate network.

In contrast, with the continuous support mode of operation, the organization maintains a stable team to support each technical asset (such as the corporate network, a business rules engine, a COTS CRM package, or an ETL product) or suite of applications (such as the suite of applications that support consumer lending in a financial institution) throughout its lifetime. The size of the team may grow or shrink depending on the level of work needed to support the asset or applications at any given time, but the organization does not start a new project for every set of objectives pertaining to the asset or applications.

An advantage of the discrete project mode is that resources and people can be concentrated to achieve objectives of significant scope at the times the extra effort is needed, leaving those resources and people free to do other work at times when the particular assets or applications do not require intense attention. Disadvantages include the cost and delay of project initiation and close-out, the logistics of managing resource demand, and the lower effectiveness of temporary teams as opposed to stable teams.

Advantages of the continuous support mode include the development of dedicated teams that have a deep understanding of the assets and applications they support, avoidance of project initiation and close-out overhead. Disadvantages include team boredom with supporting the same assets or applications for an extended time, and the logistics of managing resource demand as no asset or application requires the same level of support at all times.

Typically, a corporate IT department carries out several different types of work. The department is responsible for maintaining the technical infrastructure, for production operations, for supporting shared technical assets, for production operations, technical support, integration of COTS packages, management of interfaces with external business partners, information security, and application development. Some of these activities are naturally suited to the discrete project mode and others to the continuous support mode.

A company that provides software-based services or that sells a software product also has its own internal IT department that supports the same kinds of activities as any internal IT department. In addition, the company’s primary business operation involves software development, delivery, and support. Many software companies find the continuous support mode more suitable than the discrete project mode for managing the ongoing development of their products and services.

A relatively recent idea is the notion of continuous beta. The continuous beta model recognizes that any application is a living artifact that is never “finished” or “complete.” As long as the application remains in use, it is subject to modification. Only when the application is retired from production or withdrawn from the market can we say with certainty that no further modifications will be needed.
The continuous support mode of operation is a natural fit for continuous beta products. To date, these have mainly included Web-based social networking applications and e-mail services.

Many other types of applications, both commercial products and internal corporate applications, could be built and supported in continuous support mode. The key decision points are the frequency with which updates are made and the time-to-market that can be supported, given the realities of putting a release together. If an application remains fairly static once it is in production, then the value of maintaining a dedicated team for the application is questionable. If an application has to be updated frequently to keep pace with changing business needs or customer requests, then the continuous support mode may be more cost-effective than the discrete project mode.

There are implications for management. With discrete projects, most of the activities described in the PMBOK are necessary. With continuous support, the activities associated with starting up and shutting down a project are not needed. If continuous support is used in a situation that does not call for a fairly steady level of modification to the application, then there will be (possibly excessive) overhead for resource allocation activities, as well. Continuous support has to be applied where it makes sense; if resource demand is variable due to the nature of the asset being supported, it will remain variable even if we shift from discrete projects to continuous support.

There are implications for metrics, as some of the conventional metrics depend on the existence of a discrete project that has a defined (or expected) scope and an ending date. Any metrics of the general type, “percentage complete,” depend on having a stable definition of 100%. With the continuous support mode of delivery, there is no predefined scope to be completed. Requests are added to the team’s work queue at any time. In the context of continuous support, this does not represent “scope creep;” it is simply normal.

Furthermore, budget allocations are not lump-sum amounts intended to pay for a discrete project or a single release; they are recurring allocations that cover fixed costs over time. For these reasons, there is no meaningful way to track percentage complete of scope, of schedule, or of budget.

With a continuous beta product, the definition of “complete” is that the product is no longer used. That is certainly not the intent of tracking “percentage complete” on a development project.

Certain metrics associated with adaptive development are also sensitive to the delivery mode. A popular metric designed for the time-box process model, velocity, requires a target level of scope to be defined. Although the scope is subject to change, at any point in time there is a planned scope. Observations of velocity are used to create a burn chart that shows progress toward the project’s goals. A burn chart either shows the amount of work completed (burn up) or the amount of work remaining (burn down).

In a continuous support operation, the “scope” constantly changes as new work is added to the work queue at arbitrary times. If the team has the appropriate level of capacity to deal with the average rate that new work arrives, a burn chart tends to look like a jittery line moving more-or-less horizontally across the chart. Therefore, a burn chart cannot tell us anything about “progress” with continuous support mode, although it might be useful as an indicator that the team has too much or too little capacity.

Summary

I think that many of the decisions we might make when managing software development and support activities should be informed by (a) the approach to the Iron Triangle, (b) the general sort of process model in use, and (c) whether we are operating in a discrete project mode or a continuous support mode. When people assume or believe they are doing things in one way when in fact they are doing things in a different way, they may make choices that lead to confusion about progress, status, cost, or delivery effectiveness.

Since people have a tendency to apply the wrong labels to things, it is important for us to be able to recognize by direct observation which approach to the Iron Triangle is actually in use, which process models are actually in use, and which delivery mode is actually in use. If we just go by the labels, we may choose metrics that do not give us meaningful information to support management decisions.

Because of the numerous variations and branded alternatives, and the emotional attachments people have with their favorite frameworks and methodologies, the question of process model is often the most challenging one to answer accurately. Yet, because of the significant impact Iron Triangle management has on everything else, it is not sufficient to look solely at the process model to determine how best to manage and measure the development and delivery activities.

Whether you see any merit in this model is up to you to decide. In future posts, I will probably refer back to these ideas about Iron Triangle, process model, and delivery mode, so I wanted to have this material available so that I won’t have to repeat it again and again when discussing related topics such as metrics.

1. Iron Triangle management
2. Process model (this post)
3. Delivery mode

In my experience, no two organizations, and no two teams within the same organization, use exactly the same process model for software delivery; nor do they keep the same process intact forever. Everyone tailors their process to their own needs and to the realities of their own situation. In addition, most organizations use more than one process, depending on the nature of each particular initiative or work stream.

That is as it should be. Yet, despite the many variations, there are common patterns that can help us understand how we might plan and track the progress of software development initiatives and support activities in ways that help us make management decisions, as opposed to merely following a published guideline for planning and tracking.

My observation is that there are four basic models for software delivery. We can think of these as reference models, in that people don’t actually use any one of them in pure form. Although reference models are not used directly, their definitions provide usefully simple expressions of key concepts. They are:

• Linear
• Iterative
• Time-boxed
• Continuous flow

The tailored processes I mentioned, as well as any number of branded and/or commercial methodologies, borrow elements from two or more of these reference models. Furthermore, real-life processes emphasize different aspects of the work depending on the particular problems of interest in a given context, such as risk mitigation, alignment with stakeholder needs, software quality, or time to market.

Even considering all that, it seems to me that each real-world process leans more heavily toward one of the four reference models than the other three. By understanding that, we can choose metrics and other management tools appropriate to the situation at hand, regardless of buzzwords, labeling, or “belief” (or wishful thinking) about how the work is really being done. As I mentioned in part one of this series, people will say whatever they think they need to say about the process they are using, as buzzwords wax and wane in popularity. It’s more interesting to observe what people do.

Linear model

The linear model of software development is based on the premise that software is best developed (or can only be developed) by working through a series of discrete steps in a specific sequence. The steps are usually defined as the following, or as a variation on the following:

1. System requirements
2. Software requirements
3. Analysis
4. Program design
5. Coding
6. Testing
7. Operations

The list above is adapted from Winston Royce, “Managing the Development of Large Software Systems”, in Proceedings of IEEE WESCON 26 (August 1970): 1–9.

The linear model is sometimes called the System Development Life Cycle (SDLC), or other words that can “spell” SDLC. Some people refer to it disparagingly as the “waterfall” model. Whatever you choose to call it, the fundamental assumption is that software can and should be developed by taking the work serially through a sequence of steps such as those listed. Ideally, a single pass through the steps will yield a good outcome. When it doesn’t, the assumption is that someone made mistakes in doing their portion of the work during one of the earlier steps.

It’s a commonplace to hear that “nobody” uses the linear model in real life anymore. In my experience, that is not the case. Many organizations do base their formal delivery process on the fundamental assumptions of the linear model. While they have change control processes and defect repair processes in place, those are considered to be corrective actions that “shouldn’t” be necessary, if only everyone would just do their jobs correctly the first time.

The assumption that linear development is a valid approach is very, very prevalent. For that reason, nearly all real-world process models have this assumption built in at a deep level, even if they are formally structured around one or more of the other three reference models. Therefore, we cannot ignore the linear reference model in our assessment of real-world models, or we risk overlooking something of importance.

The other process models can work with either approach to Iron Triangle management. However, a strictly linear model precludes an adaptive approach. Even when some measure of flexibility is introduced by defining change control procedures, when the mentality is based on the linear model then the change control procedures are usually so cumbersome that an adaptive approach is impractical. A common symptom is that lead times tend to be measured in months or years even for relatively trivial initiatives.

Iterative model

An iterative model assumes that a single pass through the canonical development steps will not result in a good outcome because it is infeasible to know everything about the solution in advance, and because unexpected events and new learning will inevitably occur despite the most careful up-front planning.

I don’t want to give the impression that these models evolved serially, but in some cases the evolution of a model has been informed by the problems people experienced with earlier models. When the industry moved toward iterative models, the move was prompted in part by particular problems people had experienced in applying the linear model. Specifically, the key problems were very long lead times and poor alignment with stakeholder needs.

By revisiting the requirements repeatedly, or iterating over the requirements, people sought to bring products to market in less time and align the functionality more closely with stakeholder needs. The key philosophical difference between the iterative and linear models is that the iterative model does not assume the reason changes occur has to be that someone made a mistake in an earlier phase. Learning and change are considered normal.

There are more variations and more examples of iterative methods than any other kind. Some iterative processes call for passing through all requirements in each pass and adding depth or refinement to the functionality in each pass. Others call for delivering high-value features early and in a complete form, and then delivering additional features in subsequent iterations. Still others call for developing a subset of functionality such as a prototype and then refining the design based on feedback from stakeholders. The simplest form of an iterative model simply repeats the linear model multiple times, biting off a subset of the scope each time.

Different methodologies based on the iterative model have different priorities. Some are intended to improve time to market or business alignment, some are intended to manage risk effectively, and some are intended to assure high quality in terms of some key criterion such as defect density or usability. So, methodologies may differ in their details, but any iterative process has the characteristic that a single pass through the requirements is not expected to result in the final version of the product.

Time-box model

A time-box is a predefined time period within which a given set of objectives is to be met. In the event the objectives are not met, we do not extend the time-box to allow for completion. Instead, we try to adjust our practices such that we can complete the next set of objectives within the limits of the next time-box.

At the risk of suggesting process model evolution has been serial, I will observe that the concept of the time-box was a response to problems people experienced in applying the iterative model. While iterations did help solve the problem of poor alignment with stakeholder needs, many software delivery projects had difficulty staying on schedule.

The basic iterative model does not require anything to be delivered to production or to the market in each iteration, although some of the specific methodologies based on the model do so. There was still a tendency, as with the linear model, to extend timelines whenever there were challenges in delivering the expected functionality within the expected time. Of the three dimensions of the Iron Triangle, schedule seemed to be the most susceptible to unplanned change.

One answer to this problem was to define fixed time intervals within which working software had to be produced. The idea was to force teams to find ways to get a set amount of work done by a set time, again and again. The main purpose was to avoid the problem of schedule slippage.

Methodologies based on the time-box model usually resemble Matryushka dolls, with time-boxed activities layered one inside another. A fixed release time-box contains two or more fixed iteration time-boxes. Each iteration contains two or more working days, which are treated as time-boxes for certain purposes. Each regular working day within an iteration may contain still smaller time-boxes to drive fine-grained technical activities, similar to (or based directly on) the Pomodoro Technique for personal time management.

An iteration may also contain time-boxed activities for touching base about progress and impediments, short-term planning, stakeholder demonstrations, and team-level process improvement exercises. Different methodologies based on this model prescribe different time-boxes and different specific activities, but the general idea is that the time-boxes limit the time available to complete specific packages of work with the overall goal of delivering a production-ready increment of the solution in each iteration.

A side effect of the time-box approach has been that scope became the dimension of the Iron Triangle that was most susceptible to unplanned change. The old question, “When will all the features be delivered?” was often replaced by a new question, “How much functionality will be delivered by the release date?”

Continuous flow model

The continuous flow model borrows, adapts, and combines concepts from a number of sources, including Theory of Constraints, the Toyota Production System, Lean Thinking, pull systems, Complex Adaptive Systems, and Systems Thinking. The basic premise is that software development is a value stream whose purpose is to deliver something of value (a capability supported by a software product) to a customer. The customer defines the meaning of value.

Let me reiterate that these models did not evolve serially, but it may be a useful oversimplification to say that people turned to the continuous flow model because of problems they experienced in applying the time-box model. Any process we might use will incur a certain amount of overhead just to satisfy the formalities of the process itself. That is not a problem, provided we are getting value in return for the overhead. As teams and organizations become increasingly proficient with time-box methods, the overhead of planning and managing the various time-boxes yields proportionally less value.

Some teams and organizations reached a point that they were spending proportionally too much time on process-related overhead activities. Rather than shortening time to market and improving alignment through frequent stakeholder feedback, the time-box model was hindering delivery by imposing too much ceremony on day-to-day work, and interfering with stakeholders’ own work flow by asking them for too-frequent participation in formal feedback activities involving very fine-grained solution increments.

One answer to this problem has been to dispense with the fixed time-boxes and instead focus on keeping a small number of discrete work items flowing smoothly and steadily through the process from start to finish. Methodologies based on this model draw heavily on Theory of Constraints and Systems Thinking. The key concepts are:

1. Focus on customer-defined value
2. Keep the work moving steadily
3. Eliminate waste from the process

Representative examples

Quite a few branded software development methodologies have been created over the years. Most of them are hybrids that combine ideas from two or more of the four basic reference models, and most were developed in the trenches in response to the needs of a specific software development project before being promoted as general solutions.

Most software development organizations officially use one or more branded or home-grown methodologies that exhibit a mixture of characteristics from two or more of the reference models. Within those organizations, individual teams or working groups adapt the methodologies to their own needs. There are many, many variations.

To help us understand management practices and metrics that make sense in a given context, it is helpful to break down the methodologies we see around us in terms of the inherent characteristics of the four reference models. Here are a few examples of commercial or branded methodologies to give you an idea of what I mean.

Please bear in mind these are not meant to be complete descriptions of the various processes or methodologies; they are only illustrations of the reference models. If you want detailed information about any of these processes, please refer to their respective official websites or other information sources.

In addition, please bear in mind the list is not intended to include every process framework or methodology that was ever conceived. It is only a list of examples to illustrate the reference models. There are other noteworthy methods, such as Thoughtworks’ method (which has no formal name, as far as I know), Industrial XP (created by Joshua Kerievsky and promoted by Industrial Logic), and the Crystal Methodologies (created by Alistair Cockburn). Most of them represent some variant of the Iterative or Time-box reference model, and we already have several examples in that category.

Finally, please bear in mind that the descriptions are not intended as criticisms or comparisons. They are only meant to point out how each process maps to one or more of the reference models. There are no value judgments here with regard to whether one process is “better” than another. The point is to be able to recognize the salient characteristics of the process actually in use in any given organization so that we can make appropriate context-aware choices of metrics and other management tools and techniques.

Anderson Foundation Method/1: (Linear model)

In the mid-1980s, I worked on a project that used a methodology called Anderson Foundation Method/1, a product of Anderson Consulting. Anderson has since been incorporated into Accenture, and that company still markets a product line with names like Foundation something/1, although these are considerably evolved from their 1980s-era ancestors.

The version I used long, long ago was based on the strict linear process model. It was very specific to the Cobol programming language and a hierarchical database management system from IBM called IMS/DB. Most of its documentation artifacts spelled out very detailed specifications for IMS-specific implementation details. It was typical of the formal software development methodologies of the era in that it was firmly grounded in the linear model and was closely tied to specific technologies.

V-Model: (Linear model)

The V-Model was first elaborated at Hughes Aircraft in the early 1980s in connection with a proposal for a major FAA project (see Wikipedia article for more history). The V-Model emphasizes the importance of functional and non-functional verification at all levels of system design and creation, including projects that involve both hardware and software development.

Conceptually, the V-Model looks a bit like this:

As requirements are elaborated at each successive layer, the corresponding verifications are also defined. One way to think of the V-Model is as an early incarnation of the concept of specification by example or acceptance-test driven development, except that in those days the tooling did not exist for executable specifications. Once the details have been elaborated, both on the functional side of the V and on the verification side, then development proceeds, with the verifications carried out at each level of detail all along the way.

Spiral Model: (Iterative model)

Barry Boehm invented the Spiral Model in the mid-1980s and published it in “A Spiral Model of Software Development and Enhancement”, ACM SIGSOFT Software Engineering Notes, ACM, 11(4):14-24, August 1986.

The basic idea is to build the product through a series of iterations through design and prototyping activities, gradually adding functionality and design refinements with feedback from stakeholders.

A diagram of the Spiral Model is available at http://qualityguru.files.wordpress.com/2010/08/sm2.png

You should not assume that iterative models necessarily imply rapid development. As originally envisioned, the Spiral Model called for iterations of 6 months to 2 years in length. It was, in part, a response to the fact that large-scale software development projects were so long that they tended to drift away from customer needs by the time they were finally delivered. By demonstrating gradually-refined prototypes at regular intervals, the goal was to keep development aligned with needs over the course of very long release cycles.

Projects using the Spiral Model need not define iterations of such long duration. The model itself does not depend on any particular iteration length.

The Spiral Model addresses problems caused by very long release cycles, but does not explicitly break with traditional notions of SDLC-style development steps within each iteration.

You may be accustomed to thinking of “iterative” as implying “time-boxed,” but this is not so. As with most iterative methods, Spiral does not call for a production-ready solution increment to be delivered in each iteration. It only calls for a prototype. Prototypes are, by definition, meant to serve as examples and to be discarded. They are not meant to become the basis for the “real” code.

Rational Unified Process: (Iterative model)

The Rational Unified Process (RUP) was created by Philippe Kruchten at Rational Software in 1996, based in part on the Objectory process the company had acquired from Ivar Jacobson. IBM acquired Rational Software in 2003, and still sells RUP. It is one of the most widely-used and best-known software development process frameworks.

This image comes from Wikipedia:

RUP is not a prescriptive methodology, but rather a customizable process framework specifically designed to support software development activities. Depending on the characteristics of a particular project, the team may elect to use all or only a subset of RUP practices and artifacts.

The main focus of RUP is on risk management, rather than on technical practices or product lead time.

RUP is firmly rooted in the SDLC mentality in that it defines four discrete development phases: Inception, Elaboration, Construction, and Transition. The four phases map naturally to typical SDLC process steps.

The RUP model cross-references specialized activities, which it calls disciplines, with the four phases, including Business Modeling, Requirements, Analysis & Design, Implementation, Test, and Deployment. Thus, RUP maintains traditional thinking regarding the need for expert specialists in various aspects of software development.

A unique feature of the RUP model is that it suggests the relative workload within each discipline at each phase of development. This is shown in the diagram by the thickness of the colored regions that represent the contribution of each discipline as the team progresses through the four phases.

RUP as a commercial product includes software tools to maintain a common repository of project information so that the various artifacts will all be based on the same version of the same information. This is a way of mitigating the risk of miscommunication between specialists.

The four phases may be carried out within each iteration, or there may be multiple iterations for each phase. This is up to the project team or project management. As with most iterative processes, RUP does not require the delivery of production-ready solution increments in each iteration, or even with each pass through the four phases.

Evo: (Iterative model)

Evolutionary Project Management, or Evo, was developed by Tom Gilb. It is a holistic, engineering-focused approach that aims to deliver the maximum value to all stakeholders of a project. It is based on ten basic principles (see http://nrm.home.xs4all.nl/EvoPrinc/index.htm):

• Real results, of value to real stakeholders, will be delivered early and frequently.
• The next Evo delivery step must be the one that delivers the most stakeholder value possible at that time.
• Evo steps deliver the specified requirements, evolutionarily.
• We cannot know all the right requirements in advance, but we can discover them more quickly by attempts to deliver real value to real stakeholders.
• Evo is holistic systems engineering – all necessary aspects of the system must be complete and correct – and delivered to a real stakeholder environment – it is not only about programming – it is about customer satisfaction.
• Evo projects will require an open architecture – because we are going to change project ideas as often as we need to, in order to really deliver value to our stakeholders.
• The Evo project team will focus their energy, as a team, towards success in the current Evo step. They will succeed or fail in the current step, together. They will not waste energy on downstream steps until they have mastered current steps successfully.
• Evo is about learning from hard experience, as fast as we can – what really works, and what really delivers value. Evo is a discipline to make us confront our problems early – but which allows us to progress quickly when we really provably have got it right.
• Evo leads to early, and on-time, product delivery – both because of selected early priority delivery, and because we learn to get things right early.
• Evo should allow us to prove out new work processes, and get rid of bad ones early.

A diagram of Evo is available at http://www.gilb.com/tiki-page.php?pageName=Evolutionary-Project-Management

Evo emphasizes building truly complete solution increments in each iteration, with well-designed architecture and full support for non-functional requirements, and providing value to all stakeholders. It also takes a more expansive view of the meaning of “stakeholder” than most other processes.

Scrum: (Time-box model)

In 1986, Hirotaka Takeuchi and Ikujiro Nonaka described a holistic or rugby approach to product development, in which a cross-functional team created a new product collaboratively, passing the ball to one another as their various skills were needed. The idea gained wider recognition with the publication of the 1991 book, Wicked Problems, Righteous Solutions, by Peter DeGrace and Leslie Stahl. Shortly after that, Jeff Sutherland, John Scumniotales and Jeff McKenna at Easel Corporation, and Ken Schwaber at Advanced Development Methods crafted a process framework based on the rugby approach and named it Scrum. Since the early 2000s, Scrum has become one of the most widely-used process frameworks for software development, superceding RUP in general popularity.

A diagram of Scrum is available at http://www.mountaingoatsoftware.com/scrum/overview

Scrum calls for a cross-disciplinary team comprising whatever skills and knowledge are necessary to complete the objectives of the project. It defines specific roles that have specific responsibilities with respect to the process as such. Scrum requires that a production-ready solution increment be delivered in each iteration. However, it does not prescribe any particular practices or methods for the team to use. Ken Schwaber has described Scrum as “a lightweight management wrapper for empirical process control.”

Extreme Programming: (Time-box model)

Extreme Programming (XP) was developed in the trenches of a corporate development project, and was not created as an academic exercise or theoretical model. As such, it is a very no-nonsense approach to software development. For the same reason, it is strongly focused on a single team carrying out a single project.

In 1996, Kent Beck became the leader of a project at Chrysler to develop a payroll system. The project was known as C3, short for Chrysler Comprehensive Compensation System. XP did not create anything completely new. The basic idea was to put together proven-effective software development practices from the past and “turn all the knobs up to ten.”

XP combines a timeboxed iterative process model with a set of software engineering practices. It is unique in that it specifies particular development practices and not just a management framework.

A diagram of XP is available at http://osl2.uca.es/wikiCE/index.php/Extreme_programming

The Matryoshka doll metaphor applies very nicely to XP. The cited diagram shows the various feedback loops built into the XP process model. From seconds to minutes to hours there are no explicit time-boxes. From one day to days to weeks to months, each loop adheres to a predefined time-box. In practice, many teams define explicit time-boxes for the smaller loops, as well, as a way to manage their work flow.

Like Scrum, XP calls for a production-ready solution increment to be delivered with each iteration.

Kanban Software Development: (Continuous flow model)

The term kanban has become popular in software development circles in the past few years. Unfortunately, it has two different meanings. In a broad sense, kanban is a change management method for effecting process improvement and organizational transformation. In a narrower sense, a software development process based on continuous flow is also known as kanban.

The basic idea of kanban as a software development process is to assure smooth work flow by controlling the level of work in process (WIP). The main mechanism to provide high throughput is to adjust the WIP limits in specific steps of the process, guided by principles of the Theory of Constraints and Lean Thinking.

The kanban software development process came along at a time when the popularity of the time-box model was at an all-time high. One of the challenges people faced in applying the time-box model was the difficulty of producing a truly production-ready solution increment in a short iteration. While iteration length in the Spiral model was originally measured in months, by 2010 most teams were running iterations of one or two weeks’ duration. It seemed as if we were running into an inherent boundary of the time-box model; once you drive down the iteration length to a certain level, it becomes significantly more difficult to adhere to the model.

To address this issue, the kanban development method uses the concept of cadence. A cadence is a regular rhythm or heartbeat for the work, but isn’t treated as a time-boxed iteration. By decoupling the development cadence from the release cadence, teams can delivery a production-ready solution increment in each release without necessarily having to do so within the scope of a much shorter development cadence.

Naked Planning (Continuous flow model)

The lightest-weight process model I have seen was developed by Arlo Belshee when he was running a software startup incubator. The goal was to take a new idea from concept to cash (at least one real sale) in six weeks. To meet this ambitious goal, developers needed a process model that wasted no time in process compliance overhead. The result was this Lean-based model, called Naked Planning.

If this description is wrong, it is my fault. It is based on my memory of a conversation with Arlo a few years ago. Hopefully the description will be adequate for the purpose of this blog post, which is only to provide a few summary examples of real-world processes that illustrate the four reference models.

A new idea at the time was the MMF, or minimum marketable feature. Since the goal was to realize a sale no later than six weeks from project inception, the unit of product represented the smallest subset of functionality that a real customer would be willing to pay for. What was unique about this was the fact a deliverable was defined by the customer’s willingness to purchase (which is the most definitive expression of customer-defined value), and not by criteria such as level of effort, estimated development time, technical complexity, or budget allocation (all of which are of interest to development teams, but not necessarily to customers).

Naked Planning is a single-piece pull system. Given a prioritized list of MMFs, the development team takes the top item in the list and builds it. Whether the item is large or small makes no difference; it is the next-most-important thing to do from the customer’s perspective, so the only activity that focuses on “value” is to build that item. Thus, Naked Planning as such does not directly deal with planning activities upstream from development. In that sense you might think of it as a partial process.

Staged Delivery (hybrid of Linear and Iterative models)

The idea of staged delivery was proposed by different luminaries in the field of software methodologies, including (independently) Tom Gilb and Steve McConnell. It is essentially a linear SDLC model with one adjustment: Once the high-level, initial work has been completed, the detailed development work of low-level design, coding, testing, and deployment is repeated through two or more stages. Thus, at the “front end” of a project it is a plain linear SDLC process, and at the “back end” it is an iterative process.

Staged delivery embraces the reality that “[t]he existence of the software artifact changes the requirements for it.” (see C2 wiki: http://c2.com/cgi/wiki?EvolutionaryDelivery)

Importantly, each staged delivery provides real business value to stakeholders. The software is put into production and used for real work. Feedback from stakeholders informs the plan for the next release, and is based on real-world use of the product rather than on contrived use in a test environment.

DSDM (hybrid of Linear and Iterative models)

The Dynamic Systems Development Method (DSDM) represents another way in which elements of the linear SDLC model and elements of the iterative model can be combined.

In this case, each iteration does not result in any portion of the product in a fully-elaborated and production-ready form. Instead, the iterations represent major milestones in an SDLC-style development process.

The iterations provide opportunities for feedback, but with a different focus than in the case of staged delivery. With DSDM, a prototype is created and used to solicit stakeholder feedback. Incorporating that feedback into the next iteration, a more fully-elaborated prototype is developed and used to solicit further feedback. Ultimately, the “real” product is delivered.

SDLC elements include the sequential nature of the process and the emphasis on end-of-iteration reviews and sign-offs. Iterative elements include the delivery of specific artifacts per iteration and the emphasis on customer feedback to drive improvements in the design.

MSF (hybrid of Linear and Iterative models)

The Microsoft Solutions Framework (MSF) is a model some people refer to as an iterative waterfall. A large program is broken up into a series of smaller projects. The program is completed by delivering each of the smaller projects in turn. This avoids many of the problems inherent in the linear SDLC model when it is applied to large programs.

A diagram of the MSF is available on Microsoft Technet at http://technet.microsoft.com/en-us/library/Bb497060.ors01_02_big%28en-us,TechNet.10%29.gif.

Each of the smaller projects is carried out in an SDLC style. If you unroll the diagram on the slide and lay it out flat, it would look like the canonical “waterfall” model.

Scrumban (hybrid of Time-Box and Continuous flow models)

“Scrumban” is a term coined by Corey Ladas to describe a Scrum process that evolves by incorporating ideas from Lean Thinking and the Kanban software development approach. There is no rigid definition of scrumban. It is a blending of Scrum and Lean concepts and techniques.

Source: Lean Software Engineering

In the illustration, note the board is not a kanban board, strictly speaking, as there are no queues between the value-add steps, and yet it does show WIP limits, as a kanban board would do. What happened is that a Scrum team added WIP limits to help themselves see how the work was flowing, and to help them improve their delivery effectiveness.

A useful aspect of scrumban is that it represents a pragmatic approach to improving a defined process model.

Implications

A common error is to assume that the metrics associated with a particular process model (if any) are always applicable when that process model is used. In my experience, the choice of metrics depends on the combination of the approach to Iron Triangle management and the process model. In some cases, the delivery mode (covered in the next post) may also influence the choice of metrics, but it is less significant.

For example, Scrum is currently one of the most popular process models for software development work. It is an example of a time-box process model. Due to its popularity, Scrum is used in a very wide range of contexts and domains. The majority of teams using Scrum plan their work by sizing the work items in terms of an arbitrary, relative point system which they call “story points,” “T-shirt sizes,” “units,” or a similar name. The number of such units the team can produce in a single iteration (called a “sprint” in Scrum parlance) is the team’s velocity. By making empirical observations of velocity, one can project the team’s likely future performance by drawing a trend line from a series of observations.

A forward projection of performance based on a series of observations is a typical planning approach when we are doing adaptive development. We are focused on producing the best possible outcome in view of reality, rather than on completing a predefined scope within a predefined time frame. Besides that, when doing adaptive development we do not have a stable definition of 100% at the outset, so there is no meaningful way to use any sort of “percentage complete” metric. (That doesn’t stop people from using them in non-meaningful ways, of course.)

However, forward projection based on empirical observations is not appropriate when we are doing traditional development; that is, when approach (a) as defined in the previous post is used for Iron Triangle management. For traditional development, we need to detect variances from plan, because the plan is the definition of “correct.” To do that, we typically use some flavor of “percentage complete” measurements. In this case, a metric such as velocity is meaningless. Hence the importance of Iron Triangle management in choosing appropriate metrics.

What does “percentage complete” have to do with Scrum? Although it was originally conceived as a framework to support adaptive development, it turns out that Scrum is used far more often in conjunction with traditional methods than it is with adaptive methods. Because people assume velocity is a metric that automatically applies to Scrum, they try to use it in a traditional context. This often causes confusion, as the measurements are not meaningful in that context.

I’ve seen teams (or their managers) redefine “velocity” in terms of “percentage complete,” simply because it’s the only way they can make sense of the numbers they are able to obtain. The adulterated terminology then leads to further confusion about what the metrics mean; they are measuring percentage complete but calling it “velocity,” which confuses people who know what “velocity” means.

The real solution to this problem is to align metrics (and other management tools and techniques) with the methods people are really using, regardless of the buzzwords with which they label those methods. Labels are not reality.

In the next post, I will cover the third aspect of management that I think has significance for the ways in which we run software development initiatives: Delivery mode. That is a term I coined to distinguish between the discrete project model and the continuous support model of delivering software. Delivery mode has very significant implications for the approach to portfolio management and program management, but only limited impact on line management choices such as metrics to track progress.

1. Iron Triangle management
2. Process model
3. Delivery mode

The reason I think these aspects are important is that they affect the way we handle various management issues, particularly our choice of metrics. Metrics that apply to one option in one of these areas may be meaningless or misleading when applied to a different option.

As George Box famously wrote, all models are wrong but some models are useful. The model I present here is based on my own experience and observation. It is wrong by definition by virtue of the fact it is a model; but hopefully it is useful, as well.

The order in which the aspects are listed reflects their relative effect on our choices, in my opinion. The strongest effect comes from our approach to managing the Iron Triangle. That is the subject of this post.

The process model is also important, especially because many people conflate the process model with the approach to Iron Triangle management. For example, a common conceptual error is to assume that if we are using a timeboxed iterative process then we are, by definition, doing adaptive development. When we then have difficulty making sense of the metrics we are using, it is because we are trying to measure things that are not happening. I have seen people get pretty frustrated about this. My observation is that Iron Triangle management and process model are orthogonal. When we are clear about that, we can choose metrics appropriate to our own situation.

The final aspect, delivery mode, refers to the difference between continuous product support and discrete projects. It has substantially less impact on our choice of metrics than the first two factors, but sometimes can be important.

Before going further, let’s clarify the meaning of Iron Triangle. You may know it by the name, triple constraint. It refers to three key elements of any project or initiative: Scope, schedule, and budget. Some people include quality in the list, but in my view quality is part of scope, since a product that doesn’t meet the necessary quality standard to be acceptable can’t be considered “done.” So, by Iron Triangle I mean scope, schedule, and budget.

In my experience, there are basically just two ways to manage the Iron Triangle, and the choice depends primarily on how the project plan is used to guide and track progress. To assess our progress, we compare our actual results with our plan. When we see variance, we change something. The two approaches are (a) change reality to conform with the plan, or (b) change the plan to take reality into account. Either way, the plan is our reference for assessing progress.

The choice of approach has implications for the way we construct the plan, the metrics we use to track progress, and the corrective measures we tend take when we detect variance from plan. When the plan itself is the definition of success, we use approach (a) to deal with variances. Sticking to the plan is, by definition, the right thing to do. When the plan is treated as a point-in-time navigational aid to help us stay on track toward customer-defined goals, we use approach (b) to deal with variances. In this case, the plan defines expectations but not success. We always have some sort of plan and some sort of expectations.

There’s a saying that appears in many variations and in many contexts that goes something like this: “No plan survives first contact with the enemy.” I don’t know who first said it, or who traditionally gets credit for doing so. In any case, the idea applies to software projects as well as to other activities. Even when we are using approach (a) to deal with the Iron Triangle, the plan is subject to change. There will be formal change control procedures that can be used to make minor changes. To make major changes to the plan, people re-assess risks and costs and “re-baseline” the plan.

Even with these mechanisms for change, the general assumption with approach (a) is that the plan itself is the definition of success. When there is variance, managers ask, “What are you doing to get back on plan?” If we are to understand the mentality about the Iron Triangle so that we can choose appropriate metrics, we must be clear about this fact, even if people are adamant that they are “flexible” about accommodating change. It isn’t a question of making people feel good about being flexible in the midst of an inflexible process; it’s about choosing metrics that will help us make well-informed business decisions throughout the project.

With approach (b), the definition of success focuses more on what will be delivered at the end of the initiative than on what was envisioned at the beginning. The general approach is sometimes called adaptive development. The idea is that we learn as we go along. Everyone involved learns more about what will really be needed in the solution, about what the solution can look like, and about how to progress from where we are today to where we need to be in the future. Frequent change is part and parcel of the approach, and so there are few formalities around changing the plan. When there is variance from expectations, managers ask, “What is the best result that can realistically be achieved?”

What are the key differences in these two approaches that inform our choice of metrics? I think there is just one: The level of detail in the plan at the outset.

With approach (a), it is necessary to resolve or mitigate all risks, identify all architectural issues, and define the full scope of the work in advance. Otherwise, it would not be possible for the plan to serve as the definition of success. Metrics used with this approach are usually based on the notion of percentage complete in one form or another. The comprehensive, detailed plan firmly defines the meaning of 100%. It defines 100% of scope, 100% of schedule, and 100% of budget in advance. Popular metrics such as Earned Value and Earned Schedule compare actual performance to date with the plan’s definition of 100%. When the plan is re-baselined, we establish new definitions of 100%, which are considered to be stable for purposes of percentage-based metrics.

With adaptive development, people identify and resolve or mitigate risks at the macro level, establish a high-level estimate for overall cost and time, and define a vision for the capability to be developed. Up to that point, the work is very similar to approach (a), but then they begin work based on the high-level plan or roadmap. They analyze needs and design detailed portions of the solution incrementally as they progress. They defer decisions so that they will avoid investing in work that is ultimately discarded or revised. They deliver incrementally in order to solicit feedback from stakeholders along the way, and adjust course accordingly.

With this approach there need be no stable definition of 100% of any of the three sides of the Iron Triangle before development begins, in principle. In practice, people do firmly define at least one factor. When time-to-market is the key business driver for the project, then schedule may be firmly defined at the outset. When scope has to be delivered in full for the solution to be meaningful, then scope may be firmly defined at the outset. When a fixed amount of funding is available, then budget may be firmly defined at the outset. But the plan is not comprehensive, and one or two sides of the Iron Triangle are meant to be flexible. Lacking a stable definition of 100%, percentage-based metrics are meaningless. The adaptive approach calls for different metrics.

Identifying the approach in use

Since people tend to use popular buzzwords to mean whatever they need the words to mean, how can we recognize the approach to Iron Triangle management that people are really using in any given organization? In my experience, there are several (but not too many) key distinguishing characteristics that tell us which approach people are using. They are:

These are mostly of equal weight in making a judgment about how the Iron Triangle is managed, but the first one, funding, is more significant than the others. It’s a commonplace to say that nothing happens without funding. How the funding happens gives us a strong clue as to the prevailing mentality about the Iron Triangle in the organization.

The funding model and the Iron Triangle

Conventionally, funding is determined before work begins on an initiative. The initiative has a predetermined budget allocation. Ongoing reassessment of funding needs is not part of the formal delivery process. Changes in the budget allocation are treated as exceptions.

Here is the tell-tale clue by which you can recognize this funding model: Changes in funding are driven from the project management level. When the PM realizes that some required change in scope or some unanticipated occurrence calls for a change in funding, he/she has to initiate a formal process to request that change.

With the adaptive approach, funding is allocated for shorter periods of time; say, three or four months. Periodically, all work in progress in the organization is reassessed against the ever-evolving understanding of business needs, market conditions, and revised business plans. Each initiative then receives more or less or the same funding for the next three or four month period, as well as any adjustments in other resources that may be deemed appropriate.

Here is the tell-tale clue by which you can recognize this funding model: Changes in funding are driven from the portfolio management or program management level. Reasessment for purposes of allocating funds is a recurring and regular part of the overall delivery process. Changes in funding are expected and planned for, and not considered exceptions.

Here is the key implication for recognizing the organization’s approach to Iron Triangle management: If a conventional funding model is used, it is all but impossible for any other aspects of the work to take an adaptive approach. Therefore, this is a clear sign that the organization uses approach (a) to manage the Iron Triangle.

However, the opposite assertion is not true. The fact an organization uses incremental funding does not automatically mean they take an adaptive approach to the Iron Triangle. It only means an adaptive approach is not precluded by the funding model. In this case we have to look at additional factors to determine how the Iron Triangle is managed.

Scope definition and the Iron Triangle

Given an incremental funding model, we know it is possible to take an adaptive approach to the Iron Triangle. We can usually tell whether that is actually true by looking at the way in which scope is defined for software development initiatives.

To support approach (a) to Iron Triangle management, the organization’s delivery process typically includes relatively extended analysis and design “phases.” The output from these phases will look like a comprehensive WBS and a technical design that, while possibly not fully detailed, is more detailed than really necessary, and that includes work far out in the future in the project plan. Project management tools will be Gantt-like, notwithstanding the buzzwords in the tools’ names or on the titles of formal progress reports. Metrics may be in use that are usually associated with adaptive methods, but their meaning is twisted to represent some flavor of “percentage complete,” which by definition requires people to know what “complete” means in some detail.

In contrast, if you see that the formal process interleaves analysis and design work with other aspects of development, such as coding, testing, and creating end-user documentation, and if the scope definition defines work items far out in the future at a high level of detail, then the indication is that people are taking an adaptive approach to the work.

Attitude also plays a role here. When people use incremental funding in conjunction with a comprehensive WBS, consider their reasoning for doing so. If they are concerned about minimizing losses in the event the planned scope, schedule, and budget will not be met, the implication is that the Iron Triangle is king, and they are using approach (a). If they are using an incremental approach to the comprehensive WBS as a means of delivering the maximum possible value to stakeholders, the implication is that they are using an adaptive approach to tackle a project whose scope happened to be well-known at the outset, and they are using approach (b). Note that people will almost always tell you they understood the “requirements” at the outset, if the organizational culture requires them to say so; and they will almost always say they are working in an adaptive way if they believe that is what people wish to hear. Consider, instead, the way they work and the attitudes they exhibit.

Other factors

The remaining distinguishing characteristics in the table provide supporting evidence for our observations of whether the organization takes approach (a) or approach (b) to the Iron Triangle. They are not the primary determining factors, though. They are more in the nature of effects than causes.

Laurent writes,

What is odd, to any experienced software engineer, is that the Cone diagram is symmetrical, meaning that it is equally possible for an early estimate to be an over-estimate or an underestimate. This does not really square with widespread experience of software projects, which are much more often late than they are early.

Er…wait a second. Isn’t that very assertion an example of “anecdotal evidence?” Is it another Leprechaun? Who says projects are more often late than early?

Silly Dave! Everyone knows who said that. It was the Standish Group, in their famous Chaos Report of 1994. Slightly more than 70% of IT projects fail. That’s why every office building in the industrialized world is on fire, and white-collar refugees are streaming from the cities, leaving their belongings behind except for whatever personal electronic devices they can carry.

Only…they aren’t.

To the contrary, companies seem to be doing okay with their IT initiatives. Enthusiasts of one software development approach or another are quick to say Everyone Else is “failing,” (as a way of creating a sense of urgency about adopting their Favorite Thing, I guess), but Everyone Else isn’t too sure who they’re talking about. If anything (and I’ll risk tossing out a bit of anecdotal evidence myself, here), larger companies can get away with being pretty inefficient.

With size, apparently, comes a steady flow of revenue, no matter what (almost) a company does. In large organizations that shuffle information around instead of building things, nearly every moment of every day is spent waiting for someone else to tick a checkbox on an electronic form or to sign off on a document. Most information workers have a large queue of outstanding requests for checkbox ticks and document approvals. They are very busy waiting for each other, and they can prove it because they charge all this time to specific cost buckets.

The proportion of time spent adding value to a product varies inversely with company size. And yet, it seems to make no difference at all to the bottom line. My observation is that once an information-shuffling company grows beyond a certain threshold (perhaps 30,000 people, give or take 5,000), they have to try really, really hard to “fail.” That doesn’t mean they never have any problems, but failure is final. They might have a process cycle efficiency in the IT function of under 2%, and it has no effect on their business success. The bit about the burning buildings and columns of escaping white-collar refugees, or lack thereof, suggests that this is not a problem. There is no crisis.

Sure, they’re wasting a lot of time and money, and the quality of working life for the technical staff isn’t the best, but they aren’t “failing” left and right.

Well, if they’re so inefficient, then why aren’t they failing? It seems to me that an information-shuffling company doesn’t really face the same time-to-market pressure as a thing-making company. Consider a bank or an insurance firm or a credit card issuer. How frequently do their customers sit down and assess the alternatives? Do people switch banks or insurance providers every couple of months? Not really. People don’t think about those things every day. Services like those are in the background of people’s lives. Customers consider switching only when they experience problems dealing with their current provider.

Other sorts of information-shufflers share this trait, even if they do have physical infrastructure to worry about. People don’t switch wireless phone companies or television providers any more often than they switch banks or insurers. It isn’t mainly because of the long-term contracts those companies encourage people to sign; it’s more basic than that. As long as the TV signal doesn’t die during the Big Game and the phone network doesn’t drop their calls too often, customers have more important things on their minds, or at least more interesting things. What this means is these companies can be pretty slow about getting software projects done, and it doesn’t impact their customers very much.

Where’s the crisis, then? Well, the kind of IT worker who entered the field in anticipation of facing creative challenges every day and working on technically-interesting projects alongside deeply passionate and committed colleagues isn’t going to find much professional satisfaction by making routine modifications to the same legacy code base over and over again, all the while dragging their projects through a maze of pointless administrative procedures as if dragging an anchor through a swamp. That’s understandable, but it doesn’t amount to an industry-wide business problem of “crisis” proportions. Nor is the feeling shared by the majority of IT workers, for whom the job is just a job, and whose personal passion in life lies elsewhere.

So, what about the Chaos Report? In a 2005 article in IEEE Software, IT Failure Rates: 70% or 10-15%?, Robert L. Glass explains “what” about it. He suggests (these are direct quotes):

• “Failure” and “success” are tricky words in the software business. How do you categorize a project that is functionally brilliant but misses its cost or schedule targets by 10 percent? Literalists would call it a failure, realists a success.
• People tend to focus on organizations that fail. For example, an early US Government Accounting Office study of projects thatfailed, which came up with an astounding failure rate, turned out to have examined projects already known to be in trouble when the study began.

I agree that “failure” and “success” are tricky words. I often say they reflect binary thinking, because it’s either one or the other. The words don’t allow for a mixture of positive, negative, and neutral outcomes, such as we usually encounter in the mystical land known as Real Life. Besides, those words feel very final to me. Once you’ve failed, you’re done. You might as well slit your belly open with a tanto. On the other hand, once you’ve succeeded, you’re done. You might as well buy a villa in Tuscany and retire.

That technically brilliant project that missed its budget and schedule target had a mixed outcome. In some respects the outcome was positive. In some respects the outcome was negative. Binary, final words like “success” and “failure” tend to steer us away from analyzing why and how those different outcomes occurred. We celebrate or we rationalize, but we don’t bother to analyze and learn. Maybe that’s okay, if you’re not interested in continuous improvement.

I wonder if the real problem with IT project success rates is that we’ve been working from a faulty definition of “success,” lo these many years. The canonical definition of “success” for an IT project is “all features delivered on time and on budget.”

That definition suggests a cost center mentality. We get our budget, our timelines, and our requirements from Someone Else. Our fiduciary responsibility is to use the budget allocation for its intended purpose, and prevent its being squandered for other things, like office supplies and pizza. What’s missing from the definition? Any mention of a customer or of value. People do like to talk about value, of course. Consider our old friend, Earned Value, which re-casts “budgeted amount used up” in the award-winning role of “value delivered.” It’s still just “budgeted amount used up,” when you scrape the makeup off. But value sounds so much better!

So, what might a definition of “success” for a software project look like, if it didn’t reflect a cost center mentality? Maybe it would be something like this: “The business capability our customer requires, delivered at the right time, at the right price point, and with an appropriate level of quality for the purpose.” That definition could stand some refinement, but I like the fact it mentions the customer and his/her needs, and doesn’t just talk about how smoothly we used up someone else’s money to deliver something that was envisioned months or years earlier by someone who was trying to guess what the customer might like to have.

I’m still not too happy about the word “success,” though. The definition may be better, but the word is still the wrong word. You see, as soon as the customer has this thing in hand, this thing that supports the business capability he/she needs just now, at the right price etc., he/she is going to be looking ahead to the Next Thing. So, is it “success,” really, or just one step on a longer journey? Should we check out that villa in Tuscany, or start thinking about the Next Thing?

As words go, “failure” is no better than “success.” Rudyard Kipling got it right when he called triumph and disaster “impostors.” People like to say they learn from their failures. Okay. I guess it means 85-90% of the time (or 29%, if you accept the 1994 Chaos Report), they learn nothing from their experiences. I prefer to learn from the outcomes that I experience; that mixture of positive, negative, and neutral results that offers so many learning opportunities. I don’t learn only from failure, but from all experience, if I can.

Of course, it’s easy for me to learn, as I don’t know much to begin with. There’s no problem pouring Guinness into an empty glass, provided one doesn’t try to do it too fast. Some of the smarter folks out there may be holding a full glass already. That’s a whole nother problem.

Small things are easier to move around than big things. I can’t cite any published academic studies that prove that assertion, but you can conduct a quick-and-dirty experiment right now. Find the largest, heaviest piece of furniture in the room where you are. Try to move it across the floor. Oh, come on. You’re not even trying. Put your back into it! Heave! Heave!. Yeah, that’s more like it. Now imagine what it would be like if you had to line up the furniture one piece behind another to represent a “priority sequence” of furniture. What would happen when your customer demanded a sudden and immediate change of priority? How easy would it be to shuffle all those heavy pieces around into a different sequence? How easy would it be to “cancel” a heavy piece of furniture by picking it up and throwing it out the window?

Now, locate a small, light object on your desk; perhaps a wireless mouse, a thumb drive, or (if such objects still exist in your world) a pencil. Try to move it around the room. How did the level of effort compare with that of moving the piece of furniture around the room? Line up a few of these objects on the desktop to represent a “priority sequence.” Now pretend your customer demanded a sudden and immediate change of priority. How difficult was it to re-order the small, lightweight objects? How hard would it be to “cancel” one of them by dropping it into the wastebasket? Did you notice a difference in the level of effort required as compared with rearranging the furniture?

It’s the same with the changeable things in our work. Small things are easy to shuffle around and relatively cheap to drop in mid-stream. Big things tend to take on a sort of inertia. They are hard to turn and harder to stop. There are even some insidious problems with large chunks of work that don’t seem to occur with smaller chunks. For instance, people tend to associate size with importance. Rather than considering the true business value of various projects, people tend to assume the larger projects are inherently more important than smaller ones. People don’t like the idea of canceling large projects, but they will cancel smaller ones without worrying about it too much. It’s the same psychology casinos take advantage of to keep people on a losing streak: We’ve sunk so much money into this already, we just have to see it through to the end! We don’t usually feel that way about small projects; not even when it’s the same work as the larger project, but divided up into smaller pieces.

The same rule of thumb applies at all levels from managing a portfolio to defining a set of test cases, and to all activities involved in software delivery from requirements elaboration to solution deployment.

• Portfolio management — it’s easier to reshuffle, rescope, and cancel small projects than large ones; break large initiatives into a series of smaller ones.
• Budgeting — keep cash resources as liquid as possible; have funding available for exigencies; take periodic checkpoints of projects in flight, reassess priorities, risks, and potential value on a regular basis; adjust budget allocations accordingly.
• Requirements — create a high-level vision for a solution; refine requirements in detail for features soon to be developed, mitigate risks for features somewhat farther out in the future, avoid details for features planned for the distant future; think of “requirements” as options that have potential value and an expiration date; keep fine-grained feature definitions as independent of one another as possible, so they can be rescheduled without excessive impact on other work items.
• Coding — develop code in small chunks, interleaving unit test activities with coding activities to build quality in; strive for small, cohesive code units, loose coupling, and non-brittle interfaces; practice incremental refactoring to keep the codebase clean and easy to modify without introducing regressions.
• Integrating code — refresh your working copy of the codebase frequently; check in changes a little at a time and frequently; minimize the impact of collisions, minimize the need for tedious merges.
• Testing — test incrementally as code becomes available; avoid delays in providing feedback to upstream process steps for defect correction; participate directly in requirements elaboration to ensure all system behavior is testable, to minimize the number of different artifacts that express the same information, and ensure everyone involved in development has the same understanding of “done.”
• Writing user documentation — develop documentation incrementally, in sync with code development; don’t wait until all the code is finalized before beginning; avoid uneven workloads and the confusing results that can be caused by waiting long periods of time and then rushing to complete a mass of documentation quickly.
• Packaging for deployment — package and repackage incremental results frequently so that the packaging process does not become overwhelming and error-prone, and to expose problems early in the development process.
• Deployment — practice deploying the solution throughout development, to avoid surprises and significant implementation effort at the end; incrementally refine deployment scripts and procedures as the solution is incrementally built up.

I can’t help noticing the resemblance between the pattern of growth and collapse in historical states and empires and the pattern of growth and collapse in corporations. It seems as if corporations, like states and empires, undergo a process of increasing complexity as they learn to deal with competitive challenges and as they increase their market share, their size, and the scope and range of their business activities. Historian Arnold Toynbee famously described the pattern of ascendance and decline of civilizations in terms of internal moral decay. The anonymous author of a Wikipedia article on Societal Decay put it quite nicely when he or she wrote, "societies that develop great expertise in problem solving become incapable of solving new problems by overdeveloping their structures for solving old ones."

It is a "circle of life" pattern. Corporations rise when their creative ideas generate innovation that stimulate the economy. They decline when their thinking ossifies and they lose the ability to adapt to change. They collapse, leaving former employees free to innovate and to form new, creative companies to pursue fresh ideas. Then the cycle repeats. As in nature, this sort of cycle is healthy for the economy; it allows the old to die and the young to test their ideas in the crucible of the marketplace.

When companies are part of a larger organization, such as a nation-state, nature is not always allowed to take its course. We saw this effect in the financial crisis of 2008-2009, when the US government deemed some of the declining corporations "too big to fail," and handed them free money from tax coffers to sustain them and to enable them to occupy space in the market that would otherwise have been taken by creative entrepreneurs, in much the same way as a molecule of carbon monoxide occupies a receptor site for oxygen, causing the organism to die.

A government bail-out is more akin to reanimating dead tissue a la Frankenstein than to restoring true health. Fortunately, it is not the only way a company can avoid death. It is possible for the leadership to recognize the need for new thinking and new approaches to problems. Change is inevitable; death is optional. But choosing life over death may not be as simple as it sounds. The models of societal growth and decay offered by Toynbee and Tainter may provide clues about the nature of large organizations that help us understand why so many large corporations choose decay and death. Understanding that nature may help at least some corporate leaders make better choices.

Toynbee, Tainter, and business

In Toynbee’s historical model, civilizations pass through several distinct phases: Genesis, growth, time of troubles, universal state, and disintegration. The model strikes me as very similar to the pattern of life we all experience as individuals: Birth, childhood, adolescence, adulthood, and old age. Corporations can’t reverse the pattern and revert to a simpler, more nimble, more innovative form any more than we as individuals can "grow backward" toward childhood.

Toynbee calls those who figured out how to solve the "old problems" the Creative Minority. Over time, the Creative Minority transforms into the Dominant Minority. No longer creative, they force the majority to comply, and they forget how to embrace new ways of thinking. The Wikipedia article author writes, Toynbee "argues that creative minorities deteriorate due to a worship of their ‘former self,’ by which they become prideful, and fail to adequately address the next challenge they face." The Universal State comes into existence when the Creative Minority transforms into the Dominate Minority, and innovation beyond the well-defined boundaries set by the leadership is no longer tolerated.

Toynbee sees the root cause of the decline of civilizations as mainly a question of moral or spiritual decay from within. With the rise of the Universal State and the suppression of creativity and innovation in the name of conformity with the Dominant Minority, people become jaded about the spiritual values of the society. For that reason, the ancient Romans eventually stopped believing in the traditional gods, became cynical about the notion of emperor-worship, and were mostly skeptical of then-emerging alternative religions such as Christianity and Mithraism. They had lost their spiritual compass.

In the context of business, the equivalent of a spiritual compass is a company’s core values. Every company starts out with a set of core values with which the founders all agree — these are the ways in which we want to make the world a better place; here is our plan for doing so; and this is the line we will not cross to achieve our goals, for ethical reasons. Most new companies set their core values down in writing and the founders pledge to honor them. When the company reaches the point that it begins to grow rapidly — that first big sale, first big account, or first market-changing innovation — people start to deal with the practical challenges of rapid growth and forget the original core values. Pretty soon the corporate equivalent of the Universal State settles into place, and the old Creative Minority turns into a Dominant Minority that worships their own "former self," or perhaps a sanitized and glamorized image of the "former self."

Tainter’s model shares some elements with Toynbee’s, but focuses more on the inverse relationship between a society’s investment in complexity and the return on that investment over time. When the point of diminishing returns is reached, the society collapses. Thus, archaeologists find unfinished stonework and workers’ tools among the remains of Mayan cities, left in place when the workers walked away. I find the many parallels with the business world interesting.

Complex societies … are at least partly built up of social units that are themselves potentially stable and independent, and indeed at one time may have been so. Thus, a newly established state may include several formerly independent villages or ethnic groups, or an empire may incorporate previously established states. (Tainter, page 23)

Large corporations are at least partly built up of smaller organizations that are themselves potentially stable and independent, and indeed at one time may have been separate companies that were acquired by the larger firm. Thus, a large corporation may include several formerly independent companies.

Personal political ambition is either restrained from expression, or channeled to fulfill a public good. [...] [P]olitical organization extends beyond the community level. Accordingly, economic, political, and ceremonial life transcend purely local concerns. [...] There is a political economy in which rank conveys the authority to direct labor and economic surpluses. [...] Economic specialization, exchange, and coordination are characteristic features. (p. 25)

Real innovation and out-of-the-box thinking are either restrained from expression or channeled to fulfill an objective of the leadership. Economic, political, and ceremonial functions transcend purely departmental concerns. There is a political economy in which rank conveys the authority to direct labor and allocate resources. Functional specialization, formal interaction between functional silos, and coordination across administrative boundaries are characteristic features of large companies.

Occupational specialization is a prime characteristic, and is often reflected in patterns of residence. (p. 27)

Functional silos are a prime characteristic, often reflected in formal organizational structures in which individuals are slotted into management hierarchies dedicated to occupational specialties.

[A] ruling authority monopolizes sovereignty and delegates all power. [...] This ruling class supplies the personnel for government, which is a specialized decision-making organization with a monopoly of force, and with the power to draft for war or work, levy and collect taxes, and decree and enforce laws. (p. 27)

A ruling authority at the top of a hierarchical power structure monopolizes decision-making. This ruling class determines who is or is not on the "fast track" for promotion up the hieararchical chain of authority. Management is a specialized decision-making organization with a monopoly of force and with the power to hire, fire, and assign personnel as it sees fit, to establish and control employee compensation and benefits, and decree and enforce workplace rules.

The government is legitimately constituted, which is to say that a common, society-wide ideology exists that serves in part to validate the political organization of society. [...] States tend to be overwhelmingly concerned with maintaining their territorial integrity. This is, indeed, one of their primary characteristics. (p. 27)

Management is legitimately constituted, which is to say that a common, company-wide ideology exists that serves in part to validate the organizational structure of the firm. Corporations tend to be overwhelmingly concerned with maintaining the business models and operational standards that were devised by the original Creative Minority, even after those things have ceased to serve the company’s interests.

Despite an institutionalized authority structure, an ideological basis, and a monopoly of force, the rulers of states [have] the need to establish and constantly reinforce legitimacy. [...] [L]eadership activities and societal resources must be continuously devoted to this purpose. [...] No societal leader is ever far from the need to validate position and policy, and no hierarchical society can be organized without explicit provision for this need. (p. 27)

Large companies indulge in a variety of rituals and ceremonies that aim to reinforce the legitimacy of the management elite in the eyes of employees, stockholders, customers, business partners, and the public. They frequently reorganize, although not to the extent of questioning assumptions about functional specialization or administrative structure, or making deep changes in the established power hierarchy. They frequently announce process improvement initiatives such as Six Sigma, TQM, Kaizen, Lean, and others. They frequently engage outside firms to survey employee opinion, or solicit employee feedback through internal suggestion-box mechanisms of one kind or another. All these activities serve to remind everyone of exactly who is in charge. Usually, they have little lasting or meaningful effect beyond that.

Decline in support will not necessarily lead to the fall of a regime, for to a certain extent coercion can replace commitment to ensure compliance. Coercion, though, is a costly, ineffective strategy which can never be completely or permanently successful. Even with coercion, decline in popular support below some critical minimum leads infallibly to political failure. (p. 27)

In many large corporations, employees just perform the functions assigned to them without giving much thought to the company’s overarching mission or business goals. They do the things on which their performance is measured so that they will receive the standard pay rises and bonuses that accompany a "satisfactory" performance evaluation. They mock the ridiculous procedures that may once have added value even as they perform those procedures to the letter. Their true passion and the bulk of their emotional and mental energy are reserved for their families and hobbies. Their innate creativity and intelligence are unavailable to their employer.

Death is optional

It may be true that the Romans and the Mayans couldn’t conceive of a different way of thinking, and followed the usual pattern through decay and dissolution, but a modern-day corporation doesn’t have to fall victim to the same pattern. If the Dominant Minority is able to maintain an open mind about new ideas, they can lead their company through economic change time and again. Two examples from my own past experience illustrate this.

In the mid-1980s, I worked with a division of JCPenney known as Business Services. The organization operated a transaction processing network across the United States. That doesn’t sound unusual today, in the age of the Internet, but at the time it was unique. JCPenney was, and still is, a company whose primary business is retail store operations. They developed the network to support their own needs for point of sale credit authorization services. They realized that the facility could become a revenue generator in its own right, and they began to sell credit authorization services to other companies. Eventually, they spun the operation off and it became part of a new company formed in 1996, Alliance Data Systems. Today, in 2012, both JCPenney and Alliance Data Systems are still in business and profitable. Recently, JCPenney announced changes in its operating model that represent yet another adaptation to changing market conditions. The leadership of these two companies have not lost their original flexibility and innovative spirit.

A counterexample is a financial services firm where I worked in the early 2000s. The company followed the pattern described by Tainter. By early 2008, the company’s complexity could no longer be supported by its revenues. The leadership begged investors for a $7 billion-with-a-B cash infusion to keep the company afloat. When the financial crisis struck in late 2008, they had no way to absorb the shock. It wasn’t long before the company was bought out by a competitor. It no longer exists.

I recall that the management of the IT department at that company perceived their department as a sort of "external" service provider to business stakeholders in the firm. They had established a working relationship with the rest of the company in which line-of-business managers would request IT services, such as a new business application, and the IT department would submit a bid to the line of business for doing the work. Since they seemed to like the model of the external service provider, I proposed that we respond to RFPs from outside the company, so that we could use our delivery mechanism to generate revenue directly. IT management were shocked and appalled at the suggestion. It broke all the existing rules and represented innovative thinking beyond the boundaries they had established. I was severely reprimanded for floating the idea.

In hindsight, my suggestion was very much like the one someone had made long earlier at JCPenney, to use their credit authorization network to generate revenue, and to transform Business Services from a cost center into a profit center. That suggestion, apparently, had been more warmly received.

The point is that we can learn from the patterns observed in history. It isn’t necessary to fall prey to the same pattern of decay and death over and over again. It may be that nation-states can’t avoid the pattern, but corporations can. It all comes down to the choices they make.

Utilization thinking

In the era of the Industrial Revolution, many manufacturing operations became mechanized. Machines were expensive, and factory owners wanted to be sure they got their money’s worth from each one. Mechanized manufacturing was something that had never existed previously, and the manufacturing industry had no collective experience on which to base a clear understanding of how to achieve the highest return on investment using this sort of operation. A naive assessment of a mechanized manufacturing operation easily leads to the conclusion that the way to get the most value for the investment in the machines is to run each unit at its maximum capacity all the time. That was the way people thought about it in those days, and many people still think that way today. This mentality is called utilization thinking.

From the point of view of plant operations, utilization thinking leads to local optima at the expense of the global optimum. When we assess the operation of each machine or each step along the production line, we find that each resource is operating at its own peak capacity; it is fully utilized at all times. Based on utilization thinking, this is all good.

However, inventories of unfinished product tend to accumulate between the production steps, since not every machine on the line has exactly the same capacity or can operate uninterrupted for an indefinite time. Production pauses locally for changes in machine set-up, to switch from making one component to another. Machines with higher capacity generate unfinished goods that cannot be consumed at the same rate by machines downstream on the line. Eventually the factory floor becomes clogged with unfinished goods, and the factory manager must resort to transporting inventory between the production floor and warehouses. All this activity reduces the efficiency of the production line as a whole.

Meanwhile, when a downstream machine has higher capacity than an upstream machine, it cannot be operated at full capacity all the time. Rather than worrying about end-to-end production, plant managers often worry about the fact a particular machine is not running at full capacity, and they solve the wrong problem. An online guide to implementing the Theory of Constraints, written by Dr K.J. Youngman, provides a concise and useful overview of various approaches to shop floor scheduling, including a good description of this problem.

Resources and humans

To make matters worse, as of the turn of the 20th century human beings were considered to be equivalent to machines, or perhaps more accurately, interchangeable machine parts, with the factory itself as the machine. Building on this assumption, the notion of human beings as “resources” became entrenched in mainstream management thinking very early on in the Industrial Age, and was taken for granted by the beginning of the Information Age.

This attitude toward workers was expressed powerfully in the classic 1906 Upton Sinclair novel, The Jungle, and was formalized and codified through the work of Frederick Taylor, most famously with the publication of Principles of Scientific Management in 1911. The following quote from Taylor, cited by David Montgomery in his 1988 book, The Fall of the House of Labor: The Workplace, the State, and American Labor Activism, 1865-1925, illustrates the key underlying assumption of Scientific Management:

“I can say, without the slightest hesitation,” Taylor told a congressional committee, “that the science of handling pig-iron is so great that the man who is … physically able to handle pig-iron and is sufficiently phlegmatic and stupid to choose this for his occupation is rarely able to comprehend the science of handling pig-iron.”

Montgomery also notes:

With the triumph of scientific management, unions would have nothing left to do, and they would have been cleansed of their most evil feature: the restriction of output.

So, the fact that humans did not actually function as machine parts was taken by management not as a symptom of basic reality, but rather as evidence that workers desired to “restrict output” because they were inherently “evil,” not to mention “phlegmatic and stupid.” Thus Taylor was able to write, without irony, in Principles of Scientific Management,

It is only through enforced standardization of methods, enforced adoption of the best implements and working conditions, and enforced cooperation that this faster work can be assured. And the duty of enforcing the adoption of standards and enforcing this cooperation rests with management alone.

From the 1970s through the 1990s, most companies treated information systems workers (and other employees, too) as if they had to be forced to cooperate with management’s goals by forcing them to comply with standard methods. Humans were measured on the same basis as resources (e.g., chairs, telephones, or staplers), with the assumption than any suitably trained human would function identically in a given role as any other suitably trained human, just as any chair will serve as well as any other.

Like machines, the worth of a human was measured in terms of productivity — the quantity of output per unit time. Also like machines, the assumption was that the cost of a human was his/her salary plus benefits and facilities overhead, and that the best way to recoup that cost was to keep him/her running at maximum capacity at all times. This tended to create a local optimization problem analogous to that experienced in physical manufacturing plants. It is also the basis of the preoccupation with tracking individual workers’ hours against pre-assigned tasks, rather than allowing people to use their own brains to achieve the common goals of the company.

Throughput

Eliayu Goldratt elaborated the Theory of Constraints over the course of a number of years. The evolution of the idea can be traced through the publication of three key books: The Goal (1984), Critical Chain (1997), and Theory of Constraints (1999). The constraint in Theory of Constraints is anything that hinders a system from achieving its goals. The basic idea is that any system is limited by just a few constraints, and that there is always at least one constraint. Based on the general idea of the “weakest link in a chain,” ToC defines an iterative process improvement technique known as the Five Focusing Steps, which aims to address the main constraint in a process, whatever it may be at any given time.

When the system in question is a for-profit business enterprise, then the goal is to earn a profit. Profit is earned by providing a product or service that a customer deems valuable, and will pay for. The system generates value units, which are units of currency in a for-profit enterprise, and may be some other unit of measure for non-profit enterprises and government agencies. The rate of production of value units is called throughput.

The Theory of Constraints represents an evolutionary progression of thought and application that has several lines of ancestry. Probably the main line of descent is the production system developed at Toyota in Japan. Company founders Sakichi and Kiichiro Toyoda, along with engineer Taiichi Ohno, developed an approach they called just-in-time production in the late 1940s. Their ideas received a significant boost in 1950 when the American consultant W. Edwards Deming began to work with business leaders in Japan. Building on those foundations, Ohno, along with Shigeo Shingo and Eiji Toyoda, developed the Toyota Production System (TPS), which reached its final form around 1975. TPS became the basis of Lean Manufacturing, currently the most popular trend in the manufacturing sector.

In adapting these ideas to the field of software development, we usually find ourselves working with only a subset of the overall end-to-end delivery process of a company. We are concerned with the portion of the process that involves creating software. Thus, the value unit is some unit of measure of delivered software, and throughput is the rate at which the software development process delivers those units.

The TPS and Lean Manufacturing, as well as the broader umbrella term, Lean Thinking, combine both the mechanics of a process and the human element. It differs from Scientific Management in its assumptions in both those areas. On the mechanical side of things, Lean Thinking focuses on global optimization of the end-to-end delivery process, rather than on the utilization of individual resources. On the human side of things, Lean Thinking explicitly calls for respect for people and for teamwork. Rather than assuming workers want to “restrict output” because they are “evil,” the assumption is that workers have the same business-related goals as the company owners — to build the success of the company by maximizing throughput. Thus, with Lean Thinking we have both a focus on efficient mechanics, and practical leverage of people’s innate desire to do good work, add value, and produce useful things.

Beyond the conceptual barrier

When people apply utilization thinking, they tend to overlook the problems that lead to low throughput because they are focused on individual resource utilization, on limiting employees’ activities to those allowed in their formally-defined job roles, and on tracking the time each employee spends on externally-assigned tasks. People feel as if they must remain visibly busy at all times, even when that means working on things that do not contribute to throughput. People feel they are not allowed to do anything outside the boundaries of their defined roles, and that there is no place in the organization for their innate creativity.

When people apply throughput thinking, they tend to see the problems that reduce delivery effectiveness because they are mindfully looking for those problems. Many of the details of Lean Thinking become obvious to them, and no longer require lengthy explanation. When a person sees a problem, whatever his/her role or position on the organization chart, he/she feels empowered to solve it, and to involve whomever else in the organization has the skills and tools to help solve it.

Throughput thinking helps people see time in a different way. They begin to look for points in the process where work flow is delayed, blocked, uneven, or when the same piece of work has to be re-done. These things become much more important than worrying about whether John Smith spent exactly 6.25 hours on task #125.46(a) last Tuesday. They shift from Cost Accounting to Throughput Accounting, which gives them a much better idea of how well they are achieving the true goals of the enterprise. They start to rely on metrics that track the outcomes achieved by the process rather than how busy individual resources are; metrics like throughput, cycle time, lead time, and process cycle efficiency. They begin to take advantage of the rarely-tapped creativity of their co-workers and the power of genuine teamwork, rather than locking people into narrowly-defined functional silos and trying to march them in lock-step along a static path.

Utilization thinking makes all these things much more difficult to implement, because they never feel quite right. A critical success factor, then, is to break through the conceptual barrier of throughput thinking, and leave utilization thinking behind, back in the era of Frederick Taylor and P.T. Barnum where it belongs.

One of the most interesting sessions I attended at XP Days Germany was Developer Awareness, given by Shamsuddin Butt. Butt has a dual background in software development and psychology, which gives him a unique perspective on issues of organizational culture change, team dynamics, individual performance, and coaching.

In this session, Butt applied concepts from Timothy Gallwey’s book The Inner Game of Tennis to the question of individual and team performance in software development. In the book, Gallwey describes the “inner game” within a player between his/her Self 1, who is judgmental and controlling, and Self 2, who can realize the individual’s potential for high performance if only Self 1 would get out of his/her way. The basic idea is to feel what you’re doing and just let yourself do it, rather than trying to monitor, control, and criticize yourself from an internally-imposed third-party viewpoint.

Butt demonstrated this through a couple of interactive exercises. The first one involved all the participants in the session. All but two people stood in a circle facing each other. Then each person looked to the right to take note of who was standing there. Next, everyone rearranged themselves in an attempt to put as much distance between him/herself and the person who had been standing to the right. At that point we were ready to begin the exercise.

Butt handed tennis balls to one person in the circle, one after another, until eight balls were in play. Each participant tossed balls to the person who had been standing to his/her left originally, and caught the balls tossed by the person who had been standing to the right, and who was now more-or-less opposite him/her in the circle. The last person in the sequence just dropped the balls. One of those who was not part of the ball tossing circle collected the balls after they had been “processed” and returned them to Butt, who fed them back into the circle. Finally, the other “outsider” counted the number of balls the circle “processed.” At the end of two minutes, we stopped and tallied the results.

The whole thing proceeded rather sloppily. One person gradually moved into the middle in an attempt to catch balls from his partner more easily. Of course, other balls struck him from the sides as other members of the circle tried to toss to their own partners. (Although Butt did not use this term, I thought it was a wonderful and natural example of local optimization, a problem that occurs in many organizations and on many teams!) On the whole, we dropped many balls. As the clock ran down, we tried to speed up, with the result that we lost many more balls. In the end, we “processed” 18 balls in two minutes.

The counter announced the score, and Butt suggested we set a goal for ourselves and try it again. Without discussion or planning, we repeated the exercise. The second time around, everything went far more smoothly. The group processed fewer balls, but only dropped one. It felt as if we were working slower, and yet we managed to “process” 20 balls. Not only was our net performance better, but the “work” felt palpably less stressful than the first time.

The point of all this was to illustrate the fact that teams will find their own rhythm if only you get out of their way and let them find it. It’s a key ingredient in the agile concept of self-organizing teams. The phenomenon should be very familiar to practitioners of Scrum, XP, or other agile and lean methods that allow teams to choose the amount of work to which they can commit during each iteration, rather than imposing an arbitrary workload on them based on a predetermined “due” date. In effect, the Self 1 of each member of the circle stepped aside and allowed the Self 2 to operate without interference. The result was an example of emergent behavior.

The second exercise illustrated the Self 1 vs Self 2 concept on an individual level. Butt asked for two volunteers: One who considered him/herself very good at ball games, and one who considered him/herself very bad at ball games. The “good” player tossed tennis balls to the “bad” player, who had to try and catch as many as possible. To make it a bit challenging, the thrower was asked to make it a just a little hard to reach each ball, throwing some high, some low, some to one side or the other, and so forth. On the first try, the catcher caught one out of eight balls thrown.

He was then asked to set a goal for himself. He set a goal of three balls. Butt asked him to say, out loud, what the trajectory of the ball looked like as it traveled through the air. Concentrating on describing the trajectory of the ball, the catcher in effect distracted his Self 1 and kept him out of the way, allowing his Self 2 to catch five out of eight balls thrown. At one point, the catcher remarked that he did not see the point of all this, although I thought it was fairly obvious. Afterwards, Butt asked him how it had felt to catch the balls. The catcher said it felt as if “something” other than him was catching the balls. This is, in fact, exactly how it feels to do anything when your Self 1 is out of the way. This is how it feels to be “in the zone.”

Butt’s point in all this was to illustrate the difference between mentoring and coaching. Mentoring is about showing people and teams how to do specific things. Coaching is about helping people and teams discover their own innate ability to excel.

It struck me that the whole idea of Self 1 vs Self 2 sounded a lot like the zen concept of no mind. I asked Butt about this after the session, and he said it was so; Gallwey had in fact studied with an Indian guru. Zen comes from Buddhism, which comes from India; it is indeed the same concept, but re-cast in a way that is accessible to the dualistic Western mind. The pervasiveness of dualistic thinking in the West is demonstrated by one reviewer’s comments about the book on Amazon’s website, in which he credits Gallwey with providing useful advice in spite of “aspects of psychobabble and mysticism.” I would say the ideas are useful because of those aspects.

1. Kanban is the latest cool buzzword.
2. We want to be (perceived as) cool.
3. Therefore, our card wall is a Kanban board.

Well, actually…

No, it isn’t.

Architect != any relatively senior-level job title, even if it has nothing to do with architecture

1. Fresh out of Foo School: I’m a Foo One.
2. Two years later: I’m a Foo Two.
3. Two years later: I’m a Senior Foo.
4. Two years later: I’m a Foo Engineer.
5. Two years later: I’m a Senior Foo Engineer.
6. Two years later: I’m a Foo Architect.
7. Two years later: I’m a Senior Foo Architect.

Well, actually…

No, you aren’t. You just a Foo with a fancy title.

“Agile” or “Lean” != whatever we were doing (or selling) before, with a fresh coat of paint

1. Our Gantt Chart software supports agile project planning.
2. We’re doing agile big design up front.
3. Our project is using a waterfall process, but we’re doing it in an agile way.
4. We don’t need to use sound software development practices, because agile development is all about values and culture.
5. Our project team is agile. Every morning, we all stand up and give our status reports to the project manager, and he gives us our assignments for the day.
6. We found a Lean way to improve Process Cycle Efficiency: Ignore non-value-add time during process steps. Besides, as long as we’re thinking about working, we’re adding value to the product, right?

Well, actually…

1. No, it doesn’t.
2. No, you aren’t.
3. No, you aren’t.
4. Yes, you do.
5. No, it isn’t.
6. Gosh, you’re adding value to your product right now, aren’t you? Impressive!

Refactoring != any sort of change made to any sort of object for any reason

1. I’m refactoring my desk. It’s a mess.
2. I have too many emails. I’m going to refactor my inbox.
3. I want to refactor my old Volvo into a new Mercedes.
4. This weekend I’m going to grab the lawnmower and refactor the grass.
5. We’ll refactor the application so that it displays custom screens based on user preferences.

Well, actually…

1. No, you aren’t.
2. No, you aren’t.
3. When you figure out how to do Ferraris, let me know.
4. No, you aren’t.
5. No, you won’t.

Story Points != a euphemism for time-based task estimation

1. We’ve pegged a “story point” to one hour of clock time. It makes story sizing so much easier!
2. If we know how many work hours our team will be available next iteration, we can calculate how many story points we’ll deliver.
3. After we’ve done our bottom-up task estimation in terms of time, we can roll that up into “story points.”

Well, actually…

1. That’s not a “story point,” it’s called an “hour.”
2. If you know how many work hours your team will be available next iteration, then you know how many work hours your team will be available next iteration.
3. Yeah, roll it up and smoke it. Your iteration plan won’t be any worse, and you might feel better.

Velocity != a bludgeon for beating more work out of a project team

1. If we divide the total scope (in story points) by the number of iterations, we’ll know the velocity the team has to set in order to make the deadline.
2. If the team can’t achieve the velocity target dictated by management, it has to be because the team is either incompetent or lazy.
3. “Continuous improvement” means challenging the team to increase velocity by 20% per iteration. That will inspire them to try harder.
4. If Team A delivers 50 story points per iteration and Team B delivers 40, then Team A is better than Team B.

Well, actually…

1. Sure. They’ll just “set” the velocity and punch the “start” button. No problem.
2. It could be both, right?
3. Yes. They’ll try harder to game the numbers.
4. Be sure and remember that at bonus time. Team A will appreciate it, and so will your competitors, where the members of Team B will be working. And just watch Teams C and D adjust their estimation process. They’ll be delivering hundreds of story points per iteration!

Persona != a cool-sounding synonym for “actor” or “user type”

1. Our application has four personas: Member, Guest, Administrator, and Back-End System.
2. The only persona we need to satisfy is our boss…

Well, actually…

1. Okay, but how many personas does each of them have?
2. …and all he cares about is scope, schedule, and budget.

Productivity != effectiveness in delivering value

1. Our manager says she’s focused on business value, customer satisfaction, and product quality; but all she measures is productivity: Quantity of output per unit time. So, we generate as much crap as we possibly can.
2. Everyone needs to be 100% utilized at all times. If anyone is ever idle, it’s a waste of money.

Well, actually…

1. “Tell me how you will measure me, and I will tell you how I will behave.” (Goldratt)
2. True, if your corporate mission statement is “Keep all employees busy all the time, even at the expense of throughput.”

“Do the simplest thing that could possibly work” != build the easiest part of the solution first, then change jobs

1. There’s no need to handle exceptions, as that would make the code less simple.
2. If we keep delaying implementation of the challenging parts of the solution, maybe they’ll cancel the project before we need to worry about it.
3. If John spends 20% of his time at work searching for a new job, he might not find one before we get to the complicated part of the solution; if he spends 80% of his time searching for a new job, he can find one before things get too difficult around here.

Well, actually…

1. Obviously.
2. Hope is not a plan.
3. That’s a great example of the 80/20 rule. I hope John is a good multi-tasker.

Test-Driven Development != manually testing the whole application after the fact, or maybe just thinking about testing it, or maybe just hoping someone else will do it

1. I use test-driven development: After the users test my code in production, I drive back to work to fix the bugs.
2. I’m “test infected.” Whenever someone suggests I test my own code, I start to feel sick.
3. Test-driven development is just a cheap knock-off of design-by-contract.
4. I’m a socially conscious programmer. I don’t want to throw all those testers out of work. They have families to feed.

Well, actually…

1. Wonder if you’ll upgrade your software development practices once the price of gasoline hits $5.00 per gallon.
2. Funny thing, whenever you’re out sick the rate of technical debt accumulation declines.
3. Maybe so, but have you ever tried to enforce a verbal contract?
4. All hail Saint Bugger, The Prolific.

“Best Practice” != an excuse to stop thinking about improvement

1. I learned best practices in school. That means I already use the best possible practices.
2. I read about best practices on the Inter-Web. That means I already use the best practices currently known.
3. I’m an Ace Programmer. If I don’t already do it, it isn’t a best practice.
4. Your best practices might not be my best practices.

Well, actually…

1. You mean, you already use the best practices you know of.
2. You mean, you already use the best practices you know of.
3. You mean, you already use the best practices you know of.
4. You mean, you already use the best practices you know of.

My growing respect for him may be the reason I was struck by a comment of his that I came across recently while browsing the site. In a response to a reader’s comment dated January 7, 2008, he wrote: "Pair programming is antithetical to Lean" It was just a flat assertion with no explanation.

I was puzzled. Lean software development doesn’t speak to particular development practices, as far as I know. What might cause a practice to be antithetical to lean or, for that matter, to support lean? The only basis I could think of on which to reach such a conclusion was that the practice either hinders or helps us in applying lean principles to software development. Corey did not provide the same level of careful analysis and explanation as he does with most of the useful material on his site, so I decided to reason through the question myself.

Lean priorities and the three Ms

To be sure I shifted my brain into the right gear before exploring the question, I began by reviewing some key principles of lean software development. There is a widely-quoted statement from David Anderson that goes something like this:

Value trumps flow
Flow trumps waste reduction
Eliminate waste to improve efficiency

Unlike a manufacturing process, any given step in a creative process will run at varying speeds. When we adapt concepts from lean manufacturing to creative endeavors such as software development, we must take this into account. In a software development process, the priorities David stated might be realized by maintaining buffers of incomplete work items between value-add activities, so that value-add work won’t come to a halt when an upstream step slows down. By the same token, WIP limits prevent overfilling a buffer when an upstream step speeds up.

By definition, the buffers contain inventory, and by definition, inventory is waste. We intentionally keep a certain amount of inventory in buffers in order to maintain continuous flow, which is of higher priority than waste elimination. People sometimes use the drum-buffer-rope metaphor to describe this.

Mura and induced work

Mura generally means unevenness, inconsistency, or irrregularity. In the context of manufacturing, it refers to variation in units produced on a production line. In the context of software development, it means uneven work flow. Uneven flow consists of stopping and starting, slowing down and speeding up, waiting, rework and defect correction.

In my own work, I’ve observed that mura leads to muri and muda. From conversations with others who practice lean software development, I gather that I’m not the only one to make this observation. I call mura a First Domino — a problem that tends to lead to further problems. Any time we can prevent a First Domino from falling, we prevent all the downstream problems it would have caused.

So, it suits my inherently lazy nature to identify First Dominoes and prevent them from falling, as it relieves me of the need to deal with a multitude of secondary issues later on. The notion of eliminating unnecessary activity is also consistent with lean thinking. Maybe that’s one of the reasons I appreciate lean thinking.

Examples:

1. Waiting for someone to become available to answer a question, to provide insight into a difficult problem, or to review a piece of work.
2. Defect correction due to trivial oversights or mistakes in programming.
3. Rework due to misunderstanding needed functionality, poorly factored code, or misalignment with architectural standards/guidelines.
4. Stopping work on a value-add activity in order to address rework or defect correction (opportunity cost).

Alan Shalloway coined the term induced work to describe unnecessary work that we create for ourselves. We cause mura when we organize our work in certain ways or when we choose certain methods or techniques to carry out specific tasks. We cause stopping and starting when we choose to have too much work in process and we ask individuals to multitask across many concurrent initiatives. Our work slows down and speeds up when there is significant inherent variance in the level of difficulty of different tasks. We can cause this (or exacerbate it) based on the way we decompose work into discrete tasks. When a task is carried out by an individual or work group lacking the necessary expertise to complete it independently, we cause waiting, as the person with the answer is often in high demand and is not available immediately. We cause rework when we believe we understand what is to be done and we complete a task accordingly, but our understanding was incorrect. We cause defect correction when we overlook something in our work; usually, something minor that simply fails to catch our attention in the moment.

I observe that mura induces work by creating muda and muri:

1. Induced muda: Rework, waiting, defect correction.
2. Induced muri: Overtime or work speed-up to try and meet original commitments on top of induced muda. Induced muri can lead to still more induced muda, as people tend to make more mistakes when they rush or when they are tired.

What effects (if any) does pair programming have on induced work?

Let’s see what the general effects of pair programming are, and then we can consider how those effects relate to the idea of mura and induced work.

Dr Laurie Williams and Dr Alistair Cockburn investigated the effects of pair programming, and reported their findings in The costs and benefits of pair programming (PDF), presented at the XP 2000 conference in Sardinia and published in the 2001 book, Extreme Programming Examined, edited by Giancarlo Succi. In exploring "eight paths of software engineering and organizational effectiveness," the authors found that "all paths point to pair programming." The eight paths were economic, satisfaction, design quality, continuous reviews, problem solving, learning, team building and communication, and staff and project management.

The investigators found this result "surprising." That comment may seem insignificant, but I think it contains a clue about the reasons why many people assume pair programming is just some sort of game (and I use the word assume quite intentionally). The idea is counterintuitive. Even to people whose professional focus has been to study the effectiveness of software development processes, methods, and practices, the result was surprising.

Among other results, the authors summarize the outcome of a controlled experiment in pair programming Dr Williams conducted at the University of Utah in 1999. The experiment found that pair programming increased total development time by about 15% (not double, as one would intuitively expect), and reduced defects by about 15%. According to the paper, using conventional methods "programmers inject 100 defects per thousand lines of code. A thorough development process removes approximately 70% of these defects. Therefore, the individuals would be expected to have 1,500 defects remaining in their program; collaborators would have 15% less or 1,275 – 225 less defects. [...] Using a fairly conservative factor of 10 hours/defect, if testing finds these ‘extra’ 225 defects they will spend 2,250 hours – fifteen times more than the collaborators ‘extra’ 150 hours."

To put the result in the context of lean thinking, one could conclude pair programming can reduce lead times by 2,100 hours, given the same general conditions as the experiment. Whenever we are considering the cost of rework or defect correction, we want to consider opportunity cost, as well. Each hour spent in rework or defect correction is an hour forever lost to value-add work. In that sense, the cost of the 2,100 hours of defect correction is really equivalent to 4,200 hours.

An earlier study of pair programming was conducted by Dr Randall Jensen in 1975. Writing in Crosstalk: The Journal of Defense Software Engineering for March, 2003, he explained, "I was introduced to teamwork and pair programming indirectly as an undergraduate electrical engineering student in the 1950s. Later in 1975, I was asked to improve programmer productivity in a large software organization. The undergraduate experience led me to an experiment in pair programming."

The term "pair programming" had not yet been coined at that time, and the investigators used the term, "two-person team" to describe the technique. The way they arranged the work environment for the two-person teams would be recognized immediately by today’s software developers as a pair programming set-up. The 1975 study found that a two-person team generated 300% fewer defects than individual programmers developing the same solutions. Dr Jensen commented that "a three order-of-magnitude improvement in error rate is hard to ignore." It seems he underestimated people’s capacity to ignore things that don’t correspond with their preconceptions.

False positives?

To date, there have been very few original and well-crafted studies of the effectiveness of pair programming. One can find many published papers, but most of them are not reports of original research, but surveys of existing literature. The two studies cited here are the only ones I know of, personally, that provide credible experimental evidence about pair programming. Therefore, I wanted to look for some other form of evidence that close collaboration may be a useful technique. The anecdotal evidence from programmers is mixed; some report positive experiences and others report negative experiences.

It turns out that close collaboration — what we might call "pairing" in a general sense — works well when certain conditions are met. When those conditions are not met, then there is no benefit in pairing, and it may even yield poorer outcomes than solo work. For example, in a paper entitled "Optimally Interacting Minds," published in Science for August 27, 2010, and summarized in a piece on Discovery magazine’s website, investigators reported the results of a controlled experiment that explored the question of whether two people working in collaboration completed various tasks more effectively than people working alone. They found:

For two observers of nearly equal visual sensitivity, two heads were definitely better than one, provided they were given the opportunity to communicate freely, even in the absence of any feedback about decision outcomes. But for observers with very different visual sensitivities, two heads were actually worse than the better one. These seemingly discrepant patterns of group behavior can be explained by a model in which two heads are Bayes optimal under the assumption that individuals accurately communicate their level of confidence on every trial.

In the context of pair programming, this result suggests that the effectiveness of pairing depends strongly on the way in which the two programmers interact with one another, and on their self-awareness regarding their own skills. I see a connection here with observations I have made in the past regarding our lack of a professional corpus of knowledge that we transmit from generation to generation of software developers. Each generation tends to re-invent every wheel for themselves. (See "Can a new dog learn old tricks?" on this blog.) Software development seems to take place in Lake Wobegon, where "all the women are strong, all the men are good-looking, and all the children are above average." Most of the above average children stand on the bottom rung of the Conscious Competence Ladder. If they try pair programming and they have difficulty with it, they may not be equipped to understand why, and simply conclude that pair programming "doesn’t work."

If the value of close collaboration in technical work has been recognized at least since the 1950s, and if pair programming specifically was vetted experimentally as long ago as 1975, then why do people in the software development field still argue about it today? Looking back over the history of software development methods and techniques, I am struck by a recurring pattern: Generation after generation, various development techniques are discovered and forgotten, discovered and forgotten, discovered and forgotten over and over again. Each re-discovery re-brands the technique, and when a competing methodology dares to suggest the same technique with a different brand, the two camps attack one another with the same intensity as two closely-related sects of the same religion.

The 2007 post in which Corey made the comment about pair programming was, in fact, a rant favoring one sect over a competing, nearly identical sect within the religion of effective software development practices. It was not representative of his usual objective, analytical approach. The assertion that pair programming is antithetical to lean does not withstand an objective assessment of the effect of the practice on lean-defined forms of waste. It is an emotional statement. And, in fairness, the comment was published several years ago. The author’s understanding may have changed since that time.

Why is it that smart people can resort to statements of this kind? It may just be a feature of human nature. We all have blind spots. I appreciate Corey’s work and that of other people from whom I can learn. I hope I can also help people learn. But we would be wise to avoid just accepting any statement a person makes, even if we respect that person highly. It’s better to take what they say and run it through our own filters; reason through it independently, or try what they suggest and see whether it works for us. Then, whether the outcome is good or bad, we should try and understand why and how the outcome came about, rather than just accepting or rejecting a practice universally.

Conclusion: Pair programming is a natural fit for lean development

The effects of pair programming include reducing wait time and reducing errors. Cross-disciplinary pairing also enhances clear communication and common understanding of what needs to be done, reducing rework. Pair programming, and pairing in general, are therefore perfectly aligned with lean thinking and lean software development. They directly reduce mura, which is a First Domino, and by extension a good deal of muda and muri as well. They also promote double-loop learning by cross-pollinating the knowledge and experiences of team members, helping to support the lean concept of continuous improvement (pursuit of perfection, the fifth lean value).