Four canards that afflict us

Alert: 82% chance of pontification.

Attitudes toward continuous improvement

Question: Is it a general pattern that organizations in need of improvement tend to be satisfied with the status quo, while organizations that pay attention to improvement tend to forget that they already do many things very well?

Here are sanitized descriptions of four client environments where I’ve worked in the past few years.

Case 1: Cumbersome traditional process developed in “ivory tower” fashion by two senior managers. Large, laminated posters of the process are displayed prominently on the walls. Those managers cannot tolerate any suggestion that the process could be improved, or that staff are not actually following the process as it is specified. If everyone followed the process to the letter, no work would ever be completed. Staff would be shackled into a permanent interlocking set of wait states, queued up to review and approve each other’s interim documentation artifacts. The formal process includes numerous redundant reviews of documents, quality gates, planned back-flows for defect correction, and so forth. Of the ten-month release cycle, about six weeks is spent doing any real work.

At ground level, each work group or team has crafted ways to get around the process while reporting to management that they have followed specific process steps. No two teams or work groups operate in the same way. A handful of personnel can be credited with enabling most of the work to get done. They are working 80+ hours per week all year long. The rest of the staff of some 800 people are extremely busy for short periods of time during the 10-month release cycle, and otherwise go through the motions of holding meetings and reviewing documents. A typical work day for most staff members consists of 8 or 9 hours of meetings, and nothing else. Working there is like being trapped in the Hypercube.

Case 2: Self-styled “agile” shop comprising some 400-500 people that, at first glance, appears to have implemented first-generation “agile” methods successfully. They are quick to pat themselves on the back for being an exemplary “agile” environment. A deeper dive into the organization reveals significant dysfunction in “agile” mindset, and a well-entrenched culture of pretense. Everything about people’s words and behavior is fake. It is not politically safe for people to suggest the organization might not be fully and deeply “agile” in every way. They don’t bother with retrospectives, ostensibly because they are already perfect, but perhaps more realistically because no one would dare suggest anything needs improvement.

In practice, the organization is highly traditional in mindset and staff feel “beaten down.” People have learned to affect the required smile, chant the required mantras, and tolerate the never-ending Death March that is their working life. Management likes to boast about how “agile” they are while actually practicing Theory X management, with the X’s embedded in staff members’ backs like shuriken. They consider themselves masters of the art of software delivery, with nothing left to learn from anyone else.

Case 3: An organization that has experimented with ideas from various schools of thought, including all the various management frameworks and process models that have come along over the past few decades. They have adopted methods and practices that work well for them in their business context. They have a formalized continuous improvement process that they take seriously, and that they use to experiment with new ideas on an ongoing basis. They do a good job of delivering high-quality software, although they have their fair share of monolithic legacy systems to support, and they are constantly on the look-out for potential improvements.

Case 4: An IT shop that a few years ago reached a nadir of delivery performance, morale, and cost management, and undertook a “big bang” transformation into “agile” methods. They have turned things around very nicely and are currently delivering high-quality software in a sustainable way. They have exceptionally good morale and have internalized “agile” principles honestly. They take continuous improvement seriously and frequently try different ideas to see how they might offer benefit. The present focus of their improvement effort is to move from a one-month release cycle to a continuous delivery model.

I’ve noticed something interesting (to me, anyway) about these four organizations. I will suggest the four represent a broader pattern in industry, as well. The interesting thing is this: The two organizations that are most proud of their current state do an extremely poor job of delivery, and have the least awareness that they need to improve; meanwhile, the two organizations that are most aware of the importance of continuous improvement are pretty much unaware of how well they are performing already. The two that least deserve to feel proud of themselves feel great pride in themselves; the two that actually deserve to feel pride see themselves as poor performers. Note that this is completely independent of the methods they use – there is no connection with “agile” or “lean” or anything of the kind.

In the most dysfunctional organizations, people walk around proclaiming that they are the best IT shop on the planet. In the best-performing organizations, people walk around saying things like “We suck” and “We’re highly dysfunctional.”

Psychologists, help me understand this phenomenon.

TDD and software archaeology

Can you tell by looking at the unit test cases whether the code’s author wrote the tests first? During a TDD class I taught last week, one of the participants suggested that you can’t tell. Completed code and unit tests would look the same regardless of which had been written first. At the time I didn’t give the comment much thought.

I returned to the team I was working with at another client on Thursday, the last day of their iteration cycle. I picked up a small story so I could contribute something before the end of the day. It was a bug fix for JavaScript input validation logic in a webapp. Like many applications, this one uses date ranges to activate and deactivate certain data values that drive functionality. In this case, the input validation function neglected to ensure the expiration date was later than the effective date.

My first stop was the Jasmine unit test cases. Sure enough, there was no test case expecting the expiration date to be later than the effective date. I added three cases — effective date earlier than expiration date, effective date later than expiration date, and both dates the same. It was straightforward to change the date validation function to make the tests pass. Fortunately the code is very clean, so there was no induced work due to design debt.

The student’s observation may be correct in some cases, but I think in general we can tell whether the tests were written before the code by looking at the tests, the code, or both. In this case, the Jasmine tests told me the code had been written first. I didn’t have to look at the production code to know that. The tell-tales were:

1. There are no happy path cases. If the code had been driven by the tests — that is, if the test cases had described the functionality desired in the code, and then the code had been written to make that happen — then all the functionality would have been described in test cases, and not only the exceptions. Code driven by this set of test cases would never submit the input form, because there is no test case that drives that behavior. Based strictly on the test cases, the production code should do nothing at all except report invalid input values. Yet, this is not true — in production, the input form is submitted and the transaction is processed. Someone wrote the code to do that without reference to any test cases.
2. In every single test case, the message Jasmine displays when the case fails reads, “it returns false.” Had the code been driven by the test cases, this would mean the business purpose of each and every JavaScript function is to “return true.” Clearly, this is in no sense a “business purpose.” It’s hard to imagine a business stakeholder demanding that the software should “return true.” The software archaeologist in me suspects the author of the tests read through the JavaScript functions that already existed and wrote the test cases in terms of their return values. The test cases describe implementation rather than behavior. When code is driven by tests, the tests describe behavior.
3. The nature of the defect suggests the original author wrote the code first and the tests second. With his/her focus on the individual input fields, it would be easy for the author to forget to write validation logic that involved multiple input values. By describing behavior through test cases first, it would be hard to forget such a basic date range validation.

In this case, it was easy to see that the code had been written first by looking just at the unit tests. Writing the code first can also have a visible effect on the production code. Pairing on the same client’s back-end Java code recently, a team mate and I spent two days on a modification that should have taken no more than two hours. The code we had to work with had the following characteristics:

1. Class with multiple responsibilities
2. Long method
3. Method with multiple functions
4. Method operating at multiple levels of abstraction
5. Reliance on side effects
6. Tight coupling
7. Embedded calls to static methods
8. Deeply nested if/else structure with redundant tests of the same elements
9. Hard-coded instantiation of collaborators
10. Inconsistent naming conventions
11. Duplicate and nearly duplicate code
12. Stale comments
13. Poor use of language’s type system — everything is a string

It’s unlikely the test cases were written before the production code because it would be very hard to come up with a concise test case whose simplest possible implementation is a massive hairball. Test-driving tends to result in a larger number small methods that perform just one function each, rather than a smaller number of large methods that perform multiple functions. You can often tell by looking at the code whether the tests were written before or after the production code.

Delivering provably-correct code

A while ago on this blog I mentioned the idea that a programmer’s job is not to deliver code, but to deliver provably correct code. Twitter responses ranged from strongly negative (wrong on so many levels I don’t know where to start) to mildly negative (not completely wrong, but incompletely right). There were no positive responses.

The converse of deliver provably correct code is deliver crap code, as I read it. Yet, it seems unlikely that is what those people meant to say. Unfortunately, no one followed up to explain their interpretation. I can only speculate that my choice of words has different implications for them than it does for me.

Since no one wants to explain their interpretation of the phrase, I’ll explain mine.

Lean Kanban France 2012 will take place 18-19 October

Lean Kanban France 2012 is coming up in a couple of weeks. The conference will be hosted on 18-19 October by Valtech at their offices in the heart of Paris.

I’m honored to be included on the program of an event that features so many luminaries in the field. My small contribution will be a session entitled Structural Impediments to Lean. The premise is that the conventional organizational structures and basic assumptions regarding roles and procedures tend to present obstacles to the effective use of Lean thinking in companies. It’s a subject I started to explore in earnest a couple of years ago at Lean Kanban Europe 2010 and have continued to explore in my work since then.

Here is the session description from the program.

Cats in the monasteries of IT

Paulo Coelho wrote a nice piece about how cats came to be regarded as necessary for meditation for a few generations in Japan. The story might resonate with just about anyone in just about any context.

In my case, it reminded me of longstanding assumptions about how things “must” be done in the IT field. The following quote sums up the state of corporate IT in a nutshell: “..society continues to create some systems which, in the fullness of time, lose their reason for existence, but continue to impose their rules.”

This is how the assumptions, standards, and all the certifiable things in our field originated:

A cat
once sat
right at
the master’s side during meditation

a copy-cat
got a cat
hoping that
it knew the path to wisdom

people sat
other cats
on the mats
in the monasteries of the nation

pilgrims wandered
scholars pondered
and in academic papers laundered
a cuddly theoretical formulation

now supported by science
a feline appliance
became a fundamental reliance
for navel contemplation

Does remote work work?

First, here’s the short version for those poor souls suffering from tl;dr (too lazy, don’t read much) syndrome, that peculiar malaise that characterizes our times:

Can working from home be effective
compared with collocated teams?
Opponents are quick with invective
and full of opinions, it seems.

But what if we increased, in some way,
the ratio of signal to noise?
Could we discover a good way to
routinely deliver with poise?

And now to business.

One of the ongoing debates in the IT world over the past few years has been about the relative merits of team collocation, including intense collaboration, paired work, and continuous osmotic communication, versus solo work, including work from home and other forms of remote work as well as office spaces fitted with individual cubicles.

In my own experience, both forms of work have been effective for delivering product and both have provided significant personal and professional satisfaction. On the other hand, I have seen both forms of work result in frustration, stress, anger, waste, and poor outcomes.

If both forms of work have been proven effective, and at the same time both forms of work have been proven problematic, then it seems logical to conclude neither approach guarantees positive outcomes. There must be factors that enable one approach or the other to be effective under certain conditions.

Rather than asking whether team collocation is universally superior to solo work or vice versa, we might do well to ask which conditions call for one or the other approach. If we can understand that, we stand a chance of organizing teams in ways that maximize the strengths of each approach in context.