Saturday, February 02, 2008

Check. Check. Check.

In the mid-1930s the US Air Force was on the hunt for a long range bomber. They had very specific requirements about how far the 'plane would need to be able to travel, how fast it could fly and the size of payload it could carry. There was plenty of competition - from Boeing, Martin (now Lockheed Martin) and Douglas (McDonnell Douglas). Boeing's entry into the show was code-named the "Model 299."; it could fly twice as far and twice as fast as the others as well as carry up to 5 times more payload but it was also expensive. It looked like an easy choice. The top brass were summoned to a flight test in October 1935. The Model 299 taxied, took off, climbed and promptly nose-dived into the ground, killing some of the crew. That, on the face of it, was the end of that and the Model 299 could have been no more.

The crash investigation that followed determined that the pilot had forgotten to disengage a control, known as the "gust lock"; this was supposed to be locked in place when on the ground and disengaged before take-off. The Model 299 was probably the most complicated 'plane ever produced.

The Model 299 had its fans - and they were reluctant to see that as the end of the runway for such a 'plane. They got together and tried to figure out what to do. I imagine that they had a long list: they could go back to the design and try and simplify the controls, maybe they could eliminate the gust lock, they could have proposed significantly more training for potential pilots before they were allowed to take the air. They proposed none of those.

What did these engineers, test pilots and management propose?

A checklist. That's right - a pre-flight list of checks that had to be carried out and manually ticked off before the 'plane should being its taxi.

The introduction of this, on the face of it, very simple device allowed the Model 299 to come back to life. It became the B-17 bomber, known as the "Flying Fortress". Arguably, at a time when the Royal Air Force had no such long range bombing capability, this plane (and its descendants) helped win the Second World War - there were nearly 5,000 in operation by the end of the war.

There is, of course, lots more to this story - and plenty of online sources for information, such as Wikipedia, and the History of War.

As we forward wind 70 years, the check list has a vital and necessary role in day to day activities - especially in the complex field of technology delivery and implementation. And yet, I believe that it is tragically under-utilised because today's technicians think that they have it all in their heads and have no need of such an apparently basic tool. After all, when we get in a car and go to drive away, we don't use such a list; but, before we go on holiday, I'm pretty sure that most of us have a little list somewhere to make sure we don't forget our passports, our foreign currency, our flight confirmation, the address of the hotel and so on. Perhaps a checklist is only necessary for something we do rarely?

Medical evidence would seem to dispute that - even for the most basic, most regularly performed tasks, a checklist can make a significant and life-saving difference.

The work of Peter Pronovost demonstrates this. Pronovost is the Medical Director of the Center for Innovation in Quality Patient Care and an Assistant Professor in the Department of Anesthesiology/Critical Care Medicine at Johns Hopkins University's School of Medicine. Why? He introduced checklists into the day to day medical process in more than 70 hospitals in the USA (involving 110 ICUs). His initial premise was simple - mistakes happen so how do we prevent them?

For instance

In the United States alone, 7 percent of patients in academic medical centers experience a mistake with their medication resulting in up to 98,000 deaths a year. Those numbers are mirrored in Australia and the United Kingdom.

Hospitals that have implemented his checklists have seen dramatic drops in infection rates - saving hundreds of lives per year. Some of his checklists are surprisingly simple - yet they save lives. For instance, before inserting a line: wash your hands, use full barrier precautions (cover the patient), clean the insert site with chlorohexadin, avoid the femoral area for insertion. With this simple process, infection rates have dropped by 60% or more - some have seen 6 months or even a year with no infections. The hospitals involved have saved over $170 million and - more importantly - 1,500 lives.

So why am I writing about this here?

In many roles over the last few years both in and out of government, I've seen implementations delayed, projects fail, IT disasters occur and systems fall over because of basic errors. We have an incredibly complicated world of technology where even a small organisation has a myriad of different systems and applications installed. Big organisations have a full suite of operating systems and off the shelf packages, hooked together with miles of bespoke code. Those who like to live at the bleeding edge have complicated server utilisation models using virtualisation at CPU and storage level.

And yet, manual operation, manual installation and poor documentation seem to be the norm. "Finger trouble" is heard once a day in many projects and even in live systems as a reason for a delay or an outage.

In the medical environment, you can often rely on the team, whether it's a nurse or another doctor, to notice an error and correct it - were someone to die, "finger trouble" inevitably leads to a lawsuit.

In the technical world, where it often seems to fall to individual experts to get things done (you know the story, you have a supposedly leveraged team with masses of expertise available whenever you call but, in the end, there's only one guy or girl who knows your system's intricacies sufficiently well to get the job done), check before do would seem to be a sensible model.

Measure twice, cut once has, after all, been true for centuries.

(This post was inspired by an article I read in the New Yorker magazine which I subsequently couldn't find, so I rewrote it using online sources).

2 comments:

  1. ah! a favourite of mine measure twice cut one, perhaps this is one of culture, IT workers are referred to as consultants not engineers and good engineering practises are rarely understood or adopted. The Companies delivering these projects are not engineering companies; they are consultants, Systems Integrators etc. Is it that engineering is seen as a dirty industry associated with metal bashing, or that engineers these days are merely (kwik fit) fitters. Engineering applies to many disciplines, from building a bridge to implementing systems.

    ReplyDelete
  2. always better to be a consultant and be consulted than an engineer and actually have to engineer. it's think versus do isn't it. the dyson chap has very strong views on this which i can't help but agree with - of course he then moved all of his assembly work to malaysia or similar.

    ReplyDelete