Sunday, May 20, 2007

Beta Blockers

What is it about government IT projects that make them fail so spectacularly and so regularly? Only this week Joe Harley, the CIO at DWP, was lamenting that the failure rate is 70%. Speaking at the Government IT summit, he said"For the government's IT spend of £14bn, "7,000 primary schools every year could be constructed", about 600,000 nurses could be employed or more than three million state pensions paid." I love that the number is still £14 billion - I calculated it at £13 billion or so some 4 years ago and wondered how much it would climb to once the NHS IT and the Home Office ID cards got going. Still I digress. I love the other numbers too - 600,000 nurses at £23,333 a year, 7,000 primary schools at £2,000,000 or 3,000,000 pensions at £4,666 per year. Eye catching numbers, if disturbingly round. Joe notes that the £14 billion isn't all project-related of course, there is a bundle spent on desktops - with a spend of anywhere from £700/unit to £2,400/unit (and, I imagine, far higher costs depending on who loads what into each unit price - one of the problems of comparing such numbers, like the "cost of a website" is different departments see costs in different ways". On the project front, Joe isn't entirely clear what "failure" actually means but he did add: Projects fail for a variety of reasons but sometimes have predictable weaknesses such as inadequate requirements. Even if a project or programme is three months late it's doubtful "you could call this a success". The "inadequate requirements" point is interesting. With OGC Gate Reviews either about to be or now made public, I'm sure we're going to see a lot more on that. OGC are already facing criticism for passing too many projects with status "green" that turn out later to be outright failures (otherwise known as the "how could you not spot that the horse they were putting together had 3 humps?" attack) and senior civil service management will likely take flak for ignoring too many "red" reports (known as the "what do they know about horses? ours needs those 3 humps for exta ballast" defence). On the current trackrecord, I think most of us would be pleased if some thing was only 3 months late, depending on what occurred because of the lateness. Obviously, a missile intercepting an incoming nuclear warhead being fired 3 months late wouldn't be much use; and an IT system designed to ship patient data around wouldn't do much good for the patients that passed away during the 3 months that it wasn't around - but, in general, a 3 month delay wouldn't be terrible. I'm intrigued by the 30% success rate figure. If success takes just 3 dimensions - meets requirements, is on time and on budget, then I'd venture that no government or business entity in the world has a hit rate of much greater than this figure. Filtering out the very short-term projects - the ones that are changes to existing code bases or the ones where time isn't costed to the project, so establishing say a minimum spend of £10 million (small by government standards, moderate by corporate standards), how many companies would stand up and present a success rate much greater? Of course, the deck is loaded here. The Chaos Chronicles, published by the Standish Group, tell us the real story: These numbers are a significant improvement over the previous survey conducted in 1995. That survey indicated that 80% of IT project failed. In 2003 this number shrank to 66%. Although fewer projects fail nowadays, the general trend is that more projects are delivered late; statistically this is at 82%, up from 62% in 1995. However, there is also good news: the average cost overrun is currently at 43%, down from 180% in 1995 The true genius lies in the solution The most effective way to avoid cost and schedule overruns is to get better at making software cost estimates. Software has become more complex and increased in size, which makes estimation more challenging, however, several studies (e.g. Standish Group, Capers Jones) have shown that by using software cost estimation techniques alone, the probability of completing a project successfully doubles. Estimating the schedule, cost, and resources needed for the project is paramount for project success. That is to say that if you doubled your estimate of time to complete, increased your budget by 75% and cut your requirements in half (all at the same time), you'd be absolutely certain to be successful. Sadly, more current data than 2003 doesn't seem to be available on the web (indeed, the Standish Group website lists data only from 1994 and they want $5,000 for membership or $500 for just one quarterly report). I've done more than a few presentations on the same topic, usually involving a story about wheels falling off wagons and using whatever was current from the press at the time, whether it was problems at Sainsbury, Nike, Cigna, the Inland Revenue the Department of Work and Pensions or wherever. Sometimes I ended with this quote, from, I think, an anonymous source: "A carelessly planned project takes three times longer to complete than expected A carefully planned project takes only twice as long" This week I've been playing* Microsoft's new Xbox 360 game, Halo 3. The whole game isn't out until the end of September but Microsoft's developer, Bungie, have thoughtfully releases a beta version. I have no idea how many people in total are accessing this beta but earlier this evening over 50,000 were online playing at the same time as me (not, fortunately, all in the same game universe as me). One way into the beta was to buy a specially flagged copy of another game, Crackdown, for about £40. Some people treated this as buying the game and getting the beta for nothing, others complained about having to pay £40 for a ticket to the beta (which only lasts three weeks or so) and planned to throw away what turned out, not surprisingly as the game was written by the author of the original Grand Theft Auto, to be a very good game. When the beta kicked off, last Wednesday, those who had bought Crackdown anxiously awaited for it to begin at 1pm on the dot. Come that time, no one could access it. For hours afterwards, access was denied, and the Interweb forums filled with vitriol, bile and every known variety of swearwords, all helpfully, on bungie.net at least, replaced by -blam-. The user population was not happy. Bungie acknowledged that there was a problem after an hour or so and then stayed quiet for ten more hours or more. In fact, all of the companies involved stayed quiet. Only when it was fixed did they announce the problem had gone away and, gracefully, extended the beta for 4 days to make up for the 1/2 day or so lost. A good recovery (and isn't the recovery from a problem so much more important than the actual problem itself?). There are a few points here 1) Betas are meant to be buggy. That's why there's a beta, to iron out the problems. There aren't, usually as far as I know anyway, problems actually starting the beta, but you'd certainly expect the game to be buggy (And so far, it seems pretty stable but there are certainly bugs) 2) Staying quiet once you know there are problems, perhaps known as the "Gordon Brown on Raiding Pensions" defence, isn't a good tactic. It angers people more. 3) When you do find problems, addressing them in the right way is critical. Just like when you go to a restaurant and they make you wait 60 mins or more for the main course, if they bring you a glass of fine red wine, you wait patiently; if they don't, you fume quietly in the corner and don't leave any service charge. SO what would happen if government ran beta tests of its major programmes? After all, we're seeing them with the cashless society (being beta tested in September in Canary Wharf) and the Downing Street petitions website continues to be in beta, despite long since proving its mettle with the road pricing petition. In the corporate world, half of the things you can do with Google seem to have been in beta for months or years (hasn't gmail only recently come out of beta after all?). But, we haven't ever seen one start or be proposed for, say, tax credits, or business taxation policies, or, shall we say, ID cards? So how about we run some beta tests on government programmes, using these steps: - Carry out limited beta tests, backed by sufficient but not too much IT to prove that the theory and practice worked. It doesn't have to do everything on day one. - Try out 2, 3 or even 5 different versions of the beta - i.e. different policies. One would have to be a control group - just like in medical tests, you need to be able to test against something that you understand and that isn't moving. - Invite a limited population - somewhere from 0.1% to 5% of the total target market (with 30 million employees in the UK, so a PAYE betea with 0.1% would be 30,000 people - plenty of a challenge). - Make the aims of the beta, the kind of people involved in it, and the details of how it's going including the final conclusions, entirely public - Carry out the beta test for long enough to see how it worked through a full cycle. If that was PAYE, i.e. employee taxation through the payroll, that could be as much as a year. - Start development of the wider programme only when all the tests had been evaluated against the original aims. This works for drug tests, it works for peer reviews of scientific papers, so it should work for government IT I know, however, that the moment the first one of these was started, the business and technology press would erupt. The opposition parties would also have a field day, whichever set they were. The obstacles raised would include: - What a waste of time. If you're not sure of your policy, why not get sure in the first place and then start the work - How dare you give these other people competitive advantage. Some of these new tax beta tests would doubtles involve companies paying less tax so that the impact on their profits, investment profile, employee count and so on could be assessed. Instead, all of these kinds of tests are carried out on complex models of the economoy built, doubtless, in Excel - What about those people who are suffering without access to the beta? This is a variant of the "how dare you give the people in Ward A the drug that might cure them of their cancer, but not the folks in Ward B" - at the outset, of course, you don't know which is going to work best. - What about if the beta fails? How do you get people off the new process and back onto the old process without disadvantaging them? I could go on for ten more pages about the list of objections that would be raised. You'll note also that I am proposing beta testing both policy and IT systems - addressing the "inadequate requirements" point in the beta test is surely essential - and so opening myself up to double criticism. The Halo 3 guys are only testing their IT - and they know that Halo 3 is just Halo 3, they don't have to test out if they need to make it more like Half Life 2 or more like Gran Turismo or whatever. Nope, there would be many who would block the idea of beta testings, let's call them the beta blockers (predictably, a group of drugs that slows down your heart rate - and therefore, doubtless, your speed of delivery). And that's probably a shame. If the success rate really is only 30% then savaging the costs by 20% and working on "inadequate requirements" at a purely technical level isn't going to solve the problem. Not for UK government, not for foreign governments and not for corporations around the world who appear to suffer from exactly the same problem. * Yes, I play video games. I've been playing since Space Invaders in, what, 1978. Then all the way through a 256 byte (yes, byte) Mk14 through the Apple II, the ZX81 and on to the Atari ST, the BBC Micro, the Xbox and now the Xbox 360.

8 comments:

  1. Anonymous6:35 am

    Now isn't that strange, I was in the audience when the keynote speaker said quite the opposite.

    I don't recall DWP asking any questions or making any comment; perhaps a case of paper tigers.

    Proposition was that there are a lot of good projects that don't make the news, so this belief sustains.

    I'll ask DWP about this, at the next plenary

    ReplyDelete
  2. ... he said "quite the opposite" of what? that the failure rate is 70%? or that a project 3 months late is a failure?

    i pulled the quotes out of the sites i linked to; and wasn't in the audience. still, would hardly be the first time that a senior figure, or any figure, had been misquoted.

    if his angle is that the failure rate only appears to be 70% because the projects that go well don't make it into the stats, I'd understand that. I'd say it was spin, but I'd understand it. The annual PAYE or SA releases that HMRC pull off every year, like clockwork, never make it into the news - they always work, and they're amongst the biggest releases they do; I imagine the same is true with Pension Credits and many other DWP programme updates.

    But the failure rate on "big programmes" - passports, tax credits etc - the ones that grab the news anyway, is where I thought he was heading.

    happy to be corrected.

    ReplyDelete
  3. I agree with your comments about Joe Harley’s speech. I don’t agree with your commentator who said he was “in the audience when the keynote speaker [Joe Harley] said quite the opposite”.

    I too was in the audience, and I wrote up Joe Harley’s comments for Computer Weekly. To avoid any confusion I have published, unedited, the relevant section of transcript of Mr Harley’s speech - http://www.computerweekly.com/blogs/tony_collins/2007/05/joe-harley-cio-at-the-departme-1.html#more

    Tony Collins

    ReplyDelete
  4. Ian C2:39 pm

    What everyone seems to forget to mention here was that target set for 'successful Govt IT projects' was 70% - and was meant to be hit by. errrm, March '06. Ooops.

    ReplyDelete
  5. Anonymous11:06 pm

    Sorry, different conference then, John Suffolk was the keynote, day after the Whatmore accenture chappie.

    DWP said absolutely nothing.

    Sorry to confuse you all.

    ReplyDelete
  6. Tony - thank you for stopping by and providing the link to the transcript.

    Anon - thanks for the speedy mea culpa. i doubt ian watmore, the coo for government, will be too pleased to be called the "accenture chappie" but i can't help but smile when i see that.

    Ian C - I guess the problem here is that whilst we (and they, government) can bandy around figures all day, we have only anecdotal evidence. I haven't seen the recent list of top 10 failed projects or, for that matter, top 10 successful projects. Although one day, when the statute of limitation has expired on my stuff, I'll talk about it. Maybe there's an FOI in there?

    ReplyDelete
  7. Anonymous9:53 pm

    I remember being told to to give as little info to MOD auditors coming to check on the reliability of our software.
    Of course this was in the cost plus days, before fixed price contracts, before the end of the Berlin wall, etc.

    On a similar note who was the executor of the Sainbury's contract, I wonder? *** (I.e. the actual people charged with providing the work,) I'll bet you'll find it was a consultancy, whom exploded the requirements until it was unimplementable, then hid behind a confidentiality agreement. It's clear to me, that most contracts fail because there's a hundred chiefs wanting to make money, and only a few indians wanting to do the work.

    *** or the inland revenue, or Nike.


    The government sanctions this huge redistribution of wealth because it feeds people in deprived areas.

    We share mutual colleagues who would rather talk about a problem than solve it. (Although we share colleagues who are very useful too.) In fact the group of people who aren't capable of solving a problem is such a big number that they drown out the genuine clever people with noise. This is made easy because all the people I know with really brilliant ICT skills have not a remote bit of business interest. They don't even want to be in the board meetings that sign it off, let alone all the other less important discussions.

    It's only when the sh*t hits the fan, do the movers and shakers
    1. shut up.
    2. pretend it was a committee decision.
    3. say nothing because they've gone already to another highly paid nonspecific (but important sounding) consultancy oriented job.
    4. Defer to the people they've known all along, who actually know what they're doing. (the heroes from your graph, who occasionally still care enough to rescue it.)

    All successful IT projects have heroes, because in reality only a few people can actually implement software, and they were born that way. In fact the software industry is so in denial about this, because it's contrary to marxist principles that all should have prizes, that it pretends you can teach ability. It's right up there with the denial of heriditary intelligence in the government sponsored lie that teaching quality makes thick kids competitive in later life.

    In fact there are all kinds of methodologies, schools of thought, toolsets, frameworks etc that are built or invented by people with ability to try to help the people without ability to be able to do something. There's always some silver bullet. Well they have their ideas and I have mind, but it's my contention that theirs is no silver bullet.

    Everyday, I see incapable people reverting to simplistic learned rules to try to solve a problem one way, only a fraction of them can actually think for themselves. Booch, Rumbagh and the other one and their UML creation is like Corgi plumbers, the number of successes is due to UML, but a significant number of failures is usually too(, but at least they were well defined failures.)

    In short, I'll always be one of the 99.999999% of people unable to be a world class marathon runner, and 99.999999% of people won't be able to create software either, because you can't learn (or teach) creativity. You can't aquire it, post birth either.

    As a final point, IT projects are bound to go titsup because MPs don't understand them because they're thick attention seekers.

    Let's face it, they won a popularity contest, and they wanted power and re-election. They didn't win an IQ contest, and wanted satisfaction of a job done. They don't care if something works two months from now and three geniuses have become millionaires, when they could wade through seven years of (sponsored) employment for the masses, to have a month of embarassment. Patricia Hewitt's not exactly thick, she's far cleverer than Harriet Harman, Ruth Kelly or any number of men, such as that thickhead from Humberside who was given a big job to keep his leftie mates thinking they were valued, in that she's in fact been quite good at identifying the areas of government she could hide in where no delivery is required, but she's not a genius either is she, and anyone who can't speak, absoulutely authoritively on IT is basically in no mans land when it comes to finishing a successful project. It stands to reason. If you can't understand it, you can't tell when it's going wrong.

    I'm sure I'll never work for the diplomatic service with outspoken opinions like this, but ICT isn't subjective, it works or it doesn't.

    (Is the internet pernicious? I'm trying very hard to be normal.)

    Ian

    ReplyDelete
  8. Anonymous5:57 pm

    Hi Alan,
    Looking back at this Joe Harley 2007 "only 30% of Govt I.T. Projects succeed" statement, I was wondering if you had any new 2009 thoughts/recommendations on how to maximise the chance of (I.T. centric) project/pgm success in today's Govt climate? I know you pay close attention to language so you will note my use of the positive, rather than the "how to avoid failure!" question! :-)
    Do you have any recent Govt (not necessarily UK) projects/pgms that you would deem worthy of praise?

    ReplyDelete