Wednesday, February 27, 2008

FOI - Army of One

iStock_000004817156XSmall.jpgLast December I suggested that government could adopt a 3 stranded approach, making use of existing tools like Facebook, Wikepedia and others to replace things that have traditionally been built and managed in-house. One of the ideas was a service that would put all FOI requests and the associated responses for all departments on one site. I said:

[W]hat if we took every FoI request - and its response - and published it online with a simple search application, driven by google or windows live or any other engine- so that before you asked your question you could see what else had been asked that was similar; you'd then either just use that information and not bother to ask your own question or you'd refine yours to get a better take. Smart journalists would use the search tool to bring together previously unrelated questions and draw even more conspiratorial conclusions. Smarter ones would phrase their next question to take advantage of the freely obtained knowledge that they already have to find something new. Government would respond, one would hope, by getting smarter about its operations and processes and would use this leverage to drive greater change and efficiency.

And then

The deluge of information would be enormous. The fragments of data would require an entire army to stitch it together into meaningful conclusions. But, let's be honest, government itself is never going to have a big enough internal army to do this stitching but, the outside world, those who want to be part of an open-source government, now maybe they'd have the willing, the time, the intellect and the energy to sort, distill and publish the very best pieces - and government, of course, would pay for such pieces once and once only. Sadly, the name YouGov is already taken by a very clever chap called Nadhim Zahawi, but maybe he'd be open to offers. Failing that, we could always go back to me.gov, the vision of access to government coined in 2000 following the [necessary] demise of open.gov.uk.

I was thinking that someone in government might take this on. What I should have realised is that somewhere in the UK, Francis Irving, one of MySociety's people was already working on exactly such a service.

It's not completely done but it works and I think it's brilliant. William Heath, who must think I disagree with him at every opportunity, will be surprised by that remark I'm sure. In this context, we disagree about only a couple of things - direct.gov's overall usefulness and the search approach within the same site.

What the [to be named] service does is let you ask a question of any department. It badges the request, ties it to a unique email address and then handles the correspondence with the department. The question is posted on the site for all to see along with its status and any responses. There are only a few questions on there so far and even fewer full responses, but it has potential.

I can see that a lot of people on the inside will want to stop this - there will be complaints that somehow FoI was never meant to be published on the Internet, that it was meant to be a bi-directional transaction between government and one citizen. Plainly that doesn't make any sense. If a journalist asks for information and then writes an article about (as has been done a 1001 times already) then what's the difference between that and this site? This site has, of course, the potential to be a much better archive of such requests and, over time, it could save a lot of money by reducing duplication and, particularly, by allowing government to say "we've already answered that - and here's the link" to mysociety's site. I think that will take some time and there will be plenty of resistance. So for now, there's voyeur mode - you and I get to see what kind of questions others are asking and engage mode where you get to ask the question yourself.

Funnily enough, William suggests the site be named open.gov.uk - which I'd completely missed in my thinking in the paragraph I highlighted from the previous post - although I'd even referred to that site. He's right, though, open.gov would be a great name. But I suspect they might have to settle for something else. Opengov.com is for sale, a snip at $7,200. Opengov.co.uk is apparently registered. Opengov.org seems to have something to do with Missouri government. Simple combinations of those words (and adding words like "free") all appear to be taken.

One question that is asked on the site, by William (on a mission from God), is, effectively, what was the story behind using ukonline.gov.uk as the domain name for the UK government's first aggregating site and did money change hands with the ISP, ukonline.com. I wasn't around then and I'd be astonished if there is a story still; and more astonished if money actually changed hands. I think all they got was the footer on ukonline.gov.uk, which I think we removed somewhere in version 2 or possibly 3 (mid 2002?). But there are few, if any, people around from that time, e-mail systems have changed 3 or 4 times and I wonder how good the archives are.

Anyway, this new site is great. It does much of what I said, better than I thought, quicker than I could possibly have imagined, before I'd even put finger to keyboard and all on a shoestring. Give it the traffic it deserves and visit it.

Sunday, February 24, 2008

e-government 3.0

Wait! What? 3.0? How did that happen? I, along with many others, have long had a theory that government should - and often does - sit behind the private sector in terms of technology. A government department taking a leap into the unknown, what used to be (and probably still is) called the bleeding edge is a rare and brave one. Often, at least more than the laws of probability would allow, such efforts end in failure. Web 2.0 is probably more like e-Government 4.0 or maybe even 5.0. Thinking about this has been languishing in my blog drafts folder for many months - the article in the Economist last week prompted me to dig this out and refresh it a little. It's all about what do governments do next? They've had the "get everything online" phase - and many have done pretty well; most moved on to driving up usage of transactions (and, again, many have done well). But what next? What's in the third phase?

Electronic Government 21.06.2001.jpg DWP Conference Slides - 30.01.2003 - stages.jpg

The slide on the left was something I first put up in early 2001 to explain the stages I thought government would go through on the journey to a truly transformed operation. If I remember correctly, the words on the bottom (Publish, Transact, Interact, Transform) were from a report writtten the previous year called "e-Government @ IT's best." I no longer have that report, but I'm pretty sure that when I created the slide I reversed the second and third stages - I felt strongly that interaction would come after transactions, i.e. that transactions were going to be one way at first (taxpayer sends tax return) and would then become two way (government says, "thank you for your tax return. we need this bit of information. did you know that you can also do this. and here's your refund transaction" etc). The yellow octagon (bottom left) is where I thought we were then (i.e. heavily mail dependent, quite some way into the publish stage but with a big bump just ahead of us as we tried to orchestrate transactions).

Later, I put some dates on the slide suggesting that each stage might take at least a year to break through - but that online private sector companies were already moving ahead of government and making the move to some truly dramatic changes. I also used a much simpler background - I have no idea what I was doing with the whole sunset at sea thing. The opening slide from the conference where I showed this slide noted that the FT was saying that we'd already spent £3 billion on e-government in the UK alone, by January 2003. By mid-2003 I think I'd abandoned the slide - I can't find it in any later conference pitches. Evidently I'd realised it didn't say anything by then (and that I was, of course, hopelessly optimistic about the dates by which any of this could be achieved).

But in 2008, I think it has relevance again. Governments, generally, seem to have stalled at the "transact" stage. All of the big plans to move to real interaction, to transformation (dramatic or otherwise) appear - to me viewing from inside government but outside the e-government race (and it was a race) - to have stalled.

If e-government 1.0 was publishing - several thousand websites per country - and e-government 2.0 was about transactions (moderate take-up of a small number of transactions, high take-up of a couple and low take-up of the rest), then what is e-government 3.0?

For me e-government 3.0 is the set of actions that need to be completed to truly have a high level of online interaction between citizen, business and government. The creators of that interaction will be both private and public sector-based. Interaction means lots of things to lots of people. I mean: high levels of usage, bi-directional flows of information (that is, citizens send to government and government engages in dialogue, including offering access to services not yet taken up by that citizen), third parties engage in service provision wrapping their existing offerings alongside government services (so, for instance, third party pension providers will show you your state benefits alongside what your employer benefits could be and potential for additional benefits with separate schemes).

In e-Government 1.0 and 2.0, one of the big surprises for me - and this is a global, not local point - was the lack of engagement from non-government businesses in providing services - that is, few third parties took up the challenge of providing services alongside or in competition with government. From 2000-2003 I figured that we were only a step or two away from seeing that engagement grow strongly. Early on I was involved, loosely I think it's fair to say, with schema development for various financial transactions with government and lots of software vendors and perhaps that gave me a false sense of how the market might develop. In hindsight, there were a few clear reasons for the absence of third party involvement:

1) It's hard to compete with something that's free. If government does something reasonably well, why would you try and charge for something that appears to only offer modest advantages. In the UK an example here is HMRC's Self Assessment process. Tools that tried to compete were provided by Which (TaxCalc) and Microsoft (TaxSaver I think, although there was another one called DigiTax perhaps) - they were good, but they weren't good enough and they died out. There is also a famous story of an online fishing licence process that with its attendant publicity, in one go, pretty much killed off aspirations for many 3rd parties.

2) Governments are not necessarily able to be open about the complex methods of interaction. The UK still, I believe, leads the field with its govtalk certification (see www.govtalk.gov.uk) but even that still contains many draft schema dating as far back as 2002 and lacks important schema in, say, the health records domain. The last major update to the documentation was in March 2005 and that only really to note that the OeE was no the eGU. Interacting with government is complicated and takes a lot of documentation to figure out. When Sun partnered with Software AG to design and build a local box (known as a DIS box) that would connect a department to the Government Gateway, reams of documentation were necessary to handle the various messages that could travel back and forth.

3) Allowing third parties in makes things complicated for government. At present, if the main provider of a service is government and they realise that there's a problem with their online service, a quick change and everything can be fixed. What if you've actually certified 5 external providers to provide that same service and furnished them with protocols, schema and instructions? And then find there's a problem. The quick change is no longer possible and overall service can suffer quite dramatically.

4) Finally, at the beginning government figures it needs to seed the market. It doesn't know how many people are going to use a service, and it can't make promises to one third party over another about the business opportunity. So figuring out how much market share is there for the taking by 3rd parties is a challenge. Government doesn't know. To stay with a UK example, HMRC's Self Assessment saw 3.7 million returns last tax year (plus a few thousand more in the couple of days after the filing deadline) - that's a pretty good number, probably close to 50% of the total returns but it's taken 7 years to build to that number and it's probably a good 2 million short of the total that could be delivered today with minimal effort from those sending in tax returns. I don't think anyone - certainly not me - starting out on the e-government trail in 2000, would have thought that it would take this long to get 50% of the returns in electronically.

I think e-government 3.0 has to take these, and other, issues head-on in an effort to really drive up usage by the population - and use the space that creates amongst the internal workforce to drive real change to processes, dramatically simplifying them, joining them up where it makes sense to do so (and recognising that "sense" is sometimes a hard word to define at the get go) - bringing it all together under a new overall mission, allied to the somewhat waning transformational government agenda. I believe people need a new flag to rally under - the momentum has gone from the original drive to get things online, progress is only incremental and the buzz has gone. E-government should, by now, be business as usual of course; there is, perhaps, no need even to continue to use the word but, at the same time, the original aims of governments that embarked on the course have not yet been entirely realised and so we need something to rally the troops.

Here are the 6 things I would propose a government do to channel their efforts towards busting the next barrier to progress in e-government:

1. A clear statement on the direction of travel for the government adopting this plan. For instance, they will no longer develop new e-government services where there is potential for commercial involvement and competition, where there are existing services offered by such third parties or in a specific list of defined transactions that are as yet undeveloped, difficult to develop, or have the potential to be widely used in specific markets. Essentially, a government would publish a list of 50 or 100 transactions that it wanted the private sector to adopt, embrace, drive usage on and join up. There I've said it: join up. Some transactions that already exist do so only as stove-pipe transactions. Government would allow third parties to join those up and not compete with them.

2. An OpenAPI will be developed, documented and published detailing how to get to and from government. I noted above that govtalk had gone a long way down this path. but there is plainly more to do. I was intrigued to see an announcement today regarding health clinics in the USA developing connections to Google for storing health information - with the idea being that it can be retrieved as the patients move from state to state or doctor to doctor. This strikes as a perfect example of an API as I imagine it. Plainly there are, in this case, privacy issues to be concerned about - but Google reads your email and may know as much about your health if you mail updates to friends and family as anyone. This would be put on a fast path, using the hit list of transactions in (0) above as the base. Government would do what the software industry does and always support at least n-1 versions, perhaps more.

3. Government would develop rules engine capability "in the cloud" - i.e. processing would no longer need to be behind the full set of government firewalls, but could be handled a step away with only the final results of transactions entering government. Government would commit to make these engines highly available, scaleable and up to date. It would likely partner with suppliers who have proven that they can do this - inevitably, that is a small list but it plainly includes the big Internet companies such as Amazon, Google, eBay and Microsoft; there would also need to be local (national) companies in this list to ensure that the local economy developed along the same path as the titans, where possible.

4. A way to encourage small businesses to get into the provision of such services would be set up. I imagine a Dragon's Den style competitions with businesses receiving funding to deliver services, based on the hit list with bigger incentives for the harder ones or where government already offers reasonable services. To date, I think governments around the world have fought shy of such an action, worried that it somehow breaks procurement rules or gives some players an unfair advantage. As I see it, anyone who wants to play can get a meeting, but they have to show up with a business case and plan that the angels (or is it dragons?) say is viable and that government is prepared to put its money into. To ensure the government lives up to its end of the deal, it will commit to invest in X deals for £Y a year - and whatever X and Y are, they'd be public and reported on monthly on the national portal - that would be direct.gov in the UK, USA.gov etc

5. To facilitate longer term development and operation, a model for paying transaction fees for businesses meeting government certification and providing accurate, timely transactions would be developed. The fees would start low and grow as volumes grew, incenting providers to seek out bigger markets, giving players that were growing fastest the opportunity to achieve real economies of scale. At some point, fees would level off and perhaps even reduce year on year, incenting the providers to find ways to deliver more efficiently. Government would pay these transaction fees by abandoning development of services where competition was being introduced, not entering new markets, no longer needing to upgrade servers and so on for its own websites. Government would have to provide some degree of technical and business support, and this would be factored into the costs.

6. Third parties would be granted the right to white label government content, with acknowledgement, using simple tools that made necessary policies, rules and instructions available to them. Ideally, content would come from the national portal which is the most likely place for the investment to be made to allow this but also may become, if it isn't already, the only source of up to date, joined up content.

Making these 6 points happen would be no easy task. It would require a step up in co-ordination inside most governments, a new reaching out to the private sector (and a large number of cynicism bypasses there), a rapid set of decisions on where the priority areas were and some fast delivery to prove that it was viable. It's very likely that there are some folks out there waiting in the wings, who have services that could be offered as pilots for this or have tools that could be used to develop services quickly, but they don't know who to talk to, how to get to them, how to fund it or how they will construct a business model.

I'd envisage some small businesses getting off the ground this time round - there were attempts at this during the early dotcom boom days; some survived, most did not - and growing into strong national transaction players who would look to do interesting stuff like take paper processing off the hands of government as a vehicle to make it electronically more quickly. There could be a mixed economy of big and small players emerging over time. Just as likely though is that existing industries - banks, accountants and so on - would step into providing their clients additional services and value.

In the background, government would work on its bigger picture for transformation - once assured that increasing numbers of transactions were coming in electronically, the smart folks on the inside of government would look for ways to redeploy staff to more customer focused areas, to develop new business models that could be based on the assumption that all transactions for, say, a start up business would be electronic and so on.

This would take time - 5 years or more. That might mean 10 for some governments. But it seems to me that governments have a better sense of time than most. At the moment, though, few have set out what it is that they want from their online services over the next 2-3 years let alone 5-10. I think a model like this could just see some real breakthroughs that would significantly benefit national economies - both on consumer and producer sides.

Dan Jellinek Transform slides - 13.05.2004 - inandout.jpg DWP Conference Slides - 30.01.2003-opening up.jpg Mark Forth Slides - Year 3 - 17.11.04 - storm.jpg

Wednesday, February 20, 2008

Economical E-Government

I love The Economist. Like The New Yorker and Mother Jones it has a style of journalism that is highly readable, exceptionally good at distilling a complex subject into interesting chunks, enormously educational and impossible to read in the time period between issues. I wish I could write like that. A guy I was at school with is one of their editors - I met him at a conference in the USA, quite by chance (in fact, I met him twice there - and there were about a million people there); I'm very impressed, he was a talented guy then and is plainly even more so now. The problem with reading those 3 magazines is that I always end up with a backlog - there's only so much reading you can get done on a crowded tube.

The February 16th issue has a feature on e-government headlined "The electronic bureaucrat." I've just got through it. First, kudos to Jeremy Gould for being marked out as a guy with an excellent blog - and, whilst it doesn't say it directly, one of the few on the payroll civil servants with any kind of a blog let alone an excellent one.

I thought there were some great threads to pull out and, cheekily, I've followed some of the bullets with slides from my collection of moderate hits, 2000-2004:

1. Inevitably e-government doesn't exist by itself. It must merge and conform to the physical word. The example is for those looking to get a visa for America: you apply online, pay money online, get an email confirmation, print it and then take it to the embassy in London (or doubtless elsewhere) where they check the barcode, allow you access, and let you collect your visa. A brilliant melding of a process that needs you to be there but doesn't need you to be there the whole time.
EDS workshop slides - 12.02.2003.jpg
IDeA slides - 28.09.2004.jpg
2. Governments don't face competition. In fact, the sentence is "Governments rarely face competition". Alistair Darling besieged by non-domiciled bankers who are all off to Geneva in the light of recent taxation demands may beg to differ. Competition exists at multiple levels. Local authorities compete for businesses to site themselves within their boundaries; regions compete for investment by big companies; cities compete for banks to put their HQ in town; countries compete for the big capital projects, major manufacturing plants and so on. This hasn't yet visibly manifested at a transaction level - but years ago I used to tell the Inland Revenue that if they made it easy for companies to set themselves up (PAYE, VAT, Corporation Tax, overall regulation etc) then they would attract new businesses because the cost of set-up would fall and that would be attractive. That hasn't changed. Sure, the average person isn't going to move to a new country just because the online services are better there (unlike if the tax treatment is felt to be better), but they are going to change their behaviour if the online service is very much better than the offline (and I say "very much better" because slightly better doesn't overcome the inertia that is present with the status quo).
IR Scotland - Future Business Challenges-comp.jpg
IR Scotland - Future Business Challenges-notsep.jpg
3. Personalisation is important. This is rule 1 of 3 by a chap from PA Consulting. Frankly I think he's off-base and even if he'd said it in 2003 rather than 2008 he'd still have been off-base. He says government should be like online shopping and record your preferences, that it should be available round the clock and that services have to be easy to use. He wants government sites to be "beautifully designed." Personalisation was a massive sink-hole from 2001-2004, chased by every department looking for a website as a "must have" feature, even if no one actually knew what it was. It didn't mean anything then and it doesn't mean anything now. You interact with government on so many levels and with so many personalities that most of the time, you're someone new each time you come back. Reminding me of what my tax return looked like last year doesn't help me if I've changed job, saved money, spent money, bought shares, sold shares or whatever. As to being around all the time and being compelling, yes, I agree. But then everyone agreed with that in 2000. Beautiful design - I'm already thinking of parrots hanging from trees - won't attract the customer. Many travel sites that I use every day are still nearly impossible to use - a rant for another time - but you use them because they're the best way to get the job done. Government needs to show it can get the job done and better than the offline equivalent. The online VAT form in the UK is still not attractive because the offline system is so fantastically well done. Fill in 9 boxes - and probably 6 if you don't trade across EU boundaries - and you're done. The only way to make that easier is to do the sums for me. Aha! Maybe we're on to something there.

Saudia Arabia slides - 23.04.2002.jpg

4. i-government is the most prevalent form of e-government. That is, putting everything you possibly can online is still very common. In the early days of the web this was called shovelware or brochureware. And it is still depressingly common. It's better than not having it online, but it doesn't deal with the massive and inevitable duplication, the confusion that entails when different pages (let alone different sites) appear to contradict each other and, as The Economist says, it doesn't mean that when you do eventually interact, your service will be any better.

Electronic Government 21.06.2001.jpg
5. Websites are closing down. This sounds like good news - and Lord knows I'm a huge fan of reducing websites (see many, many posts here - but I haven't yet seen the evidence. The statement in the article is that 551 of the 951 central government websites "have closed" - i.e. past tense. Not will or are planned to, but have. if it's true, that's a brilliant step forward and I'm delighted. When the guys in OeE put direct.gov together, the expectation was that other sites would be starved of visitors and so would eventually shut down through lack of traffic. Closing them down before that is even better. It narrows the information sources and forces the main player, direct.gov, to be more authoritative. If it isn't, people switch off and don't come back. It's like the French Encyclopaedia, Quid, no longer being printed - if there's a better source (Wikipedia in this case), people go there. But as to 400 sites closing, I'm sceptical - show me the money as Jerry Maguire might say. There's a sidebar here on the search engine in direct.gov which is hardly a new topic for this blog; I'll deal with that another time. It does appear, though, that the DG search team have read the article because the search terms used to illustrate the point now claim the third spot (as opposed to not being in the top 100).
Dan Jellinek Transform slides - 13.05.2004 - website curve.jpg

6. The digital have-nots number in the billions. My late and very dear friend, Angela Vivian, campaigned tirelessly (ok, so she tired everyone else out, but she was, herself, tireless) on this very topic for years. Whilst internet access has increased in the UK it flattened out in the high 50s percent (nudging 60% now, but only just), as it has in other developed countries. In less developed countries, it hasn't even got to the high 5s or 10s. The Economist refer to a Southern Indian State, Andhra Pradesh, that appears to have achieved enormous things with a project called "e-seva" - essentially, public service offices where Internet access is available and transactions can be completed (paying bills such as electricity and telephone, where the service is still doubtless nationalised and so part of government - just like banks in the UK). It's an impressive looking set of services and with only 119 centres they see 110,000 transactions a day. The population of this state is over 75,000,000 - not very much bigger than the UK, and certainly economically different. I believe that this total of transactions is greater than that carried out in the UK at present, assuming they're all carried online in India (and I suspect they're electronic at source - like a cheque process in a bank: the customer gives the paper, which is immediately rendered electronically and from then on the transaction is 100% online).

Loudcloud Client Event Slides - 28.11.2001.jpg

7. Proving. No scratch that. Defining the benefits remains hard. The UK is rightly given praise for trying to figure out a model for what an e-government project brings to the party. Whilst I was around when that was being done, it was nothing to do with me; but it needed to be done and was done pretty well. Other countries have tried (e.g. Australia) and found similar things - there's a general sense of e-government is better but if you try and pin down whether the public sector is smaller or the economy has benefited, well, you're going to struggle. Some of that is because figuring out the true transactional costs offline is already hard; and quantifying the saving and putting pound notes on it is even harder.

PSF Slides - 28.03.2003.jpg
8. It isn't about the technology. There's a great story about how the new mayor of the District of Columbia has shaken up process after process inside the operation of his city, tied in a single portal (dc.gov - hmmmm so much for that whole problem we had once with 2 letter domain names; in the UK he'd have been pushed down DoC.gov), splahed out on some plasma screens, used google apps (and thrown away all of their own servers) and are even looking at iPhones instead of police radios (do it - apple stock needs the juice). Alongside the technology, such as it is, they've rationalised the hiring process, stripped out every redundant step and made bureaucracy a dirty word. I just have no glib answer to that. It just sounds awesome. DC may be the heart of government for the USA but it has long been a poor and disenfranchised area with high crime rates (I think this year is the first time they've been invited to vote in the presidential primaries even). There is no slide that goes with this set of achievements.

9. e-Government is only the beginning. There is so much left undone from the original vision - join up, citizen focused, simple government, direct access government. It doesn't matter which party conjured up the words, the intent was always the same. Make government more accessible to the citizen - in both directions.
IDeA slides - 28.09.2004-joinup.jpg
The online version of the story appears to be but a shadow of the offline version - the Economist (note, it's www.economist.com, not www.theeconomist.com) is a subscription play (once it's owned by the Murdoch family, they'll unwind that I'm sure. What? It's not going to be owned by them? One day maybe).

It's a great article overall. It distills several years of learning into 16 easy to read pages. It has examples drawn from all over the world, although those from the UK tend not to be the most positive. Did we really need another story about the NHS and are ID cards and the NHS spine - not a £12.4 billion project by itself fact-checkers - as if it were the only global example of data aggregation?

IDeA slides - 28.09.2004-data.jpg

That said, it's a backward look, i.e. mostly a catalogue of what people have done rather than what they could do. But by highlighting some real bright spots - the folks in DC for instance - the writers tantalise us with what we could do, if we got it together, crashed through the barricades and made it happen.

IDeA slides - 28.09.2004-simple.jpg

Tuesday, February 19, 2008

More Entourage 2008

bang.jpgTechnology will be the death of me. It's the simplest things that go the most wrong. E-mail. Well, Microsoft e-mail.

In defence of Apple, this is plainly a Microsoft problem, I've had a pile of problems with Outlook 2008 on Windows - mostly where the whole PC would hang when receiving new email. I don't think I was alone in seeing that problem.

Anyway, back to my world of MacBook Air and, to recap:

- I'd set up all of my email, migrating thousands of old mails from my PC, via Thunderbird and using all kinds of odd scripts. I just checked and I migrated around 25,000 mails for a grand total of 3.49GB

- My MacBook Air crashed, rendering the office database corrupt and unusable. Attempts to rebuild it all failed at stage 4 of 5.

- I tried re-installing Office 2008, to no gain

_ Eventually I found the "main identity" files (in your user folder, under "Microsoft User Data" and deleted everything. That cleared the problem and I was free to work with a blank slate on Entourage

- I installed Thunderbird on my Mac and moved all of my email into it. Then I killed off a pile of old email in sent and inbox (having secured a backup first) and exported just 3,000 email to Entourage compatible files

- I set up Entourage all over again. And did a database compaction (is that even a word in this context?) which reduced the mailbox size from 1.2GB to 327MB (and even handily took a backup) Surely that would be ok for Entourage to handle?

- Within an hour it had crashed. or, at least, it removed the "quit" option from the Entourage menu whilst letting me use other emails. The database utility popped up and said my database was corrupt again, but that it couldn't quit Entourage so I needed to do something about it. I got the feeling its knickers were in a twist - kind of an error message tone thing.

- So, with great trepidation, I force quit Entourage. And fired up the database rebuild. In no time it had completed, without errors this time.

- I opened Entourage and found all of my email there, safe and sound. But when I tried to send an email, Entourage told me that I first needed to create an email account. It seems the rebuild had deleted all of my account set up details.

This thing gets one more chance. I'm a patient guy. I'll give it 3 strikes. And then it's back to thunderbird. Meanwhile, I'm leaving all of my emails on the server.

Proceed with absolute caution if you are considering Entourage as your main client. Proceed with real, real caution.

Entourage 2008 - Not Ready For Prime Time

Day three of MacBook Air ownership, day 0.01 of actually being able to use it. The effort to transfer is enormous - not just the email copying I wrote of in my last post.

First job is the "inevitable download" - bazillions of megabytes of patches and enhancements, not least Leopard 10.5.2. Second was software installation - Mac Office 2008, MarsEdit (to write this blog). Third was data file copying (all of my document folders). Fourth was "asset stripping" - getting rid of the things that just aren't needed (Garageband, iMovie etc. Haven't quite decided about iTunes so far - I might need it to sync my iPhone). Fifth and final was the great email conversion. Right there is a good 12 hours of effort.

But it was that last task that was a complete and utter bust, it turns out.

It had worked fine for all of about 60 minutes of use. Not even enough time to run down the battery (about 2 1/2 hours I think - forget all this 5 hour stuff).

I then got the Apple blue screen of death - a polite message in half a dozen languages saying that I'd need to press the power button to reboot. No problem I thought, I can do that. One crash in less than a day. Maybe I just had too much going on.

After the reboot, I tried to open Entourage. No can do said the unhappy programme. The database is kaboom. Or some technical equivalent. No problem it said, just rebuild it using this handy utility. 60 minutes later, the handy utility failed at step 4 of 5 of the rebuild. Reboot again. Rebuild again. Same failure, same place. Check the web. Dozens of instances of the same problem.

I imagine - and this is only me clutching at vague straws - that this is all to do with the size of my email inbox. I have 11,000 emails in my inbox. I don't use folders. That's what search is for. Why would I want to sort everything so that I could never find it again when search (especially in Windows Office 2008) works so well? Outlook handled the mail fine, why should Mac be any different? Sloppy code I assume.

I've now downloaded Thunderbird on the Mac (given I had to export from Outlook to Thunderbird to get to Entourage) and imported my email into that. Took about 20 minutes - far faster than the import into Entourage - and so far, doesn't seem to have a problem with the mailbox size.

It's a real shame. I thought Entourage, at least in the short time I'd been able to use it, was pretty good. I liked the "My Day" window a lot. Simple but effective.

So now I'm teaching another email programme what spam is, typing in my email account details again, setting up the preview pane the way I like it and learning some foibles.

There's another problem since that crash and reboot. Opening any file in word or powerpoint or excel brings up an error telling me that the Office database is corrupt and that I need to rebuild it (tried that dummy, don't you know that?). It doesn't stop it working, it just tells me it's bust and invites me to press "ok". Can you imagine the conversation with a heart surgeon in this scenario? "I'm sorry sir, your heart is corrupt. You have 3 days to live. Press OK to continue". Never mind cancel or reboot or "I can sort it", just press ok to die in 3 days. I figured a quick reinstall of Mac Office would sort out the database corruption problem. No joy. Same error, same time, same place.

I killed everything I could find that had the Office label on it, using AppZapper and then reinstalled again from afresh. Still get the database error, still can't rebuild it. I have no idea where this database is, how to delete it or how to resolve the problem. I'd even be prepared to use Entourage from scratch with a fresh and empty database if I could figure out how to get it to start.

The solution, at least I think it's the solution, is to kill your "main identity". Very Minority Report. Or is it 1984? I deleted the file - 3.5gb of it and restarted Entourage. A nice, shiny, empty, entirely blank email programme greeted me. What I thought I'd do now is copy just a few of my emails over, say the last 3 months. If I'm lucky, that will work and I'll never see this problem again. Or at least not for a year or so.

Technology of the 21st Century? Microsoft have to be kidding. This is version 3 of Office for Mac I think. Normally it's right by now. It obviously takes longer than it used to.

There are some famous last words about to be eaten sometime soon I fear. Linux might even start to look attractive at this rate.

Friday, February 15, 2008

Moving to Mac. The Entourage / Outlook Import and Conversion Story

Alongside moving to a new blog, I've also been working on moving to a new Mac, a Macbook Air to be exact. And when I say "moving", I mean moving from a Sony Vaio and Vista to Mac and Leopard. I've tried this before, not from my the Vaio but from various other devices and always given up after a couple of weeks of effort. This time I think it's different. And that's not famous last words. Why?

- The MacBook Air is the first Mac I've seen with a decent screen that can compete for lack of weight against laptops such as my Vaio. The Vaio is about 1.7kg plus the power brick, call it 1.9kg. The MBA is about 1.3kg with the brick (previous Macs were about 3.3kg at best). Ok, it doesn't have a DVD drive - but I never use it except for installing software, and that works fine using my iMac drive shared across the network.

- Office 2008 is a significant leap ahead versus previous Mac Office versions. It's as good as Windows Office 2008 - some would say better (there are fewer interface changes which means fewer leaps of logic to understand where, say, paste special has gone)

- Third party tools such as Parallels or VMware fusion let you run the few Windows programmes (I have some stock charting software and my money management software) that you need to run. I say "need" - I'd like to replace them, but nothing so far works as well

Now, on to the things that are still hard:

- Moving email from Outlook on a PC to Entourage on a Mac. What a palava. Microsoft could not have made this harder if they'd set out to from the beginning. Some instructions that might help (I've found most guides on the web are near useless or, at best, confused).

So, how to import from Outlook (on a PC) to Entourage (on a Mac) via Thunderbird (on a PC)

1. Install thunderbird on your PC.

2. When you start Thunderbird it will helpfully bring up an import menu with only one option which is "don't import anything." Don't worry about this and let it carry on.

3. Configure your email box if you must, but set it so that it doesn't fetch new mail (otherwise you quickly find you've got yet another email inbox)

3. On the file menu, you'll find that you can import from outlook. This function fetches everything - all the subfolders and so on - but it takes a while. I have about 3gb of email and it must have taken a good hour or so.

4. Now you'll have all of your email in Thunderbird. Well done.

5. Go to this site and download the import script

6. Go to the tools menu and pick addons. You can then install the script you just downloaded.

7. You'll end up with a file that is probably as big as your pst file in outlook. Mine was a 3gb folder with many sub-folders

8. Copy that folder to your shiny new Mac. There are a bunch of spurious files and folders in there with various extensions (such as .sbd) and sometimes there's a file (inbox, for instance) as well as folder with the same name.

9. Select each file that you want to import to Entourage and rename it ".mbx" - I suspect you'll find that none of the files have any extensions and show up with icons that look like TV screens (they do on mine)

10. Nearly there. Visit this page all about Entourage and download the script that converts files to text files. Select each file you want to convert (all the ones that you renamed as .mbx) and run the script. You can do it one by one or as a bulk convert - just keep track when you import with which ones you have and haven't done.

11. Go to the Entourage file menu and select "import". Pick "contacts or messages from a text file" and then "import messages from an MBOX-format text file" (note that whilst this says MBOX it means .mbx)

12. Choose the file - this time you have to do it one by one.

13. Entourage will go quiet for a while. You won't be able to press anything but there will be no spinning beachball. You just have to wait.

14. Eventually - depending on how big your files are - it will say "finish". Click that and you'll have to wait again, but you will get a beachball this time. This seems to be a shorter wait, but a wait nonetheless.

15. The folder you have just imported will appear on the left hand side of the screen under the "on my computer" menu. You can then decide what to do with it. I copied all of the inbox file straight into the inbox, one for one, but moved the sub-folders under it. This actually copies all of the folders rather than moves them so, once you're done, you have to delete the old ones.

I'm about to do my contacts now, but expect that's a similar process. Sadly, all of my iphone contacts are synced with Outlook, but iTunes won't let me sync them with Entourage - which would be a one step process. Like I said, they have made this too hard for no good reason. And don't babble on about monopolistic suppliers and not wanting people to convert, this is just too hard for no good or defensible reason.

It can be done. And, like I said, this time I think it will stick. If only because half way through this process, my reliable (seriously, I'd had no issues) Vista laptop crashed. When it recovered it asked if I wanted it to search for a solution. It found one and said "Vista, by Microsoft, cause this problem. Download this patch" or something like that. I downloaded it and installed it. Only for the patch install to say "this patch is not suitable for your system". Well, thank you, that was kind.

There are, though, some problems still. I don't have an external DVD and I can't figure out how to install Vista under Parallels via the Remote Disc (which installed Office without a hitch); Parallels doesn't allow this to work (they say that the MacBook Air doesn't but, as I said, it installs Office - but what do I know?); I know I don't know anything about this kind of thing, but surely, given I have a valid licence key (which it verifies before telling me I can't actually do the install), it could just get the image file off the web and download it?. Doubtless there are more of those to come.

I really, really hope that this isn't famous last words.

Wednesday, February 13, 2008

Changes

I'm making a series of incremental changes to the blog. New titles, styles, layouts, page elements, fonts, colours and so on. The move to Google's custom domain means I can now play with the templates far more easily (previously, any changes had to be made in HTML; now, everything is point and click). I haven't quite settled on how it should all look yet, but I'm getting there.

Monday, February 11, 2008

This blog has moved

If you've arrived at this page then hopefully it's because you've seen the redirect instructions for the new homepage for my blog or you've been automatically redirected.

I've had many problems in the last few months publishing to this blog - which is created using Google's Blogger application and then FTP'd to my domain name from within Blogger - and this is at least part of the excuse for so few posts recently (the main part of the excuse being that I haven't written any other ones of course).

The web is full of threads from people upset that FTP no longer seems to work consistently when using Blogger, irrespective of which domain provider or host service you use. For me, "inconsistent" means it works one time in fifty or so. No one seems to know what the problem is and, despite plenty of forum questions, Google don't seem to have acknowledged the issue - I'm sure they have plenty of other things on their minds and will get to this sometime.

For now then, to get around this, I've created a new subdomain http://blog.diverdiver.com and my blog is now set up there, using Blogger's custom domain feature (which just needed me to set up a cname alias to point to google).

I'm redirecting from the old blog using an HTML redirect which doesn't seem as good as a 301 redirect (so people tell me) but I haven't managed to get that to work. So it would be great if you could update your references to this blog:

  • If you access this blog by a favourite, please update your favourite to point to http://blog.diverdiver.com
  • If you access by RSS, please click on the this link http://blog.diverdiver.com and resubscribe (I use Google Reader and that picked up the new feed by itself - if that doesn't work for you, please leave me a comment and I'll see if there's a better way)

Sunday, February 10, 2008

Product Launches 2.0

Apple got it in the neck these last few days for launching enhanced versions of the iPhone and iPod Touch. I say enhanced: they hadded memory, 8gb for the iPhone and 32gb for the Touch. I mean it's disgraceful really, isn't it? They launch a phone 7 months ago and then go and upgrade it with no prior notice. They even had a whole show in which they could have mentioned it just 18 days ago when MacWorld was in full flow. But instead, they deliberately fooled people that there were no new announcements coming and then foisted a couple on an unsuspecting world a short while later. People, rise up and start a class action lawsuit. You know it's the right thing to do.

Meanwhile, I propose a new Product Launch Process (PLP) to counter this disgraceful behaviour by suppliers. I'm almost moved to launch a petition on the Downing Street website - I feel sure it will get more votes than Jeremy Clarkson for Prime Minister (6,000 odd more and it will break the 50,000. 4 million iphones have been sold, or maybe not, so I'm onto a winner).

So, here's the new process

1. We'll define International Launch Day (ILD) as June 1st every year. On this day, all products must be announced and communicated to the world through every possible channel.

2. Anyone purchasing a product that is subsequently upgraded in the 364 days before ILD (i.e. a full one day after the last ILD) will be entitled to take ownership of the new product for only the price delta between the old one and the new one; if there is a fall in price, they shall be entitled to receive the difference as a cash settlement.

3. Products that are launched intra-year, i.e. those who break the ILD protocol and decide to launch between ILDs, must be sent to all registered owners of similar products bought at the last ILD, for a consideration in price. That is to say, if a 32gb iPhone comes out and you bought a 16gb at the last ILD, then you will be entitled to receive the new one for only the price delta. If the price has dropped, you will receive the new model and a cash settlement equivalent to the difference in price.

4. To make this simple, product vendors are encouraged to ensure that features are kept simple and discrete. Overly-complicated feature additions will make these comparisons difficult to meet. For instance, what if Apple had launched the iPod Touch on ILD 2006 and then the iPhone on ILD 2007? That would have instantly caused confusion: would purchasers of the Touch be entitled to a free replacement iPhone, once they returned their Touch? Those kind of issues would need lawyers to resolve so it is suggested that companies stick to the basics and refrain from introducing ground-breaking innovations, lest the cost of lawsuits be too great to bear.

5. In the event that a product vendor does need to introduce a ground-breakingly innovative product, they should endeavour to cripple it by ensuring that it does not have all the features that any normal, self-respecting customer would want - or, at least, to wait for an ILD when they do have all of those features in. A good example would be the MacBook Air. Apple should, plainly, have waited until they could shoe-horn the combo CD/DVD/Blu-Ray/HD-DVD drive into the case. Or, just released a MacBook DenseAir instead.

5.1 There may be rare cases when a product vendor wishes to introduce a genuinely ground-breakingly innovative product and, at the same time, does not wish to cripple it. At that time they should appoint a fully independent ombudsman (apply here) who will identify the features that contribute to making a price difference with previously released and similar models, in as much as "similar" can be defined and interpreted, and the ombudsman will set a price rebate that will be granted to all those who purchased models that were in any way "similar", as defined by the ombudsman and those purchasers will be entitled to take on the new model having returned their old model.

6. If the product vendor feels that they need to maintain a more regular release schedule, perhaps for reasons of supply chain planning, stock list rundowns, market opportunity and so on, then they should announce 180 days before release what it is that they place to release on what will be called an Interim International Launch Day or I2LD. Following such a launch, rules 2 and 3 would, of course, continue to apply and product replacements/discounts/upgrades would be granted to those who had purchased previous models, unless they were groundbreaking innovations, in which case see 5.1.


All of this should make for a much simpler and more sensible model of technology procurement. No one will be caught by surprise by unexpected releases, companies will have plenty of time to plan for launches and ensure that there is appropriate stock and prices will be kept harmonious with the employment of Murphy's Second Law which states, clearly, that "Any technology product that costs less than the last one you bought tells you that you should have waited" and Murphy's Third Law which states that "If you bought a product the day before the new one came out, then you should have seen it coming."

There will be those who will complain that this new process will stifle innovation. They'll be wrong of course - what this will bring is price stability and predictability over the long term. Suppliers will love it, customers will love it.

Saturday, February 02, 2008

Potential Presidents as Gadgets

Watching the goings on over Presidential Primaries in the USA is engrossing. As is watching the press reviews of the MacBook Air.

Seems to me that if you were looking for products that maps to those Running for President, perhaps you'd get the following, drawing on the characterisations of the individuals that seem to appear most often in the press:

Barak Obama is the MacBook Air. Sexy, high gloss, something to aspire to, real innovation and change in the marketplace, but lacking several things that many consider should be available by default (not to say that politicians should have replaceable batteries of course). A half year ago, Barak would have been the iPhone; And 10 years ago he'd have been the first iMac - all gadgets that lacked features (remember the iMac was the first PC to have no floppy drive, and the consternation that caused). You could say, if he isn't right this time round, he will be in the future. But you don't know for sure about the MacBook Air despite its evident high desirability; those omissions nag at you and might nag at you in the future and there's enough doubt to maybe stop you getting one until the next version comes out.

Hillary Clinton is a high end Dell PC. You know the brand, you know it's had some problems in the past. You used to be able to say that Dell had two people running the company - Dell and Rollins, but you know that there's room for only one - the CEO is back in charge and he's come out fighting. The Dell machine is solid and reliable and Dell has learned its lessons - every deficiency has been analysed, inspected and improved on - as it has diversified and grown from humble beginnings to the awesome machine it is now. This machine, of course, runs Windows Vista - new and upgraded with features you haven't seen before, but based on the core of what has gone before.

John McCain strikes me as a Linux operating system running on Intel. It does what it says on the box. No frills - but lots of hidden complexity. The Intel side is the one that has been around a long time, almost as long as anyone can remember and has moved and changed as the world has changed around it. The linux side is where you have to move some way to get to where he's at and perhaps give up a few of your sacred beliefs (such as treasured devices and applications). If you have firm views about how things should be and are prepared to compromise in places, you could go Linux.

Mitt Romney is harder to game. He might be a Garmin SatNav device - one that navigates, acts as a phone speaker, does bluetooth and traffic upates. Essentially, if you change direction, it will figure out where you are and try and get you to where it thinks you should go. He could also be the new limited edition ThinkPad with the calfskin case; expensive and elite. Some might say that he's a server with virtualisation capability, running multiple operating systems across multiple cores, each tuned to a specific task or set of opinions to process, and each operating independently from the other, even when performing two contradictory tasks.

Rudy Guiliani is, plainly, the Palm Folio. Available for only a short time in a narrow geographical area and then withdrawn before anyone actually owned one.

One wonders what would happen if Dell and Apple combined to produce the DellBook Air. If Hillary and Obama ran together?

I guess I could have had just as much fun comparing the candidates to wines - so many good ones to choose from: elegant Burgundy full of the effects of terroir, fine Bordeaux steeped in centuries of tradition, exotic vintage champagne prepared in ways only the Champenois know, Australian Shiraz full of peppers and spice, New Zealand Sauvignon with its in your face fruit and acidity.

Likewise, I could have compared them with companies. Who would be Microsoft or Yahoo (oh, wait Microo!)? Who would be Google? Or Petfoodonline.com? Or General Motors? Or GE?

When so much money is being consumed by each candidate as they strive to present their views on a huge variety of policies, issues and proposals, campaigning across every state and appearing on national television at every opportunity both in debates and in paid-for advertisements, I imagine many voters are boiling each candidate down to a few key phrases describing why they like or don't like them. Gadgets is as good a substitute as any.

One of the things I find odd, when comparing US to UK elections is that there's so much about the President as an individual and nothing about the party - the other people who will be governing. An incoming President, between the election and the inauguration and handover, has to hire many hundreds of people to fulfil roles throughout the administration. In the UK, an incoming Prime Minister knows who his/her ministers are (although they reserve the right to change them or take on new ones through the House of Lords) but has, immediately, a fully staffed civil service ready to take on the new mission - with the heads of the civil service departments there to buffer the level of change so that their teams can absorb it. It's an interesting difference. You wouldn't hire anyone else for any job that you had in your own business based on how much either of their own or of other peoples' money they had spent to get in front of you for interview nor would you hire them without real evidence that they had done a very similar job beforehand. But that's politics.

These are entirely personal opinions of course. Perhaps it's a good job I'm not able to vote in these elections.

Check. Check. Check.

In the mid-1930s the US Air Force was on the hunt for a long range bomber. They had very specific requirements about how far the 'plane would need to be able to travel, how fast it could fly and the size of payload it could carry. There was plenty of competition - from Boeing, Martin (now Lockheed Martin) and Douglas (McDonnell Douglas). Boeing's entry into the show was code-named the "Model 299."; it could fly twice as far and twice as fast as the others as well as carry up to 5 times more payload but it was also expensive. It looked like an easy choice. The top brass were summoned to a flight test in October 1935. The Model 299 taxied, took off, climbed and promptly nose-dived into the ground, killing some of the crew. That, on the face of it, was the end of that and the Model 299 could have been no more.

The crash investigation that followed determined that the pilot had forgotten to disengage a control, known as the "gust lock"; this was supposed to be locked in place when on the ground and disengaged before take-off. The Model 299 was probably the most complicated 'plane ever produced.

The Model 299 had its fans - and they were reluctant to see that as the end of the runway for such a 'plane. They got together and tried to figure out what to do. I imagine that they had a long list: they could go back to the design and try and simplify the controls, maybe they could eliminate the gust lock, they could have proposed significantly more training for potential pilots before they were allowed to take the air. They proposed none of those.

What did these engineers, test pilots and management propose?

A checklist. That's right - a pre-flight list of checks that had to be carried out and manually ticked off before the 'plane should being its taxi.

The introduction of this, on the face of it, very simple device allowed the Model 299 to come back to life. It became the B-17 bomber, known as the "Flying Fortress". Arguably, at a time when the Royal Air Force had no such long range bombing capability, this plane (and its descendants) helped win the Second World War - there were nearly 5,000 in operation by the end of the war.

There is, of course, lots more to this story - and plenty of online sources for information, such as Wikipedia, and the History of War.

As we forward wind 70 years, the check list has a vital and necessary role in day to day activities - especially in the complex field of technology delivery and implementation. And yet, I believe that it is tragically under-utilised because today's technicians think that they have it all in their heads and have no need of such an apparently basic tool. After all, when we get in a car and go to drive away, we don't use such a list; but, before we go on holiday, I'm pretty sure that most of us have a little list somewhere to make sure we don't forget our passports, our foreign currency, our flight confirmation, the address of the hotel and so on. Perhaps a checklist is only necessary for something we do rarely?

Medical evidence would seem to dispute that - even for the most basic, most regularly performed tasks, a checklist can make a significant and life-saving difference.

The work of Peter Pronovost demonstrates this. Pronovost is the Medical Director of the Center for Innovation in Quality Patient Care and an Assistant Professor in the Department of Anesthesiology/Critical Care Medicine at Johns Hopkins University's School of Medicine. Why? He introduced checklists into the day to day medical process in more than 70 hospitals in the USA (involving 110 ICUs). His initial premise was simple - mistakes happen so how do we prevent them?

For instance

In the United States alone, 7 percent of patients in academic medical centers experience a mistake with their medication resulting in up to 98,000 deaths a year. Those numbers are mirrored in Australia and the United Kingdom.

Hospitals that have implemented his checklists have seen dramatic drops in infection rates - saving hundreds of lives per year. Some of his checklists are surprisingly simple - yet they save lives. For instance, before inserting a line: wash your hands, use full barrier precautions (cover the patient), clean the insert site with chlorohexadin, avoid the femoral area for insertion. With this simple process, infection rates have dropped by 60% or more - some have seen 6 months or even a year with no infections. The hospitals involved have saved over $170 million and - more importantly - 1,500 lives.

So why am I writing about this here?

In many roles over the last few years both in and out of government, I've seen implementations delayed, projects fail, IT disasters occur and systems fall over because of basic errors. We have an incredibly complicated world of technology where even a small organisation has a myriad of different systems and applications installed. Big organisations have a full suite of operating systems and off the shelf packages, hooked together with miles of bespoke code. Those who like to live at the bleeding edge have complicated server utilisation models using virtualisation at CPU and storage level.

And yet, manual operation, manual installation and poor documentation seem to be the norm. "Finger trouble" is heard once a day in many projects and even in live systems as a reason for a delay or an outage.

In the medical environment, you can often rely on the team, whether it's a nurse or another doctor, to notice an error and correct it - were someone to die, "finger trouble" inevitably leads to a lawsuit.

In the technical world, where it often seems to fall to individual experts to get things done (you know the story, you have a supposedly leveraged team with masses of expertise available whenever you call but, in the end, there's only one guy or girl who knows your system's intricacies sufficiently well to get the job done), check before do would seem to be a sensible model.

Measure twice, cut once has, after all, been true for centuries.

(This post was inspired by an article I read in the New Yorker magazine which I subsequently couldn't find, so I rewrote it using online sources).