Sunday Essay - clouds, complexity & systemic failure

Last Sunday in a short note on another of my blogs, Problems with cloud computing - and paperless offices, I commented on the slowness of the cloud computer system: 

It is easy to forget in all this that the concept (cloud computing) is not new. What is new is the scale of application. It is easy to forget, too, that the performance parameters are not new. They are just the same as any other computer system.

I mention this because the single greatest user complaint is simply slowness. Everything just takes more time as compared to either the local WAN or single computer use. The results are quite noticeable.

A week later I would make the same comment but with greater force. It's quite frustrating.

Normal work often requires multiple computer actions.

Say that you are working on a document. You open Word. As you work, you are constantly finding and checking other documents or spreadsheets. This requires new programs to open. Most of those documents or spread sheets are to be found in the electronic records system. Save and auto-save activity goes on all the time.

With a full cloud based system, all the programs are held in the cloud and everything goes via the cloud. So every one of the many computer actions requires a path out and back with external processing .

In theory, this shouldn't matter. In practice, the volume of actions and traffic, the density of interactions between different parts of the system, can slow it to a crawl or even freeze an individual user. The message "unable to save" was a new one to me, but had somewhat dire consequences.

I have written a little about the problems created by rising systemic complexity. A key problem lies in the nature of the dependencies built into the system.

As systems become bigger and more complex, the number of actions or processes dependent on other actions or processes, the number of possible interactions, rises exponentially.

My present work involves process mapping. This flowed logically into a second data base and reporting project. Here the aim is to create a data base that will facilitate both operational work and reporting.

I am not an IT person. My focus is on the business processes that will form the core of the data base. As part of this, I am helping to identify the business rules that have to be translated into data speak.

In a way, it's eye glazing stuff requiring great attention to detail. However, the nature of the work illustrates the complexity point, for even at this level hundreds of relationships have to be specified. In a complex system, you may need to specify hundreds of thousands or even millions.

You can see why things might go wrong, why some new IT or information systems simply fail. You can also see why so many organisations are reluctant to change systems once they are in place. This gives rise to very particular difficulties for staff who then have to work round growing gaps between changing work requirements and the computer systems.

In Aymever Days, I have started documenting our experiences in establishing and running a national consulting business out of Armidale. Computer based systems were already well entrenched, but in our management consulting advice we did try to warn about the danger of entrenching bad systems. Computerisation allowed you to do what you were already doing faster and more efficiently, but what if those processes were inherently inefficient in themselves?

Because of our high technology industry focus, the world we worked in was littered with continuous and sometimes disastrous system failures in either development or application. Murphy's law was a constant presence.

One of the big cultural changes that was taking place at the time was a shift from an engineering to a computing focus.

Engineering people understood complex systems. They designed systems that worked, they had to, but could also be over-engineered with high levels of redundancy. Computer people, by contrast, had a much shorter term focus. They were described as cowboys, get the sale and move on. Technology is changing, go with it. The number of non-working software systems that resulted was substantial.

Of course, there has been convergence as the computer focus moved from hardware to software and systems. But it remains true, I think, that our increasingly complex systems are more vulnerable than in the past, the risk of adverse results higher.

As a simple example, many people love electronic locking in cars. The Australian family recently trapped in their car in full sunlight because of a failure in electronic locking may now have a different view. Passers by had to smash a window to get them out.  

In 1811 a protest movement began In Nottingham, England, against the introduction of new wide-framed automated looms that could be operated by cheap, relatively unskilled labour. This resulted in the loss of jobs for many skilled textile workers. Handloom weavers burned mills and pieces of factory machinery and also clashed in battles with the British  Army.

The Government suppressed the movement by force. Many ended up in Australia as convicts.

Those involved called themselves luddites. Since then, the term has come to be applied to all those opposed to the use of new technology. 

I am not a luddite. I am not opposed to cloud based systems, for example, although I am conscious of the risks and problems involved. I am, however, opposed to the blind application of technology. I actually first wrote "new technologies", but of course most of the technologies involved aren't actually new!

Looking back at my writings around this topic, I think that the thing I have struggled most to get across is the way that technology reflects the way we think and structure, but then in return affects the way we think and structure. The result is a complex feedback loop in which technology has moved from servant to master. We are no longer what we think, but what we use.

Last year there were a number of major system failures in the Australian banking system that left millions unable to be paid, businesses unable to use EFTPOS, to pay or collect accounts. The periods weren't long, but reminded many that there were risks attached to the convenience they enjoyed.

People can understand and respond to this type of problem because the effects are quite clear. The general issues associated with growing systemic complexity, the problems with the way that systems and thought interact, are far harder to get across. It is also far harder to do anything practical about them because of the huge investment in existing systems.

Far fewer people today have real decision making powers than, say, thirty years ago, while the scope of those powers has shrunk. Decision processes have become longer and more complicated. Far more Australians are involved in simply ensuring compliance with rules and laws.

In a way, you could argue that I am my own worst enemy.

Look at the work that I am doing - process mapping leading leading to a new data base and reporting systems. Process mapping is, in fact, the hand maiden of standardisation and computerisation.

In one way or another, I have been involved with most of the things that I now complain about. In some cases, I was a relatively early exponent. My difficulty is that they haven't delivered the benefits I expected, while the costs have been far greater.

I would argue, I think, that the biggest and saddest loss in what has happened lies in the way that it has reduced the capacity for true innovation simply defined as applied imagination.

We have replaced a thousand flowers with a much smaller number of larger flowers that have begun to wither. We apply more fertiliser, technology, but without much effect. We put stakes in the ground (rules and laws) to try to hold the flowers up, to prevent them falling over, but then find that more of the same is required.

Perhaps it's not a good analogy, but I hope that it gets across a little of what I am trying to say.