Friday, November 09, 2012

Wrestling with confirmation bias

In a comment on Somewhat depressing threads, Winton Bates wrote in part:

I suspect confirmation bias in the pattern you observe, but that doesn't mean you are wrong.

The pattern I see is about the consequences of growth of government. It displaces civil society, it makes people dependent, it responds to every problem with more regulation. It invites disrespect.

But that is a pattern I see everywhere!

I had to laugh. As Winton notes, I may be guilty of it, but in his dry way he is also accusing himself!

Confirmation bias, the tendency to selectively interpret material so that it reinforces our own views, is pretty common. We saw it in some of the reactions to Julia Gillard's now famous speech. We saw it, too, in the US Republican reactions to the polls and polling results casting doubt on their candidates chances. It actually became a circular dance as one news channel or commenter quoted another in support of their views.

We see this all the time on the modern internet as people drift towards sites that support and reinforce their own views. Arguably, this has become one source of deepening polarisation.

I claim to be a social analyst. How, then, do I avoid confirmation bias, something that I get accused of directly or indirectly from time to time?

At one level, life is just too short to bother. I have a bad habit at dinner parties or social functions of trying to explain, of being too objective. Bad Jim! Wrong place, wrong time. Just listen. Your social analyst hat is just inappropriate If people are having a lot of fun piling opinion on opinion.

At other times in similar circumstances if the mood takes me, I will deliberately throw in an alternative view, something that I know will be a red flag item. Done with discretion, this can be fun, but discretion is advisable. You must be able to stand back and not actually become personally involved in the issues.

Then, at other times, I may just want a chance to talk and express my opinions without having to worry about just what I say. Here confirmation bias can be a fine thing!

Yet when trying to write as a social analyst, I am concerned with what is, what it might mean. Sometimes I go beyond this, for I try to provide solutions, responses, to the problems as I have defined them. They are my solutions, I own them, I want to argue for them.

But when writing in a professional way, I use three different techniques to try to control my natural tendency to confirmation bias.

The first is by consciously reading, and its a real effort some time, opposing views. In political terms, this takes me to the blogs and commentariat of left and right.  I tune out the things that are standard opinions beyond noting patterns. Instead, I look for the kernels that might challenge or enhance my own views.

The second thing I do is what I call point and counterpoint. I start with the particular then test with the general and then back to the particular.

The thing that  I do is to try to test by evidence. In the US election this time, everybody started talking about the impact of demographic change. This one has been obvious to Blind Freddy for some time. If you look at my own writing on change in Australia, you will see that I have often quoted demographic or other forms of statistics.

Some of my writing is not easily amenable to statistical test without special surveys. I argue, for example, that changes in management styles in public and private sectors have increased overhead and slowed decision making.

I can show through simple arithmetic that decision times slow and decision costs increase as the number of decision points rises. I can show by logical analysis the way in which the obsession with KPIs and with the measurable can twist decision processes. I can use simple mud map analysis to show how decisions such as those made by the NSW Government on the length of time required to get a driver's license have imposed costs and had a range of adverse outcomes.

What I cannot do is provide an analysis that measures the overall costs of all these things in a way that would grab attention and test my case. The data presently isn't there. I have to rely on point and counterpoint, generalising and then testing with specific examples. But it remains qualitative analysis.

Having said this, I wonder if there is more that I could do to test. But that is a matter for a later discussion. I really need to cook tea, not chat to you!


Winton Bates said...

Now you are now making me think seriously about whether we can assemble useful information from casual conversations (or even case studies). Perhaps the most they can offer is examples or anecdotes to illustrate points.

That is an important function because anecdotes seem to play an important role in helping us understand each other.

Changing the subject slightly, it strikes me that it is probably a good idea to be looking for examples of conflicting positions - for example, reasons to be optimistic as well as reasons to be pessimistic. Given my interests, I know that if I go looking for examples of 'aorta politics' I will find them everywhere (even in some of the things I say)and end up feeling pessimistic. But if I go looking for examples of human resilience - individuals meeting challenges and dealing with them - I will also find plenty of those and might end up feeling inspired.

Anonymous said...

Confirmation bias has never worried me in reading the thoughts of either yourself or Winton. After all - if you are always correct in your assessments - what is the harm in simply noting that others agree with your views?

Anecdata is good for illustration, rather than confirmation, I've always thought. But I'm seriously worried about this habitual queue-forming outside shops yet to open; and how one equates one's dying need for a packet of cigarettes with the notion that the whole country is going to the dogs.

Have a lovely day ;)


Evan said...

A reasonably considered conversation with 100 subjects is enough to give some pretty reliable indications.

With some things there may not need to even be 100 subjects I suppose.

Legal Eagle said...

I agree with KVD - I have never been worried about confirmation bias with you, Jim - you are open minded and fair.

Jim Belshaw said...

As an aside first, Winton, almost five years ago I wrote something on case studies -

I gain far more from both casual conversations and case studies than just example or anecdote, although the two are different. A casual conversation can indeed provide examples, but just listening may provide ideas or even challenges as well.

A case study is more structured. As I suggested five years ago, properly done case studies are a structured way of capturing and presenting lessons gained from experience.

With both, part of the point of the particular and the general is that it provides a structured way of both gaining and testing ideas.

Anecdotes do play an important role in helping us understand each other. But they are also a device for persuading. Sometimes in my Jim story teller role, I am trying to interest, but also persuade.

I absolutely agree with you point about conflicting ideas. That was one of my points.

Jim Belshaw said...

Oh dear, kvd. You did make me laugh! If Winton and I are always right, how come we sometimes disagree?

Actually, when writing professionally, I do have a pretty good track record. Yet tracking my writing over the years, I find constant examples of where I proved to be wrong! That is why I began that somewhat sporadic series trying to re-calibrate my views on Australia's future directions.

Jim Belshaw said...

Hi Evan. With 100 conversations, it depends on your sample! You have to ask with conversations how representative that view is. Especially with the committed, it's the shift in views that counts.

A hundred years ago in a different world when I was running election campaigns, a colleague in the Hume electorate used to follow just one tiny booth on voting night. This was a dedicated Country Party booth. He reckoned that he could forecast results for the whole electorate from tiny shifts in the vote at that booth. he seemed to get things pretty right.

There, Winton, that's an example of the use of anecdote!

Jim Belshaw said...

I try to be, LE! My thanks!