Monday, May 25, 2015

Thesis, hypothesis and scientific method in history

Note to readers: This post began as my Sunday Essay post, but I ended up rewriting it entirely. So now its both Sunday Essay and Monday Forum!
Recently on this blog we have been talking about evidence based approaches. In a postscript on
The application of evidence based approaches 2 - A note on evidence based policy, I referenced an article that came via Club Troppos' Nicholas Gruen: The Trouble With Scientists. I have been musing on that and, more broadly, the issues raised by my regular commenters on evidence based approaches.

I suppose that I could put the questions in my mind this way: how do we know what we we know, how do we know that we are right, how do we test that we are right, how do we correct when we are wrong?

By training I am both an historian and economist. Let me start with the history part in this post.

As an historian, I try to discover and tell the story of things that I am interested in. This involves analysis of evidence, the development of patterns and relationships that I can express mainly in writing for myself and the reader.

As an historian, I have always been uncomfortable with the thesis or hypothesis approach where that is defined first and then tested against the evidence. I am a curious person, I want to know what happened, why it happened, not prove a particular point. So I ask questions of the evidence. What happened? Why did it happen? What was connected? The idea that I must start with something that I want to "prove" or "test" or even that I must start with a single question makes me quite uncomfortable.

In working, I am always conscious of the problems of selection, perception and bias. This affects the questions I ask, the evidence I select and, to a degree, the way I approach the evidence. Working my way through, I work out the story that best represents the evidence as I see it. But am I right?

There are simple things that I can do to test my work. I don't especially want to refute my own work,  but I do apply certain simple tests that I have learned from experience are important.

The first is simply a date check. This may sound quite self-evident, but if a follows b, then it's sensible to check the dates of a and b. I said that date checking may sound self-evident, but we live in a world in which the idea of dates in history is sometimes treated in a rather cavalier fashion. You have to do it though. I still remember my total embarrassment a few years back when a simple later date check forced me to put a line through a total line of argument that I actually thought was rather good. I still blush.

 The second linked thought is to look at time linkages. Again, this seems self evident; a may follow b, but is there enough passage of time to make any connection between a and b? This is especially important if you are arguing causation rather then connection. Sometimes, a specific focus on time delivers apparently unbelievable results.

This is true of both the establishment of the Armidale Teachers' College and New England University College. I was very suspicious of some of the dates, I just thought they couldn't be right, but when I checked I suddenly thought how the bloody hell did you do that, it couldn't be done today. Our systems wouldn't allow it. The speed of action had suddenly became part of the story itself.

The third test is simply geography. This is a quite remarkable one that I was unconscious of  for a very long while even though I was interested in geography. At the simplest level, it's just this.Get out a map. Look. Think about climate. Look. What does it all mean for your story?

Let me give you an example. Those on the first fleet almost starved after arriving at Sydney. All the early explanations focused on things like lack of skills or equipment or the convict system. The reality, it seems, was that there was a drought, an El Nino event. Now the climate knowledge came later, but the drought stuff could, I suspect, have been picked up from the records if one had asked the right questions.

No matter how well I analyse the evidence, I know that my first pass description is my  hypothesis, a story based on evidence for later checking and refutation.

Those who wish to test my conclusions, the story I have told, may use the same evidence but operate in in a different way. They may test my story for internal inconsistency. Do the bits hang together? They may ask different questions of the same evidence, coming at the same pattern from a different angle. More often, they will find new evidence that challenges my view, leading to a new position. .

Sometimes, one can be very lucky. History builds on itself. I am still a bit staggered that some of the conclusions I drew in my history honours thesis, I was only 21, actually went on to form key building blocks in later analysis by people of far greater academic stature than me. My thesis wasn't especially well received at the time, .it got me a 2-2, but it has survived to the point that I am now using it and the subsequent discussion including opposition to create an entirely new hypothesis that can, in turn, be tested.

Quite remarkably, really! Bu that's what the study of history is about.

Later, I will extend this discussion by looking at economics.  


Winton Bates said...

I find it hard to understand how it is possible to be able to write the story of what happened without beginning with any ideas of what might have happened. There must be some evidence of some kind.
Perhaps there is report of a body being found with knife wounds. You presumably accept this provisionally but seek to test this story (hypothesis) against evidence. If the story adds up, you then presumably proceed to try to discover who done it by ruling out potential suspects. In seeking to rule out each suspect you are testing the hypothesis that they are guilty.
The alternative would be develop a hunch about the identity of the guilty party and collect evidence to support that view. I hope that is not how history is written.

Jim Belshaw said...

On you last sentence, Winton, in the crime genre that's just what can happen! You need a new angle.

There has to be something that interests you in the topic to begin with, of course. So in this case, you start with the dead body. The question that you ask is who done it? Sometimes there is a logical suspect. Did they do it? So you collect and test evidence.

Sometimes you may be able to prove things beyond reasonable doubt. Even then, you may be wrong. Think of the number of convictions over-turned by DNA evidence.

The broader the topic, the greater the uncertainty.

I think that one needs to be careful about open and closed questions. Who might the murderer be is an open question. Did so and so do it is a closed question. Hypotheses are closed questions. I favour open questions.

As in all things, one has to be careful about semantics. I regard my history as hypothesis because I generally can't know. I am putting forward a story based on evidence as I see it for critique and refutation.

I could wish that more policy makers had some form of history training. It makes one very cautious!

2 tanners said...

Policy makers should not study history, as they are often being required to replicate the mistakes of the past. Mass public service sackings at huge expense and just about every PPP that I know of are examples.

Second, if others are working off your not-well-received thesis to see further, then in an Isaac Newton/William Shakespeare mashup, does that make you a giant dressed in dwarf's robes? If so, watch the knees in those cold New England winds.

On a more serious note, I'm glad you wrote this. DG put in the scientific argument that an evidence based approach requires a null hypothesis. I'm not convinced of that. What I am convinced of in policy is that the options of doing nothing, and even discontinuing what you are doing now, should always be explored.

If neither of the above are null hypotheses (and I don't think they are, because often they don't admit of a closed answer), then you need to admit the fluid state of society and the imperfections of evidence. Jim has pointed out the difficulties of determining what actually happened - I suspect in economics he may go on to what we used to call the contribution/attribution problem: how much did the action affect the outcome?

Winton Bates said...

Thanks for the clarification, Jim. I think I might have misunderstood what you meant when you wrote that you were uncomfortable with the thesis or hypothesis approach.

Winton Bates said...

2T: I hope Jim does give prominence to the 'how much' question when he comes to talk about economics. I imagine that is also an important question in history. For example, how much influence did the actions of a particular person or organisation have on subsequent events?

Winton Bates said...

A further thought. When we say we begin with an open mind, we are often running with several hypotheses and seeking to choose among them. We may end up concluding that the weight of evidence favours explanation B rather than explanation A, but we cannot eliminate the possibility that A may have made a contribution.

Jim Belshaw said...

Smile, 2T. I have asked David to define a null hypothesis because I wasn't sure. I suspect, by the way, that there is a lot of agreement around once we get through language barriers.

On the history thing, one of the saddest comments I heard came a few months back from someone who said that the agency in question was getting rid of people with past knowledge because it impeded the change process. While I can see that this can (sometimes should)happen, I have significant professional problems with it. Here I am speaking as a fairly successful change agent.

I might look at that one Bob,contribution/attribution, although I'm more likely to place weight on correlation/causation. Winton, I am going to use a background example, that you might disagree with.

In all this, I probably won't be saying anything profound. I suspect (know) that DG could run rings around me in a theoretical sense. I am just thinking as a reflective practioner.

Anonymous said...

All swans are not white.


Anonymous said...

Which has been misinterpreted so badly that it is now one of the most cloying impediments, to innovation, in any field of endeavour.


Winton Bates said...

Please explain. A comment on Taleb?

Anonymous said...

Good morning Winton. More a comment on reactions to his analysis which I thought quite reasonable. In hindsight (if you will pardon the pun) I think it was unfortunate that his selected example was coloured black; that (non) colour seems always associated with 'bad' things, whereas he seemed to be suggesting that many such 'events' produced good outcomes - and that seems to have been lost.

Maybe we need to reinvent a Toynbee Convector.


Winton Bates said...

kvd: There seems to have been a shift in meaning of the reference to black swans. David Hume referred to black swans to make a point about the provisional nature of knowledge i.e. the impossibility of inferring that all swans were white on the basis that at that stage no-one in the part of the world where he lived knew of the existence of black swans. More recently, references to black swans seem to be about improbable events. I suppose the link is in the use of the frequency of occurrence of events in the past to assess the probability that events will occur in future.
I agree with your point that in discussing innovations there is now often too much emphasis on the worst case scenario. However, that doesn't apply in cases where the worst case scenario is really bad e.g. collapse of the banking system, or potentially catastrophic e.g. runaway global warming.
Who needs a Toynbee Convector when we have climate change modellers telling us what the future will be like?

Jim Belshaw said...

Interesting. I have actually discussed the null hypothesis previously here then in the context of if a then b as compared to if a not b and indeed a and b - causation, nullification, correlation. All this (for me) goes right back to logic in philosophy I and then, a little later, the philosophy of history.

Ah, the Tonybee Convector. I had forgotten.

Anonymous said...

Just a question for Winton if you are still reading: do you ever make use of this statistics site, and if so do you believe it is reliable?


Jim Belshaw said...

Hi kvd. Winton may have moved on. I hadn't seen that site. Its quite interesting, although some of the comparison stats are a little dated. Will bring it up so that people can look at it.