This brief note continues my discussion on what are called evidence based approaches. At the start of The application of evidence based approaches 1 - Evidence Based Medicine, I wrote:
The discussion on Monday Forum - more on evidence based approaches looked at some of the issues associated with evidence based approaches. One of the issues from my viewpoint in that discussion lay in the need to distinguish between the importance of evidence and the application in practice of what are called evidence based approaches whether in medicine, management or public policy. I thought therefore that it might be helpful, at least to me, to consolidate some previous writing on the topic, starting with evidence based medicine.
I have repeated these words to emphasise that I am concerned with the application of particular models, ways of thinking. The Wikipedia article on evidence based public policy is not especially good, but it does refer to the transmission of the particular form of the idea from medicine to public policy.
A good expression of the idea in an Australian context comes from this Australian Public Service Commission piece, there is no date but I think its 2009, featuring Gary Banks. In her introduction, Public Service Commissioner Lynelle Briggs states:
Evidence-based policy-making, while not a new concept, has recently become more prominent in public debate in Australia. The Prime Minister has called it a key element of the Government’s agenda for the public service. He wants policy design to be driven by analysis of all the available options, and not by ideology. This explicit endorsement by the Prime Minister provides us with a valuable opportunity to advance the cause of evidence- based policy-making in the APS.
In Australia, the potential application of evidence based approaches drawn from medicine to policy dates back to at least 1998. As happened with me, it seems to have spread first from medicine to the idea of a discipline of practice and then beyond.
Mr Banks and the Productivity Commission have played an active role in popularising the application of the concept.in the public policy sphere, claiming lineal responsibility for its application back to the old Tariff Board. In fact, the concept is much older than that. The economic historian C R Fay used the term evidence based public policy in 1919, and by then the approach was quite old.
The differences between the old and new approaches is that evidence based public policy now involves the application of a highly structured top down model that, to my mind, is riven with potential contradictions that arise from its misapplication.
I recognise that I have to argue this, I am as prone as anyone else to management fads and fancies. Someday I should write a mea culpa post looking at all the things that I have argued for that I then found did not work in practice. Because I have been a practitioner working across a number of fields, I have found myself badly bitten by the very things I had espoused. The problem, to my mind lies not in the concepts as such, but always in their sometimes blind misapplication.
In my next post in this irregular series, I will look at evidence based management.
Postscript
I should note, for the record, that I have a very high opinion of the work done by the Tariff Board though its various guises to the Productivity Commission today.
Postscript 2
This link came from Nicholas Gruen: The Trouble With Scientists.There are several points within it that bear upon our current discussion so I wanted to record it for later use.
Postscript
I should note, for the record, that I have a very high opinion of the work done by the Tariff Board though its various guises to the Productivity Commission today.
Postscript 2
This link came from Nicholas Gruen: The Trouble With Scientists.There are several points within it that bear upon our current discussion so I wanted to record it for later use.
.
21 comments:
Trust it to Jim to put his awesome work on evidence based policy to give support to the cause of a Newcastle based yoga blog.
I almost took it down, 2T, but it was Newcastle!
1998? What about the work of Alf Rattigan at the IAC in the 1970s? Also, in the reference cited, the authors fail to grasp that the key feature of 'gold standard' evidence in medicine is that it is double blind. Generally difficult in uncontrolled public policy environments, except perhaps where natural experiments are possible.
DG
Jim,
I think it might help my understanding if you give an example of a highly structured top-down model. I guess it has to do with the criteria used in evaluations, but that is just a guess.
In some policy areas analysts struggle to find quantitative indicators of performance, very much aware of the shortcomings of the indicators they come up with. However, once this data is published it becomes a focus of policy attention. So we have political parties promising to improve outcomes as measured by these imperfect indicators and so they develop a life of their own as bureaucratic goals.
DG first. I added the postscript on the Tariff Board to PC just to pick up your point. Evidence based public policy has been around for some time time. Consider the House of Commons inquiries during the nineteenth century. However "evidence based public policy" in the way the term is used now is recent. I am focusing on current models.
Winton, your second sentence captures a central concern of mine.
I will come back with more comments later.
In line with my previous comments a top-down policy approach can work with an evidence based policy technique. The policy aim is not defined by evidence; it is an objective agreed by the management to be desirable. The evidence base is used to determine the most efficient and effective means of achieving the preset objective.
Example: It was decided to use foreign aid to help poor communities develop their own commercial development base. A number of approaches were examined, including microfinance, education, provision of mobile phone services, provision of basic roads, development of actual rural market places etc. The successes and failures and costs of each of these were examined, along with the country context. Depending on these factors, one, or acombination, of these approaches were tried. Careful monitoring and reporting was in place, with a resulting expansion in some activities and cutback or cancellation of others. It was all low cost, finely targeted and economical in that good money was never thrown after bad. There was a heavy cost overhead in monitoring and evaluation, but I'd argue we could often do with more of that, not less.
My $0.02.
Sorry, I should have said the successes etc of previous implementations etc. Hope that was obvious.
Hi Winton and 2T. Now picking up your comments.
2T gave an example of a top down approach that seems to have worked. It started from the premise that management set the objective then evidence was collected to define the best path.
My response will go beyond evidence based public policy to link in other things that affect operational performance.
Consider, for the moment, that first sentence, management set the objective. "The policy aim is not defined by evidence, it is a policy aim agreed by management to be desirable."
If the policy aim is not defined by evidence, and this is not really a policy aim in this case but a program aim, then evidence based work is limited to the doing.
And who is management?
One of the difficulties in public administration lies in the hierarchy of decisions and targets that are meant to cascade down from the top in ever increasing detail each with its own set of performance indicators. There is less and less room for individual initiative.
2T called me a pirate. We were able to do real things because, for a brief period, the system opened up. Then came the re-centralised control at a much greater degree of rigidity than had ever existed before. That stifled initiative, limited action to the centrally mandated and broke feed back loops.
A second problem is the confusion that arises in a westminster system between the role of ministers and the executive and agencies and departments. Ministers set values and broad objectives. In doing so, they should be subject to evidence based scrutiny where appropriate, but the top level political system is actually not about evidence.
In the policy environment, the role of Departments is to provide advice taking the minister or government's view as a starting point. This may include cautions and options, but in the end the government sets the frame.
I am going to have to stop here. I need to eat! I will continue.
Continuing now. An agency that can set independent objectives is one that has a defined set of service objectives or activities. A problem arises when the agency sets objectives that actually belong to government and expresses those in KPIs. This places the agency in an impossible delivery position.
Thinking about it, my problem the way that evidence based policy models are applied lies partly in scope, the misuse of the approach outside the proper bounds, partly in the way that the approach combined with centralisation gives rise to perverse results.
Winton wanted an example of a top down problem. I will take an actual case, but will have to generalise it. I have to think about that.
I used the word management fairly coyly, I'll admit. I meant it to mean anyone who could independently make policy settings which then needed to be implemented.
Perhaps this was where DG and I were at cross purposes some threads back?
Your policy objective is values based. You decide what is desirable and from those options what is most important. There is some room at this point for evidence based policy setting on a CBA (or similar) basis, but the menu of options from which you are choosing has already been settled.
Gary Banks and Lynelle Briggs referred to evidence based policy as a tool of policy DESIGN, not basic policy selection.
The conversation is helpful in clarifying things.
One issue is the difference between the use of evidence and the very particular model or models in evidence based public policy.
The Bank/Briggs focus on the evidence based policy model as a design tool can be judged in terms of the opportunities and problems involved as I did in pointing to some of the issues. That was my original interest.
When Blair or Cameron or others talk about evidence based public policy, its not clear to me what they actually mean, what is new in the approach.
DG pointed to several issues. One was the IAC under Ratigan using an evidence based approach. He was arguing by implication that its wasn't new, but then he wasn't talking about the recent model as such. Then in the context of medicine,he referred to the difficult of applying the medicine approach (I think he referred to the difficulties of using double blind tests).
I am wondering now whether the term evidence based policy actually has any meaning beyond the obvious one that has been in play for a very long time. I can see it in a professional practice context, although its practical application in law for example can be very difficult in part because of client resistance.
Mmm. I am going to leave it there. I have to wash up!
Yes, in place for a long time: consider the case of the Bow Street Pump handle and the arrest of cholera (1854) (John Snow) - that gave ultimate impetus to the work and influence of Edwin Chadwick (already underway) on sewerage and the enormous improvement in public health and reduced mortality in 19th Century England.
DG
Good examples, DG.
It is nice to be able to agree with Nick Guen about some things.
However, the ideal of not using the same data to generate hypotheses as to test them seems impossible. We start with preconceptions which we test against the data. Perhaps we think we are involved in a measurement exercise, but the analysis produces unexpected results. So, we go back and re formulate the hypothesis to try to explain the results. I don't see the problem as long as we report honestly what we are doing.
The hypothesis is further tested later against new data to see if the predictions are borne out. In some instances it is possible to test using data for different countries.
Yes, you can accomplish a great deal by massaging data until you finally stumble on what you are seeking: as mentioned elsewhere, it's 'policy-based evidence'. I've been obliged to do this for all sorts of unworthy purposes in the past (reporting assumptions, of course, by way disclaimer) - sometimes, I'm bound to admit, to resounding effect.
DG
Like DG,I too have done a fair bit of policy based evidence work! I have also seen a lot of exclusion of evidence that doesn't fit the prevailing model, along with rejection of questions that might challenge the model.
The testing that follows doesn't test the alternatives for fit, just the results that might be expected from the model.
When the testing doesn't fit, the conclusion often is not that the model itself was wrong, but the evidence was wrong or there was some error in the formulation or in the delivery.
Jim abd DG:
So your real problem is policy-based evidence masquerading as evidence-based policy?
No. Best to carefully formulate null hypotheses before their testing so that they are properly falsifiable (a la Karl Popper).
DG
Sorry for the lagged response. No, 2T, I'm still concerned abut current models. DG, to help my thinking could you give me an example of a null hypothesis?
All swans are not white
DG
DG, I'm still trying to think through just how the null hypothesis actually works - https://explorable.com/null-hypothesis
Post a Comment