tag:blogger.com,1999:blog-24338064.post8328156537930291828..comments2024-02-11T19:28:27.997+11:00Comments on Personal Reflections: Monday Forum - more on evidence based approachesJim Belshawhttp://www.blogger.com/profile/10075614280789984767noreply@blogger.comBlogger17125tag:blogger.com,1999:blog-24338064.post-79286806510009035192015-05-10T07:21:59.970+10:002015-05-10T07:21:59.970+10:00I know that, DG!I know that, DG!Jim Belshawhttps://www.blogger.com/profile/10075614280789984767noreply@blogger.comtag:blogger.com,1999:blog-24338064.post-55831525722789668772015-05-10T00:38:19.079+10:002015-05-10T00:38:19.079+10:00Can't agree with your last para, Jim; or that ...Can't agree with your last para, Jim; or that use of evidence (which, if to be meaningful, should reduce to a metric) "precludes learning from experience".<br /><br />DGAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-24338064.post-14551784615868621522015-05-09T23:55:02.141+10:002015-05-09T23:55:02.141+10:00I must be tired tonight, for I'm getting a lit...I must be tired tonight, for I'm getting a little lost. So keeping things very simple.<br /><br />When I talk about evidence based policy or medicine or management, I am talking about a very particular thing, a currently popular model, that I happen to think has major problems. I am concerned with the way things work - or don't.<br /><br />My concerns about the current approaches can be summarised thus: they lock us into the past; they ignore the importance of experience; they limit action to the measurable; they limit assessment of the results of action to the measurable; they attempt to incorporate too much; and they are applied in a blind rigid fashion that, in a public policy context, slows decision making, prevents new ideas, precludes learning from experience and grossly adds to overhead costs. Jim Belshawhttps://www.blogger.com/profile/10075614280789984767noreply@blogger.comtag:blogger.com,1999:blog-24338064.post-4050308107759245852015-05-09T11:50:06.705+10:002015-05-09T11:50:06.705+10:00Are we in furious agreement? DG, to me your answer...Are we in furious agreement? DG, to me your answer is exactly about an evidence base (including the option of doing nothing new) and Jim's answer seems to recapitulate my descrion of the process. <br /><br />Perhaps as a clarification there is a real problem which I acknowledge between contribution (what you do) and attribution (how have you helped achieve your ultimate goal). This IS a weakness of evidence based policy.<br /><br />Finally, any policy will always need alternatives/other actions. I've never seen a magic bullet solution that worked solved a large, complex problem.2 tannersnoreply@blogger.comtag:blogger.com,1999:blog-24338064.post-22135741596983790382015-05-09T08:50:13.246+10:002015-05-09T08:50:13.246+10:00Jim, your answer is a muddle. Any serious attempt ...Jim, your answer is a muddle. Any serious attempt to employ an evidence base invites at least either a test for allocative efficiency (cost benefit) or one for technical efficiency (cost effectiveness). In the case of the latter, using the end point of a reduction in harms to women and refuges as a comparator as a against say, a crime and punishment remedy, might suggest that not all answers to public policy dilemmas reside in government intervention (with the corollary of additional layers low productivity public sector workers).<br /><br />DGAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-24338064.post-12221574506171906962015-05-08T19:03:18.849+10:002015-05-08T19:03:18.849+10:00Hi Bob. I have a little more time tonight so a ful...Hi Bob. I have a little more time tonight so a fuller response. I’m not actually sure how far apart we really are. This might help clarify.<br /><br />Recently I was looking at some papers connected with a major policy review. The document in question was the draft outcomes framework that was then to provide the base for the evaluation framework. The outcomes were expressed in hierarchical terms with broad result categories supported by performance indicators. The selection of indicators was based partly on research examples as to what was relevant, partly on the statistical measures available. <br /><br />To my mind, the whole thing was a potentially dangerous waste of time because of two disconnects. The first disconnect was that between the indicators and the core policy problems that were driving the changes in the first place. The second was between the multiple statistical indicators and what was actually happening on the ground.<br /><br />This is not a unique problem. Closing the gap has many similar features and is actually undeliverable in a statistical sense and also disconnected with needs. It is based on certain types of evidence, with results measured by another set of indicators.<br /><br />Turning now to the NSW public education example. The why is based on the belief, that a certain minimum level of compulsory primary school education was good for the community. The what or how required the provision of schools. With a cash constraint accentuated by a rapidly spreading population, the challenge was to get some education out there, so there were a whole series of improvisations. The output and outcome measure was kids in schools. <br /><br />Now going to your women’s refuge example and ignoring DG’s statist comment. The why is that women and children in certain impossible situations should have a way out. The what is refuges. Note that this deals with worst case solutions but does not address the underlying cause. This doesn’t mean that it’s a bad thing, just a partial thing. You say that safer women is the outcome. That’s what needs to be measured for it’s not always clear cut. <br /><br />In the old policy environment, you would do something because you thought that it was a good thing based on a mixture of example, analysis and politics. Then, over time, and there was time, you would seek to improve. Is this still possible today?<br />Jim Belshawhttps://www.blogger.com/profile/10075614280789984767noreply@blogger.comtag:blogger.com,1999:blog-24338064.post-22115660560413033582015-05-08T17:12:49.435+10:002015-05-08T17:12:49.435+10:00Jim, I'm not familiar with your examples. How ...Jim, I'm not familiar with your examples. How were the decisions as to WHAT to do (not why to do it) taken and why do you say the examples were successful?2 tannersnoreply@blogger.comtag:blogger.com,1999:blog-24338064.post-14996510302500590592015-05-08T10:21:11.286+10:002015-05-08T10:21:11.286+10:00Hi 2T. While most policy is based to some degree o...Hi 2T. While most policy is based to some degree on evidence, the current formulation of evidence based policy with subsequent evaluation is quite recent. It’s also very rigid. <br /><br />The NSW public school system is an example of a primarily non-evidence based policy. Other examples include the spread of adult education. This was driven by ideas of equality and social advancement. <br />Jim Belshawhttps://www.blogger.com/profile/10075614280789984767noreply@blogger.comtag:blogger.com,1999:blog-24338064.post-21666299478052228962015-05-08T07:27:13.544+10:002015-05-08T07:27:13.544+10:00DG, I didn't mention "best" anywhere...DG, I didn't mention "best" anywhere, and I implied this was an imperfect approach through the need for monitoring and evaluation. The domestic violence problem is multifaceted, and has societal, behavioural and economic roots. I am simply talking about one small policy to deal with one identified situation. I have not suggested that the policy is sufficient, simply that it has a chance ON THE EVIDENCE,of being more effective than alternative policies.<br /><br />I've put up, now it's your turn. <br /><br />Show me an example of a successful non-evidence based policy, and why it was better than any alternative. 2 tannersnoreply@blogger.comtag:blogger.com,1999:blog-24338064.post-52094574984218423312015-05-08T00:38:30.648+10:002015-05-08T00:38:30.648+10:00Really? Is this the best way to make 'women sa...Really? Is this the best way to make 'women safer'? It's a typically Australian, statist approach.<br /><br />DGAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-24338064.post-60325489365373209592015-05-07T08:51:01.110+10:002015-05-07T08:51:01.110+10:00And while you are responding, the last Government ...And while you are responding, the last Government organisation I worked for didn't like failure (in a pretty major way) but prepared independent reports on its projeccts and carefully tried to avoid repeating those failures.<br /><br />It was very prepared to experiment, and knew that context was vital.2 tannersnoreply@blogger.comtag:blogger.com,1999:blog-24338064.post-15587668776793018022015-05-07T06:41:12.584+10:002015-05-07T06:41:12.584+10:00Hi 2T. Back to you on this tonight. JimHi 2T. Back to you on this tonight. JimJim Belshawhttps://www.blogger.com/profile/10075614280789984767noreply@blogger.comtag:blogger.com,1999:blog-24338064.post-45189690271308986932015-05-06T23:24:48.467+10:002015-05-06T23:24:48.467+10:00Harkening back to your univeristy philosophy, evid...Harkening back to your univeristy philosophy, evidence will never determine the course you should follow (you cannot derive an "ought" from an "is" alone).<br /><br />Having decided what you ought to do/achieve/etc, evidence becomes useful in determining the 'how'. <br /><br />We use measurable outputs to support (not prove) claims of achievement in less measurable outcomes. E.g. We OUGHT to make women safer so we WILL build and staff refuges because there is evidence to show this works. So the number of sustainable refuges is the measurable output, building and staffing them is the action and safer women is the outcome.<br /><br />What's so hard?2 tannersnoreply@blogger.comtag:blogger.com,1999:blog-24338064.post-8717176786245756822015-05-06T04:17:59.241+10:002015-05-06T04:17:59.241+10:00I think that's right Winton. If you take Singe...I think that's right Winton. If you take Singer's point on the changing definition of rape and its impact on the stats, you have an evidence based point. He had to qualify heavily at once, to make the distinction between the impact on measurement and the question of what's right. Jim Belshawhttps://www.blogger.com/profile/10075614280789984767noreply@blogger.comtag:blogger.com,1999:blog-24338064.post-48298982694625847462015-05-05T20:30:04.193+10:002015-05-05T20:30:04.193+10:00I watched Q&A last night. The question of evid...I watched Q&A last night. The question of evidence based policy was brought up at several points in the discussion of various issues. <br />I think that helped leave me with the impression that the level of discussion was unusually sensible.Winton Bateshttps://www.blogger.com/profile/07383561940886657594noreply@blogger.comtag:blogger.com,1999:blog-24338064.post-26772008859796562922015-05-04T19:48:38.925+10:002015-05-04T19:48:38.925+10:00I think that I've become perhaps too cynical, ...I think that I've become perhaps too cynical, 2t. <br /><br />I agree with your point, we tried to do this, but two problems arise. One is the way that outcomes are measured Today only measurable outcomes are allowed. The second lies in the failure to recognise the importance of experimentation and failure. Jim Belshawhttps://www.blogger.com/profile/10075614280789984767noreply@blogger.comtag:blogger.com,1999:blog-24338064.post-46608401030753223482015-05-04T14:48:34.772+10:002015-05-04T14:48:34.772+10:00I wrote a huge long screed and deleted it. If you ...I wrote a huge long screed and deleted it. If you know what outcome you want and you fail to research the environment, policy options, current economic movements and past relevant policy failures and successes, the chances are very small that you will succeed. <br /><br />They are very large that you will recapitulate history, having refused to learn from it.2 tannersnoreply@blogger.com